AI Data Pipelines in Seoul

Semiconductors / Advanced Memory Manufacturing

A Korean memory semiconductor manufacturer (KRW 280T annual revenue, operating 6 memory fabs and 2 logic fabs) needed unified data pipeline infrastructure to support their yield improvement and predictive maintenance AI initiatives across all fabs. (1) Data volume overwhelm: each of 6 memory fabs generated 1.2 billion data points per month — totalling 7.2 billion data points monthly across the memory fab fleet. This data existed in separate databases per fab — process parameter databases, SPC databases, defect databases, metrology databases, and equipment sensor databases — with no unified pipeline enabling cross-fab analytics. (2) Data latency: yield excursion detection relied on SPC charts monitored by engineers — but SPC data was updated hourly (batch processing), meaning a yield excursion could run for up to 60 minutes before detection. At the manufacturer's production volumes, each hour of a yield excursion at a single fab cost approximately KRW 2B in reduced yield. (3) Cross-fab analysis was manual: when a yield issue occurred at one fab, engineers manually checked whether similar conditions existed at other fabs — a process that took 2-3 days because data had to be extracted from each fab's separate databases. (4) Equipment sensor data was underutilised: each fab had 1,500+ process tools generating continuous sensor data (vibration, temperature, pressure, RF power, flow rates), but this data was stored in separate equipment-level databases without aggregation or standardisation — making cross-equipment and cross-fab analysis impossible. (5) Model training data was difficult to prepare: data scientists building yield prediction and defect classification models spent 70% of their time on data preparation — extracting data from multiple databases, aligning timestamps, handling missing values, and creating features — leaving only 30% for actual model development.

We built a unified fab-wide data pipeline over 16 weeks. (1) Real-time data ingestion: streaming ingestion from all 6 memory fabs — process parameters, SPC data, defect inspection results, metrology measurements, and equipment sensor data — into a unified streaming platform (Kafka) with standardised schemas. Data volume: processing 7.2 billion data points per month in real-time (240 million per day, 10 million per hour, 2,800 per second sustained — with peaks of 15,000 per second during lot starts). Latency: end-to-end from sensor reading to queryable data: under 5 seconds (from 60 minutes for SPC batch updates). (2) Unified data lake: all fab data stored in a unified data lakehouse (Delta Lake on cloud storage) — with standardised schemas enabling cross-fab queries. Data partitioned by fab, date, process step, and equipment — enabling efficient queries across any combination. Historical data: 3 years of historical data migrated from per-fab databases into the unified lake — enabling cross-fab historical analysis for the first time. (3) Real-time yield monitoring: streaming analytics detecting yield excursions within 5 minutes of occurrence (from 60 minutes) — correlating process parameters, metrology, and defect data in real-time to identify the affected product-process-equipment combination. Automated root cause suggestions: when a yield excursion was detected, the pipeline automatically queried historical excursion data with similar signatures and suggested probable root causes. (4) Cross-fab analytics engine: standardised queries enabling cross-fab comparison — "compare defect density trends for this process step across all 6 fabs" or "which fab has the best yield for this product and what are the key parameter differences?" Cross-fab knowledge transfer automated: when a process improvement was implemented at one fab, the pipeline tracked whether similar improvements could benefit other fabs. (5) ML feature store: centralised feature computation and serving for all fab ML models — yield prediction features, defect classification features, equipment health features, and process optimisation features. Point-in-time correct retrieval ensuring model training data did not contain look-ahead bias. Data scientist productivity: data preparation reduced from 70% of time to 20% — the feature store provided pre-computed, validated features for model training.

Yield excursion detection: from 60 minutes to under 5 minutes — each hour saved during an excursion prevented KRW 2B in yield loss. In the first year, faster detection saved an estimated KRW 185B across all 6 fabs. Cross-fab analytics: first time cross-fab comparison was possible — identified 23 process parameter optimisations from cross-fab analysis that collectively improved average yield by 2.1% across the fleet. Revenue impact of 2.1% yield improvement on KRW 280T revenue: approximately KRW 5.9T in additional good-die output value. Equipment predictive maintenance: unified sensor data enabled ML-based predictive maintenance across all equipment — unplanned downtime reduced from 3.2% to 2.1%. Production value recovered: approximately KRW 55B annually. Data scientist productivity: data preparation from 70% to 20% of time — 40 data scientists effectively gained 50% more productive time for model development. Model development velocity increased by 3x — 15 new production models deployed in the first year (from 5 in the prior year). Pipeline availability: 99.97% uptime — 2 minutes of unplanned downtime in the first year. Annual value: KRW 185B (faster yield excursion detection), KRW 5.9T (cross-fab yield improvement — this is an estimate of the full annualised impact), KRW 55B (predictive maintenance), KRW 35B (data scientist productivity and accelerated model deployment) = approximately KRW 6.2T total impact. Development cost: KRW 1.5B. Payback: under 2 hours.