AI Data Pipelines in Doha, Qatar

Energy / Oil & Gas / Industrial IoT

A Qatari energy company (major LNG processing facility — processing 77 million tonnes per annum, 4,200 employees, facility spanning 30 km² in Ras Laffan Industrial City) needed AI data pipelines to enable predictive maintenance across critical equipment. (1) Equipment data: the facility operating 280+ critical rotating equipment units (compressors, turbines, pumps, and generators) — each equipped with 15-40 sensors monitoring vibration, temperature, pressure, bearing condition, and lubrication. Data volume: 45,000+ sensor data points generated every second across the facility. Current state: sensor data collected by SCADA/DCS systems — stored in historian databases (OSIsoft PI) with limited analytical capability. The gap: historical data existed but wasn't being used for prediction — maintenance still primarily time-based (scheduled) or reactive (fix when broken). (2) Unplanned downtime: critical equipment failures causing unplanned production stoppages — averaging 14 incidents per year affecting production. Average downtime: 18 hours per incident. Production impact: each hour of LNG train downtime costing approximately QAR 3.5M in lost production. Annual cost: 14 incidents × 18 hours × QAR 3.5M = QAR 882M in production losses. Maintenance team: 380 maintenance engineers and technicians — 65% of work being reactive (responding to failures) rather than predictive or preventive. (3) Data silos: maintenance-relevant data scattered across systems. SCADA/PI: real-time sensor data and historical trends. SAP PM: maintenance work orders, equipment master data, and spare parts inventory. Inspection system: inspection records, thickness measurements, and corrosion data. Document management: equipment manuals, P&IDs, and maintenance procedures — many in Arabic and English. ERP: procurement data, vendor performance, and cost information. The problem: no unified view — a maintenance engineer needing to check 4-5 systems to understand an equipment unit's complete condition. (4) Prediction gap: the company wanting to predict failures 2-4 weeks before occurrence — giving enough time to plan maintenance, order parts, and schedule downtime. ML requirement: predictive models needing clean, timestamped, unified data — combining sensor readings with maintenance history, operating conditions, and equipment specifications. Current analytics: basic threshold alerting (alarm when sensor exceeds limit) — no pattern recognition, no degradation tracking, and no failure prediction. (5) Arabic documentation: 35% of maintenance records, inspection reports, and operator logs in Arabic — valuable context for maintenance decisions but inaccessible to data analytics systems. Arabic data: operator log entries like "لوحظ اهتزاز غير طبيعي في المضخة P-3401 خلال الفترة الصباحية" (abnormal vibration observed in pump P-3401 during morning shift) — containing maintenance intelligence that the data pipeline needed to capture.

We built a comprehensive AI data pipeline over 8 weeks — connecting 6 data sources into a unified platform. (1) Sensor data pipeline: real-time equipment monitoring. Architecture: SCADA/PI → OPC-UA connector → Apache Kafka (streaming platform) → Apache Flink (stream processing) → TimescaleDB (time-series storage) → ML inference engine. Data flow: 45,000 sensor data points/second ingested into Kafka topics (partitioned by equipment unit). Processing: Flink performing real-time calculations — rolling averages, rate of change, frequency analysis (FFT for vibration), and statistical anomaly detection. Storage: TimescaleDB storing processed sensor data with 1-second resolution — 90 days hot storage, 5 years compressed cold storage. Volume: processing 3.9 billion data points daily — pipeline designed for 5x headroom (19.5 billion points/day). Latency: sensor reading to processed metric in under 3 seconds — enabling real-time condition monitoring. (2) Maintenance data pipeline: historical context. Sources: SAP PM (work orders, equipment master), inspection system (inspection records, measurements), and document management (procedures, manuals). Extraction: CDC (Change Data Capture) from SAP — real-time work order updates flowing to the data lake. Inspection: API integration — inspection records synchronised daily. Documents: document processing pipeline — PDF extraction, Arabic OCR for scanned documents, and text embedding for searchable document content. Transformation: unified equipment model — each equipment unit having a complete profile combining real-time sensor data, maintenance history, inspection records, and reference documents. Arabic processing: Arabic maintenance records processed through Arabic NLP pipeline — entity extraction (equipment IDs, failure descriptions, actions taken), sentiment/urgency classification, and structured data generation from unstructured Arabic text. (3) Feature engineering: ML-ready data. Features: creating predictive features from raw data — time-domain features (mean, standard deviation, peak, RMS for each sensor over multiple time windows), frequency-domain features (FFT components, dominant frequency, harmonic ratios for vibration analysis), degradation features (trend analysis, rate of change, deviation from baseline), operational features (operating hours since last maintenance, load profile, start-stop cycles), and maintenance features (days since last maintenance, maintenance type, historical failure frequency). Feature store: centralised feature store — pre-computed features available for ML model training and real-time inference. Arabic features: Arabic text features — maintenance event descriptions vectorised, operator observation sentiment, and historical Arabic maintenance intelligence accessible to models. Pipeline schedule: features recalculated hourly (batch) with real-time features updated continuously (streaming). (4) ML serving pipeline: prediction delivery. Model training: training pipeline — feature store data extracted monthly, models retrained on latest data, model performance validated against holdout set, and champion model promoted to production. Model types: equipment-specific models — each critical equipment unit having a tailored predictive model trained on its specific sensor patterns and failure history. Inference: real-time inference pipeline — latest features fed to model every hour → failure probability calculated → ranked by severity → maintenance recommendations generated. Output: prediction dashboard — equipment health scores (0-100), failure probability (next 7/14/30 days), recommended maintenance actions, and confidence levels. Alerting: automated alerts when failure probability exceeds threshold — maintenance planner notified with equipment details, predicted failure mode, and recommended action. Integration: predictions pushed to SAP PM — creating maintenance notifications and suggested work orders directly in the maintenance planning system. (5) Dashboard and reporting: unified view. Operational dashboard: real-time facility health — all 280+ equipment units displayed with current condition, trend, and prediction status. Equipment detail: drill-down showing sensor trends, maintenance history, inspection records, Arabic operator notes, and prediction details. Management reporting: weekly equipment health report — predicted maintenance needs, budget forecast, and spare parts requirements. Arabic reporting: reports generated in bilingual Arabic/English — executive summaries in Arabic, technical details in English (matching the facility's communication conventions).

Results over 12 months. Prediction accuracy: predictive models achieving 87% accuracy in identifying equipment failures 2+ weeks before occurrence (from 0% — no predictive capability previously). False positive rate: 12% — acceptable for maintenance planning (better to inspect unnecessarily than miss a failure). Detected failures: 38 equipment issues predicted and addressed before failure — each a potential unplanned shutdown. Downtime reduction: unplanned downtime from 14 incidents/year to 4 incidents/year (71% reduction). Average downtime: from 18 hours to 8 hours (56% reduction — predicted issues resolved faster because pre-planned). Production value: downtime reduction saving approximately QAR 620M annually (from QAR 882M in unplanned downtime costs). Maintenance efficiency: reactive work from 65% to 25% of maintenance activity. Planned work: from 35% to 75% — maintenance team now predominantly proactive. Work order accuracy: predicted maintenance targeting the right equipment, right component, right action — 92% of predicted work orders confirmed as needed upon inspection. Spare parts: predictive maintenance enabling parts ordering 2-4 weeks ahead — emergency procurement reduced by 60%. Arabic data: 12,000+ Arabic maintenance records processed and accessible — operator observations now contributing to predictive models. Arabic intelligence: Arabic operator logs containing early indicators of issues — "abnormal vibration" and "unusual noise" observations now captured and correlated with sensor patterns. Data pipeline performance: processing 3.9 billion data points daily with 99.97% availability. Latency: sensor-to-insight in under 3 seconds (real-time monitoring) and under 1 hour (predictive scoring). Data quality: 99.4% data completeness — missing sensor data detected and flagged within 30 seconds. Arabic processing: Arabic maintenance records processed with 94% entity extraction accuracy. Financial: annual value QAR 620M (production losses avoided). Pipeline development: QAR 1.8M (8-week project). Annual operating cost: QAR 480,000 (cloud infrastructure, ML compute, and pipeline maintenance). ROI: 344x return — the pipeline development cost recovered in less than 2 days of avoided downtime. Strategic: the data pipeline infrastructure enabling future AI applications — energy optimisation, emissions prediction, and supply chain intelligence — all building on the same data foundation.