Our Lisbon AI data pipeline development follows a methodology designed for GDPR-compliant, production-grade data infrastructure: (1) Data assessment (weeks 1-2): understanding the data landscape. Source inventory: cataloguing all data sources — databases (PostgreSQL, MySQL, Oracle, SQL Server), SaaS platforms (Salesforce, HubSpot, Shopify, Google Analytics), files (CSVs, Excel, JSON), APIs (internal and external), and streaming sources (Kafka, IoT sensors, event streams). Portuguese data sources: Portuguese-specific systems — Multibanco/SIBS data, AT (Autoridade Tributária) tax data, Portuguese social security (Segurança Social), and Portuguese banking APIs (PSD2/Open Banking). Data quality: assessing data quality across sources — completeness (missing values), consistency (conflicting data between sources), accuracy (verified against known truths), timeliness (how fresh the data is), and relevance (whether the data actually serves the AI use case). GDPR audit: mapping personal data across all sources — what personal data exists, under what legal basis it's processed, how long it's retained, and whether consent covers the intended AI use. The GDPR audit: determining pipeline design constraints — anonymisation requirements, retention policies, and consent enforcement. AI readiness: assessing whether the data can support the intended AI use — sufficient volume, representative coverage, appropriate labels (for supervised learning), and temporal coverage (enough history for time-series models). (2) Pipeline architecture (weeks 2-3): designing the data infrastructure. Architecture selection: choosing the right pipeline approach. Batch processing (scheduled data processing — hourly, daily, or weekly. Best for: analytics, reporting, and model training where real-time isn't required. Technologies: Apache Spark, dbt, Airflow), stream processing (real-time data processing — events processed as they occur. Best for: fraud detection, real-time recommendations, and operational monitoring. Technologies: Apache Kafka, Apache Flink, AWS Kinesis), and hybrid (batch for historical analysis and model training, streaming for real-time serving — the most common pattern for AI data pipelines). Data platform: selecting the infrastructure. Cloud (our default for Lisbon businesses — AWS eu-south-2 (Spain, nearest to Lisbon), Azure West Europe, or GCP europe-west1. GDPR: EU-region deployment ensuring data remains within the EU), on-premises (for highly regulated businesses requiring on-site data processing — Portuguese banks and healthcare organisations with specific data residency requirements), and hybrid (sensitive data on-premises, processed results in cloud — balancing security with scalability). Data modelling: designing the data structures — raw layer (data landed as-is from sources), staging layer (cleaned, validated, and standardised data), curated layer (business-ready data, modelled for analysis), and feature store (ML-ready features, computed and cached for model training and serving). GDPR design: privacy by design — PII identified and tagged, access controls at the column level, anonymisation applied in the appropriate layer, and deletion capability throughout the pipeline. (3) Pipeline development (weeks 3-5): building the data infrastructure. Ingestion: building data connectors for each source. Database connectors (CDC — Change Data Capture — for real-time database synchronisation. Debezium for PostgreSQL/MySQL, Oracle GoldenGate for Oracle), API connectors (scheduled API calls for SaaS platforms — handling pagination, rate limiting, and error recovery), file ingestion (SFTP, S3, and cloud storage monitoring for file-based data sources), and streaming (Kafka producers for real-time event capture — application events, IoT sensors, and transactional systems). Transformation: data cleaning and preparation. dbt (our default transformation tool — SQL-based transformations with testing, documentation, and lineage. dbt: enabling analysts and data engineers to collaborate on transformation logic), PySpark (for complex transformations requiring Python — ML feature engineering, NLP preprocessing, and custom business logic), and Great Expectations (data quality testing — validating data at each pipeline stage, catching issues before they affect downstream models). Portuguese data transformations: NIF (Número de Identificação Fiscal) validation, Portuguese address standardisation, European Portuguese NLP preprocessing, and Portuguese date/number formatting (DD/MM/YYYY, comma as decimal separator). Feature engineering: computing ML features from raw data. Feature store (Feast or custom — computing, storing, and serving ML features. Online store: for real-time model serving. Offline store: for model training), feature computation (aggregations, ratios, time-series features, text embeddings, and categorical encodings — computed from curated data and stored in the feature store), and feature versioning (tracking feature definitions, enabling reproducibility and feature reuse across models). (4) GDPR compliance layer (weeks 4-5): embedding privacy into the pipeline. Data classification: automated PII detection and classification — names, email addresses, phone numbers, NIFs, and other personal data tagged throughout the pipeline. Access control: column-level security — PII accessible only to authorised roles. Anonymised views: for analytics and ML that don't require personal data. Anonymisation: k-anonymity, l-diversity, or differential privacy applied where data is used for analytics or ML training. Portuguese GDPR: ensuring anonymisation meets CNPD expectations — true anonymisation, not pseudonymisation disguised as anonymisation. Right to deletion: data deletion capability across the entire pipeline — from raw storage through transformed layers to feature stores and model training data. Deletion verification: confirming data removal across all pipeline stages. Consent tracking: consent metadata flowing through the pipeline — data processing restricted to consented purposes. Consent withdrawal: triggering automatic data restriction across the pipeline. Audit logging: comprehensive logging of all data processing — who accessed what, when, and for what purpose. GDPR accountability: logs enabling demonstration of compliance to CNPD. (5) Monitoring and operations (weeks 5-6+): keeping the pipeline healthy. Pipeline monitoring: real-time tracking of pipeline health — job success/failure, processing latency, data volume, and error rates. Alerting: automated alerts for pipeline failures, data quality issues, and anomalies. Data quality monitoring: continuous data quality checks — schema drift detection, distribution drift detection, and freshness monitoring. Model drift: tracking input data distribution changes that might affect model performance — alerting data science teams when feature distributions shift. Cost management: cloud cost monitoring — tracking compute and storage costs, identifying optimisation opportunities, and right-sizing infrastructure. Portuguese cost sensitivity: Lisbon businesses often operating on tighter budgets than London or San Francisco — cost efficiency critical in pipeline design.