← All projects
European Data Ingestion Platform
ETL and ingestion platform consolidating European data sources for analytics, reporting and operational use.
Context
The problem
Data from multiple European sources had to be ingested, transformed, validated and made available reliably across cloud analytics workflows.
Ownership
My contribution
- Rapidly transitioned from C#-based migration work into Python data engineering and orchestration responsibilities.
- Analyzed requirements with data analysts and proposed practical implementation options.
- Built ingestion and transformation workflows using Python, Airflow and Azure Data Factory.
- Integrated third-party APIs and data movement tools with Snowflake and Azure services.
- Implemented validation, monitoring and operational support patterns.
Impact
Outcomes
- Received the DART Innovation Award for an Azure Data Factory solution.
- Improved reliability and repeatability of data ingestion.
- Earned strong recognition from business stakeholders and analysts.
Design
Architecture themes
ETL pipelinesBatch orchestrationCloud data warehouseAPI ingestionData quality controls