This collection showcases end-to-end data engineering solutions and advanced analytics implementations. Below, you’ll find highlights of my work with Azure tools.
An end-to-end solution leveraging the Medallion Architecture to ingest, process, and analyze data using Azure Data Factory (ADF) and Azure Databricks.
Key Features:
- Staged data transformations across Bronze, Silver, and Gold layers.
- Automated pipelines for data ingestion, transformation, and load.
- Integration with Power BI for seamless analytics.
A demonstration of serverless analytics with Azure Synapse Analytics, showcasing advanced SQL features and seamless integration with Power BI for insights.
Key Features:
- Implemented CETAS (Create External Tables As Select) and Incremental Load Design.
- Utilized Change Data Capture (CDC) for real-time updates.
- Demonstrated SQL performance monitoring and query optimization.
Multiple pipelines developed to demonstrate below functions
- Data Ingestion Pipeline: Automated ingestion of structured and unstructured data into Azure Data Lake.
- Transformations Pipeline: ETL workflows built for scalable data processing.
- Orchestration Pipeline: Dependencies managed using pipeline chaining, conditional execution, and alerts for monitoring.