#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault
This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project in 3 hours. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.
The tools that are covered in this project are,
1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI
The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.
Chapters
TimeStamp:
0:00:00 - Project Overview
0:13:40 - Environment Setup
0:22:34 - Data Ingestion using ADF (Part 1)
0:45:48 - Data Ingestion using ADF (Part 2)
1:07:36 - Data Transformation using Databricks (Part 1)
1:22:57 - Data Transformation using Databricks (Part 2)
1:43:46 - Data Transformation using Databricks (Part 3)
1:58:59 - Data Loading using Azure Synapse Analytics
2:22:22 - Data Reporting using Microsoft Power BI
2:36:59 - Creating Security Group using Azure Active Directory (AAD)
2:44:00 - End to End Pipeline Testing
Please like, share and subscribe if you like the content and leave your comments below.
For contact,
Email: mrktalkstech@gmail.com
Instagram: mrk_talkstech_tamil
