#azuredataengineer #endtoendproject #azuredataengineeringproject #azureintamil #azuredatafactory #azuredatabricks #azuresynapseanalytics #azuredatalake #datalake #powerbi #keyvault


This is a long awaited video of mine- Lets build a complete End to End Azure Data Engineering Project. In this project we are going to create an end to end data platform right from Data Ingestion, Data Transformation, Data Loading and Reporting.

The tools that are covered in this project are,

1. Azure Data Factory
2. Azure Data Lake Storage Gen2
3. Azure Databricks
4. Azure Synapse Analytics
5. Azure Key vault
6. Azure Active Directory (AAD) and
7. Microsoft Power BI

The use case for this project is building an end to end solution by ingesting the tables from on-premise SQL Server database using Azure Data Factory and then store the data in Azure Data Lake. Then Azure databricks is used to transform the RAW data to the most cleanest form of data and then we are using Azure Synapse Analytics to load the clean data and finally using Microsoft Power BI to integrate with Azure synapse analytics to build an interactive dashboard. Also, we are using Azure Active Directory (AAD) and Azure Key Vault for the monitoring and governance purpose.

Follow the below link to import the database which is used in this project to the SSMS (I used the light weight version)

[ Ссылка ]

Chapters

TimeStamp:

0:00:00 - Project Overview
0:11:02 - Environment Setup
0:19:40 - Data Ingestion using ADF (Part 1)
0:41:31 - Data Ingestion using ADF (Part 2)
1:01:42 - Data Transformation using Databricks (Part 1)
1:16:22 - Data Transformation using Databricks (Part 2)
1:36:26 - Data Transformation using Databricks (Part 3)
1:51:58 - Data Loading using Azure Synapse Analytics
2:15:34 - Data Reporting using Microsoft Power BI
2:30:35 - Creating Security Group using Azure Active Directory (AAD)
2:36:44 - End to End Pipeline Testing

– – – Book a Private One on One Meeting with me (1 Hour) – – –

[ Ссылка ]

– – – Express your encouragement by brewing up a cup of support for me – – –

[ Ссылка ]

– – – Other useful playlist: – – –

Azure Data Factory Playlist: [ Ссылка ]

Azure General Topics Playlist: [ Ссылка ]

Microsoft Fabric Playlist: [ Ссылка ]

Azure Databricks Playlist: [ Ссылка ]

Azure End to End Project Playlist: [ Ссылка ]

Databricks CICD Playlist: [ Ссылка ]

End to End Azure Data Engineering Project: [ Ссылка ]


– – – Let’s Connect: – – –

Email: mrktalkstech@gmail.com
Instagram: mrk_talkstech


– – – Tools & Equipment (Gears I use): – – –

Disclaimer: Links included in this description might be affiliate links. If you purchase a product or service with the links that I provide, I may receive a small commission. There is no additional charge to you! Thank you for supporting me so I can continue to provide you with free content each week!

DJI Mic: [ Ссылка ]
Dell XPS 13 Plus 13.4" 3.5K : [ Ссылка ]
Rode VideoMicro Vlogger Kit: [ Ссылка ]
DJI Osmos Action 3: [ Ссылка ]
DJI Mini 3 PRO: [ Ссылка ]


– – – About me: – – –

Mr. K is a passionate teacher created this channel for only one goal "TO HELP PEOPLE LEARN ABOUT THE MODERN DATA PLATFORM SOLUTIONS USING CLOUD TECHNOLOGIES"

I will be creating playlist which covers the below topics (with DEMO)

1. Azure Beginner Tutorials
2. Azure Data Factory
3. Azure Synapse Analytics
4. Azure Databricks
5. Microsoft Power BI
6. Azure Data Lake Gen2
7. Azure DevOps
8. GitHub (and several other topics)

After creating some basic foundational videos, I will be creating some of the videos with the real time scenarios / use case specific to the three common Data Fields,

1. Data Engineer
2. Data Analyst
3. Data Scientist

Can't wait to help people with my videos.

– – – Support me: – – –

Please Subscribe: [ Ссылка ]

свернуть