Welcome to the 23rd episode of Synapse Espresso!
In this video, Yennifer & Stijn share how to overcome automation challenges when working with a large volume of Delta Lake files by using Serverless Pools and Synapse pipelines.
The smart ingestion is driven by a JSON configuration file, that copies or updates only new or updated data as defined in the configuration file and creates the partitioned views mapped to those same files.
Previous versions of the delta files and views are dropped and recreated just if changes happen in the source system.
Finally, ACLs permissions at the Data Lake level and Data retention type of requirements are also demoed and included in the sample. These learnings and more are publicly available on the Modern Datawarehouse repository in GitHub and the content is always being extended and improved.
GitHub Repository: [ Ссылка ]
Stijn Wynants - Fasttrack Engineer
[ Ссылка ]
[ Ссылка ]
[ Ссылка ]
Yennifer Santos - Principal Software Engineer - Industry Solutions Engineering
[ Ссылка ]
