Log Analytics provides a way to easily query logs and setup alerts in Azure. This provides a huge help when monitoring Apache Spark. In this video I walk through setting up Azure Synapse Apache Spark to connect to Log Analytics, adding custom log messages from PySpark, and how to query logs from Spark log4j using Azure Log Analytics.
Written tutorial and troubleshooting steps:
[ Ссылка ]
More from Dustin:
Website: dustinvannoy.com
Twitter: @dustinvannoy
Github: [ Ссылка ]
Outline:
0:00 Intro
0:42 Setup Log Analytics
1:14 Key vault secret
1:52 Configure Spark Pool
4:00 PySpark logging
5:05 Log workspace query
