Running into issues with your data pipeline? Join us on March 15th for a live session on troubleshooting Apache Beam issues in Dataflow, where we'll:
- Provide an overview of running Apache Beam pipelines on Dataflow
- Cover common challenges you might face along the way
- Demonstrate troubleshooting and debugging tips to get you back on track.
02:40 Situation: A meta template where source and sink can be configured
09:35 Building pipelines workflow recommendations
12:51 Developing integration tests
17:19 Troubleshooting scenarios and tips
19:50 Scenario 1: low throughput when writing out
24:57 Scenario 2: How does the autoscaler work
29:27 Scenario 3: Straggler workers (job takes a long time to finish)
31:41 Scenario 4: Work is not making progress (garbage collector thrashing)
34:47 Scenario 5: Hard to explain why it's slow/not sure the issue
37:02 Managed data pipelines
42:00 Q&A
Ask questions and find answers in the Data Analytics Community: [ Ссылка ]
Share your feedback and ideas for future sessions: [ Ссылка ]
DataflowTemplates integration test framework: [ Ссылка ]
Dataflow Prime (vertical autoscaling): [ Ссылка ]
DataflowTemplates: [ Ссылка ]
