Ingesting data from a CSV file into a Kafka topic can be done easily using Kafka Connect. Kafka Connect is part of Apache Kafka, and only requires a JSON file to configure - no coding!

This video shows you how to stream CSV files into Kafka, how to manage the schema, and as a bonus also shows streaming the same data out to a database such as Postgres, as well as processing it with ksqlDB.

To learn more about Kafka Connect see [ Ссылка ]

Code used in this video: [ Ссылка ]
--

ℹ️ Table of contents

00:00:44 Brief introduction to Kafka Connect
00:01:39 Checking that the correct Kafka Connect plugin is installed
00:02:46 Kafka, Bytes, and Schemas
00:05:06 Creating a connector to ingest data from CSV file and set the schema based on header row
00:09:43 File metadata stored in the Kafka message header
00:10:18 Setting the message key
00:14:35 Manipulating the schema / changing data types
00:19:22 Ingesting raw CSV into a Kafka topic without a Schema
00:22:43 Streaming data from CSV into Kafka into a Database
00:28:27 Updating database rows in-place from CSV file ingest
00:28:42 Impromptu Kafka Connect troubleshooting ;-)
00:32:34 Filtering and aggregating CSV data in Kafka using ksqlDB

--

💾 Download Kafka Connect Spooldir plugin: [ Ссылка ]

🎈 Kafka Connect Spooldir docs: [ Ссылка ]
🎈 Single Message Transforms (SMT): [ Ссылка ]
🎈 Cast SMT: [ Ссылка ]
🎈 Kafka Connect JDBC Source connector: [ Ссылка ]
🎈 Kafka Connect JDBC Sink connector docs: [ Ссылка ]

🎥 Installing JDBC Driver for Kafka Connect: [ Ссылка ]
🎥 Streaming data from Kafka to a database: [ Ссылка ]
🎥 JDBC sink and schemas: [ Ссылка ]
🎥 Introduction to ksqlDB: [ Ссылка ]

--

☁️ Confluent Cloud ☁️
Confluent Cloud is a managed Apache Kafka and Confluent Platform service. It scales to zero and lets you get started with Apache Kafka at the click of a mouse. You can signup at [ Ссылка ] and use code 60DEVADV for $60 towards your bill (small print: [ Ссылка ])

свернуть