[ Ссылка ] | Implementing an event-driven data pipeline can be challenging, but doing so within the context of a legacy architecture is even more complex. Having spent three years building a streaming data infrastructure and being on the first team at a financial organization to implement Apache Kafka® event-driven data pipelines, Danica Fine (Senior Developer Advocate, Confluent) shares about the development process and how ksqlDB and Kafka Connect became instrumental to the implementation.
By moving away from legacy batch processing methods to real-time, streaming data pipelines with Kafka, data can be distributed with increased data scalability and resiliency. Kafka decouples the source from the target systems, so you can react to data as it changes while ensuring accurate data in the target system.
In order to transition from monolithic micro-batching applications to real-time microservices that can integrate with a legacy system that has been around for decades, Danica and her team started developing Kafka connectors to connect to various sources and target systems.
1. Kafka connectors: Building two major connectors for the data pipeline, including a source connector to connect the legacy data source to stream data into Kafka, and another target connector to pipe data from Kafka back into the legacy architecture.
2. Algorithm: Implementing Kafka Streams applications to migrate data from a monolithic architecture to a stream processing architecture.
3. Data join: Leveraging Kafka Connect and the JDBC source connector to bring in all data streams to complete the pipeline.
4. Streams join: Using ksqlDB to join streams—the legacy data system continues to produce streams while the Kafka data pipeline is another stream of data.
As a final tip, Danica suggests breaking algorithms into process steps. She also describes how her experience relates to the data pipelines course on Confluent Developer and encourages anyone who is interested in learning more to check it out.
EPISODE LINKS
► Data Pipelines course: [ Ссылка ]
► Guided Exercise on Building Streaming Data Pipelines: [ Ссылка ]
► Migrating from a Legacy System to Kafka Streams: [ Ссылка ]
► Join the Confluent Community: [ Ссылка ]
► Learn Apache Kafka on Confluent Developer: [ Ссылка ]
► Live demo: Intro to Event-Driven Microservices with Confluent: [ Ссылка ]
► Use PODCAST100 to get $100 of free Confluent Cloud usage: [ Ссылка ]
► Promo code details: [ Ссылка ]
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#datapipeline #apachekafka #kafka #confluent
ABOUT CONFLUENT
Confluent is pioneering a fundamentally new category of data infrastructure focused on data in motion. Confluent’s cloud-native offering is the foundational platform for data in motion – designed to be the intelligent connective tissue enabling real-time data, from multiple sources, to constantly stream across the organization. With Confluent, organizations can meet the new business imperative of delivering rich, digital front-end customer experiences and transitioning to sophisticated, real-time, software-driven backend operations. To learn more, please visit www.confluent.io.
#datapipeline #apachekafka #kafka #confluent
