loading...

Kafka Connect JDBC Sink: tips & tricks - video walkthrough

rmoff profile image Robin Moffatt ・1 min read

Kafka Connect examples (6 Part Series)

1) Streaming data from Kafka to S3 - video walkthrough 2) Streaming data from Kafka to a Database - video walkthrough 3 ... 4 3) Kafka Connect JDBC Sink: tips & tricks - video walkthrough 4) Kafka Connect JDBC connector: installing a JDBC driver 5) Streaming data from Kafka to Elasticsearch - video walkthrough 6) Loading CSV data into Kafka - video walkthrough

Using ksqlDB you can enrich streams of data, and write the resulting Kafka topic to a database. You can also build stateful aggregations and write these to a database.

🎥 https://rmoff.dev/ksqldb-jdbc-sink-video

Alt Text

Check out this tutorial for a walkthrough of how to do this and learn tricks such as:

  • Troubleshooting Kafka Connect JDBC sink
  • Un-nesting data for loading into the database (solves the error "(STRUCT) type doesn't have a mapping to the SQL database column type")
  • Handling timestamp fields
  • Idempotent database updates
  • Update rows in place based on a declared key

Whilst the tutorial is built around ksqlDB, the tips & tricks for the JDBC sink apply equally to Kafka Connect used on its own.

You can find the Docker Compose and notes on how to try this out for yourself here: https://rmoff.dev/ksqldb-jdbc-sink

Kafka Connect examples (6 Part Series)

1) Streaming data from Kafka to S3 - video walkthrough 2) Streaming data from Kafka to a Database - video walkthrough 3 ... 4 3) Kafka Connect JDBC Sink: tips & tricks - video walkthrough 4) Kafka Connect JDBC connector: installing a JDBC driver 5) Streaming data from Kafka to Elasticsearch - video walkthrough 6) Loading CSV data into Kafka - video walkthrough

Posted on by:

rmoff profile

Robin Moffatt

@rmoff

Robin Moffatt is a Developer Advocate at Confluent, and regular conference speaker. He also likes writing about himself in the third person, eating good breakfasts, and drinking good beer.

Discussion

markdown guide