DEV Community

Cover image for How to set up a CDC pipeline to capture and analyze real-time database changes with Parseable
Jen Wike Huger for Parseable

Posted on

3 2

How to set up a CDC pipeline to capture and analyze real-time database changes with Parseable

Databases are critical for any application. Data constantly gets updated, inserted, and deleted. In most of cases it is important for the business to keep track of these changes due to security concerns, auditing requirements, and to keep other relevant systems up to date.

Change Data Capture (CDC) has gained popularity, precisely to address this problem. CDC is a technique used to track all changes in a database and capture them in destination systems. Debezium is a popular CDC tool that leverages database logs as the source of truth, and streams the changes to Kafka and compatible systems like Redpanda.

The key thing to note here is that while databases are typically mutable, the changes captured by the CDC system should be immutable. Only then users have a system of record that can be trusted. Parseable is built for immutable logs and streams of events and data. It is a great fit for CDC targets. The way we designed Parseable allows for long term retention and querying of logs, and it also provides a rich set of tools for analysis and monitoring.

In this tutorial, we use Redpanda, PostgreSQL, and Debezium to ingest CDC events into Parseable.

CDC pipeline with Redpanda, PostgreSQL, Debezium, and Parseable

Read more in the full tutorial to:

  • Set up Docker Compose
  • Set up Debezium Connect
  • Set up Parseable
  • Set up Redpanda Connect
  • Test the CDC pipeline

Top comments (1)

Collapse
 
martinbaun profile image
Martin Baun

Great work compiling this, Jen. About to check out the full tutorial!

👋 Kindness is contagious

Discover a treasure trove of wisdom within this insightful piece, highly respected in the nurturing DEV Community enviroment. Developers, whether novice or expert, are encouraged to participate and add to our shared knowledge basin.

A simple "thank you" can illuminate someone's day. Express your appreciation in the comments section!

On DEV, sharing ideas smoothens our journey and strengthens our community ties. Learn something useful? Offering a quick thanks to the author is deeply appreciated.

Okay