DEV Community

Cover image for I Love Debezium Server. Operating It Was Another Story, So I Built BeziSync
daun-gatal
daun-gatal

Posted on Edited on Originally published at Medium

I Love Debezium Server. Operating It Was Another Story, So I Built BeziSync

TL;DR: Debezium Server is a great way to do change data capture without running Kafka Connect, but operating it in production is painful. I built BeziSync, a web UI and control plane that checks your database before you deploy, protects the source, turns operations into buttons, and only shows green when there's proof. There's a playground you can sign in to with Google.


The first time I got Debezium Server running, I was honestly a bit surprised. One container, one properties file, and rows from Postgres were showing up in Kafka a few seconds after I inserted them. No Kafka Connect cluster to babysit. No plugin folders. It just worked.

The second time was less fun. A pipeline had been stopped for a weekend, nobody noticed, and on Monday the replication slot was holding a lot more WAL than anyone was comfortable with. The disk alert on the database got to us before any alert about the pipeline did.

That gap between "it works on my laptop" and "I trust this in production" is what this post is about. It's also why I ended up building BeziSync.

Debezium Server, quickly

If you haven't used it: Debezium reads the transaction log of your database (the WAL in Postgres, the binlog in MySQL, the oplog in MongoDB) and turns every insert, update and delete into an event. That's change data capture, or CDC.

Most people meet Debezium as a set of Kafka Connect plugins. Debezium Server is the other way to run it. It's a standalone app that runs one connector and sends events straight to a sink: Kafka, Pub/Sub, Kinesis, Redis Streams, NATS, HTTP, and more.

A minimal setup looks something like this:

debezium.source.connector.class=io.debezium.connector.postgresql.PostgresConnector
debezium.source.database.hostname=shop-db
debezium.source.database.user=debezium
debezium.source.database.password=...
debezium.source.plugin.name=pgoutput
debezium.source.topic.prefix=shop
debezium.source.table.include.list=public.orders
debezium.sink.type=kafka
debezium.sink.kafka.producer.bootstrap.servers=kafka:9092
Enter fullscreen mode Exit fullscreen mode

For a lot of teams that's a much better deal. You don't need Kafka just to move data from Postgres into a lakehouse. You run one small process per pipeline and you're done.

At least, that's how it feels on day one.

Where it starts to hurt

Debezium Server is simple to start. It is not simple to operate. After running it for a while, these were the things that kept coming back:

  • The database has to be set up just right. wal_level=logical, a user with replication rights, a publication, the right binlog format, REPLICA IDENTITY on tables without keys. Every cloud does it differently. RDS has parameter groups, Cloud SQL has flags, Azure has its own thing. You usually find out what's missing when the connector crashes.
  • A stopped pipeline can hurt the source. A Postgres replication slot keeps WAL until the consumer reads it. If the connector is down, that WAL piles up. Leave it long enough and you're in trouble.
  • Everything is a properties file. Snapshots, offsets, transforms, sink settings. All of it is debezium.source.* and debezium.sink.* keys. One typo and you're reading Java stack traces.
  • "Running" doesn't mean "working". The container is up, the health endpoint says UP, and no events have arrived for an hour. Which one do you believe?
  • Simple operations aren't simple. Re-snapshot one table. Reset the offset. Roll back yesterday's config change. Each one is a small research project.
  • It doesn't scale to people. In practice one or two engineers end up as "the Debezium people", and everyone else files a ticket and waits.

None of this is Debezium's fault, really. It's a great engine. What's missing is the layer around it.

So I built BeziSync

BeziSync is a web UI and control plane for Debezium Server. The idea is easy to say: someone who knows the platform sets things up once, and after that any engineer can create and run CDC pipelines without becoming a Debezium expert first.

An admin connects a runtime (a Kubernetes namespace or a Docker host) and sets the guardrails. Then anyone on the team can go through a short wizard: pick a source database, pick the tables, pick a destination, review, deploy. BeziSync writes the application.properties, starts one Debezium Server per pipeline, and keeps an eye on it from then on.

Right now it works with PostgreSQL, MySQL/MariaDB, SQL Server and MongoDB as sources. For destinations: Kafka, Apache Iceberg (REST catalogs, AWS Glue, S3 Tables), JDBC databases, Pub/Sub, Kinesis, SQS, SNS, Event Hubs, Pulsar, Redis Streams, NATS and HTTP.

You can still see the generated config at any point. I didn't want to hide Debezium. I just didn't want everyone to have to type it by hand.

What it does about each of those problems

Checks before anything deploys

When you add a source, BeziSync checks it the way Debezium will use it: logical replication, privileges, retention, primary keys, TLS, network access (including through an SSH bastion). There are more than 150 checks right now. When one fails, it doesn't just say "failed". It shows the exact fix for where your database runs, whether that's self-hosted, RDS, Cloud SQL or Azure. There's even a "send to my DBA" message you can copy, because often you're not the one who can change wal_level.

The same goes for the pipeline itself. If a transform would make Debezium stop with a duplicate field name, the deploy is blocked before it happens. You don't find out from the logs.

Protecting the source database

Every pipeline page shows what the pipeline is costing the source, in that database's own terms. For Postgres, that's the replication slot, how much WAL it's holding, and how much room is left before max_slot_wal_keep_size drops it. For MySQL, the binlog file being read and whether the next purge would remove it. For MongoDB, the oplog window. For SQL Server, the CDC retention.

There's a default limit on how much log a pipeline may hold, an alert when it gets close, and maintenance windows so big snapshots run at night instead of at peak time. That's the weekend story from the start of this post, handled.

Operations are buttons

Pause, resume, restart. Re-snapshot a few tables while streaming keeps going. See the stored offset, or reset it, with a typed confirmation. Every saved change is a version, so rolling back is choosing an older version and deploying it. Before you confirm, BeziSync tells you what will happen: whether the connector restarts, what changes for consumers, and whether someone needs to approve it.

Health you can believe

This one matters to me most. A pipeline is green only when there's fresh evidence that it's working. If BeziSync can't measure something, it says Unknown and shows it in grey, never green. There's also an optional delivery canary: every few minutes it writes a tiny row on the source, looks for it at the destination, and tells you "verified 2 minutes ago, 4 s end to end". If the row doesn't show up, you get an alert. That's how you catch "the container is up but nothing is moving".

A pipeline's Overview page in BeziSync: source protection, throughput and a verified delivery canary

It also watches the schema of the tables you capture. A dropped column or a type change is flagged as breaking before your consumers notice.

Safe for a whole team

Secrets are encrypted and write-only. Once you save a password, nobody can read it back through the UI or the API. There are roles per project (viewer, operator, editor), approvals for production changes, policies, an audit log, SSO, and alerts to Slack, PagerDuty, webhooks or email. If you prefer code over clicking, pipelines can be exported as YAML and managed from a CLI in CI.

How it fits together

Under the hood it's still plain Debezium Server. BeziSync doesn't fork it or patch it. It just runs it for you, one instance per pipeline.

How BeziSync fits around Debezium Server

The controller deploys each pipeline to a runtime you've connected: a Kubernetes namespace, a Docker host, or a remote cluster through a small agent that only makes outbound connections, so you don't have to open your network. Offsets and schema history live in Postgres, so a pipeline can move or restart without losing its place. There's no extra queue or broker to run either.

Try it and tell me what's wrong

BeziSync is still early, and I'd rather shape it with real users than guess. There's a playground you can sign in to with a Google account: bezisync.dglabs-ai.dev. It already has sample databases with live traffic, Kafka and other sinks, and runtimes on both Kubernetes and Docker. You can build a pipeline end to end in a few minutes, break things, and see what BeziSync tells you.

What I'd love to hear:

  • Where you got stuck, or where the UI said something that didn't make sense.
  • Which source or destination you'd need that isn't there.
  • Whether your team would actually use something like this, and what would stop you.

There's a feedback button inside the app, or drop a comment below. And if you've been the "Debezium person" on your team, I'd really like to hear your war stories.

Top comments (0)