Debezium Postgres to Kafka: Every Setting Explained

About fifteen lines of JSON. That's all it takes to stream every insert, update and delete from Postgres into Kafka with Debezium. And two of the defaults hiding in those lines will hurt you in production if you leave them alone.

This is Part 2 of CDC in Production. Part 1 explained what change data capture (CDC) is. This post opens up the most common open-source setup, Debezium in Kafka Connect reading Postgres: what to prepare in the database, what every setting does, what happens on first start, and what lands in each Kafka message.

Prefer to watch? Here's the 9-minute video version with animated diagrams:

Video link: Debezium Postgres to Kafka: Every Setting Explained

The moving parts

Read the pipeline left to right:

Postgres (WAL) -> replication slot + publication -> Debezium in Kafka Connect -> Kafka topics -> your sinks

• Write-ahead log (WAL): Postgres writes every change here before it counts as committed. It's the database's own ledger.
• Logical replication slot: a named bookmark in the WAL. It's the database's promise to keep the log from that point on until a reader confirms it has processed it.
• Publication: the list of tables whose changes Postgres decodes for logical replication.
• Kafka Connect: a framework for running connectors that move data in and out of Kafka. Debezium is a source connector inside a Connect worker. It reads the decoded changes, turns each one into an event and writes it to a Kafka topic.

Kafka Connect also stores the connector's position (the last log sequence number, or LSN, it handled) in an internal offsets topic. Offset plus slot is how Debezium resumes after a restart. Part 3 is about what happens when they disagree.

Step 1: prepare Postgres

Postgres needs three things before any connector config.

-- postgresql.conf (restart required)
wal_level = logical
-- On Amazon RDS, set rds.logical_replication = 1 in the
-- parameter group instead (static parameter, reboot needed).

-- A dedicated CDC user with only what it needs
CREATE ROLE debezium WITH LOGIN REPLICATION;
GRANT SELECT ON public.orders, public.customers TO debezium;
-- Set its password from psql with: \password debezium
-- On RDS, grant the rds_replication role instead of the REPLICATION attribute:
-- GRANT rds_replication TO debezium;

-- Run as the table owner: capture only the tables you need
CREATE PUBLICATION dbz_shop
  FOR TABLE public.orders, public.customers;

wal_level = logical makes the WAL carry enough row-level detail to decode. The user needs REPLICATION plus SELECT (for the initial snapshot). And the publication lists only the tables you want.

Why create the publication yourself? With pgoutput, Debezium's default publication.autocreate.mode is all_tables. If the publication doesn't exist, it tries to run CREATE PUBLICATION ... FOR ALL TABLES, which needs superuser rights that managed databases rarely hand out. Set publication.autocreate.mode to filtered so Debezium builds it from your table list, or to disabled and own the publication like any other database object. This post uses disabled.

Replica identity: what goes into before

Replica identity is the table setting that decides how much of the old row Postgres writes to the WAL for updates and deletes. With the default, the before image in a Debezium update or delete event carries only primary key columns (and for an update, only when the key itself changed). If downstream consumers need the full old row, switch that table to FULL:

ALTER TABLE public.orders REPLICA IDENTITY FULL;

The trade-off: every update and delete on that table now writes the whole old row to the WAL, so the log grows faster.

Step 2: the connector config, line by line

You register a connector by sending JSON to the Kafka Connect REST API. Here's the whole thing:

{
  "name": "shop-cdc",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "tasks.max": "1",
    "database.hostname": "shop-db.internal",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "${file:/secrets/pg.properties:password}",
    "database.dbname": "shop",
    "topic.prefix": "shop",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_shop",
    "publication.name": "dbz_shop",
    "publication.autocreate.mode": "disabled",
    "table.include.list": "public.orders,public.customers",
    "snapshot.mode": "initial",
    "heartbeat.interval.ms": "10000"
  }
}

Register it and check its status:

curl -X POST http://connect:8083/connectors \
  -H 'Content-Type: application/json' \
  --data @shop-cdc.json

curl http://connect:8083/connectors/shop-cdc/status
# {"name":"shop-cdc","connector":{"state":"RUNNING",...},"tasks":[{"id":0,"state":"RUNNING",...}],...}

To change it later, PUT just the inner config object to /connectors/shop-cdc/config.

What each line does:

• connector.class names the Debezium Postgres connector. tasks.max is 1 because this connector always runs a single task (one slot is read by one reader).
• database.* says where to connect. Don't put the password in plain text. That file: reference in the password line is a Kafka Connect config provider: the worker reads the value from a file at runtime. Enable it in the worker config:

  config.providers=file
  config.providers.file.class=org.apache.kafka.common.config.provider.FileConfigProvider

• topic.prefix becomes the first part of every topic name. The Debezium docs say not to change it later, so pick it carefully.
• plugin.name is the first default that bites. The default is decoderbufs, a logical decoding plugin you'd have to install on the database server yourself. pgoutput is built into Postgres 10 and later, so set it explicitly.
• slot.name and publication.name default to debezium and dbz_publication. Give every connector its own. Two connectors on one slot means changes go missing without an error.
• table.include.list is the allow-list of tables to capture. snapshot.mode is covered below.
• heartbeat.interval.ms is the second default that bites. It's 0 (off) by default. Turn it on. Without heartbeats, a connector whose tables rarely change can stop confirming its position, and the slot holds WAL until the disk fills. Part 6 covers that trap in depth.

Step 3: snapshot, then stream, with no gap

So what happens on the first start?

1. Debezium creates the replication slot if it doesn't exist.
2. It opens a transaction and records the current log position. Call it P.
3. It scans every included table, emitting one event per row with op = r (read).
4. It records in its offsets that the snapshot is complete.
5. It streams changes from P.

That last step is the whole trick. Changes made during the scan wait in the slot after P and are streamed once the snapshot is done. Nothing is skipped. If an interviewer asks how Debezium avoids a gap between snapshot and stream, that's the one-sentence answer.

One caveat: if the connector dies halfway through the initial snapshot, the snapshot starts again from the beginning.

Other snapshot.mode values worth knowing:

Mode	What it does
initial	Snapshot on first start (no offsets), then stream. The default.
always	Snapshot on every start, then stream. Can help after a failover.
no_data	Skip the snapshot; stream only new changes.
initial_only	Snapshot, then stop. No streaming.
when_needed	Snapshot only if there are no offsets or the stored position is no longer available.

For tables too big to scan in one go, there are incremental snapshots: you send a signal (Part 3 shows one), and Debezium copies the table in chunks while streaming keeps running.

Topics, keys and partitions

Each table gets its own topic, named <topic.prefix>.<schema>.<table>. So public.orders becomes shop.public.orders.

The message key is the row's primary key, for example {"id": 1042}. That matters because Kafka sends every message with the same key to the same partition, and a partition is read in the order it was written. So all changes to order 1042 stay in order. Across different keys there's no ordering promise at all. Part 4 builds on exactly that.

You'll also see a heartbeat topic (named __debezium-heartbeat.shop with this prefix) and Kafka Connect's internal topics for configs, offsets and status. Decide partitions, replication factor and retention for your change topics up front, either by creating the topics yourself or with Connect's topic creation settings in the connector config:

"topic.creation.default.partitions": "6",
"topic.creation.default.replication.factor": "3",
"topic.creation.default.retention.ms": "604800000"

Don't let change topics inherit whatever the broker defaults happen to be.

Inside a change event

Say order 1042 moves from paid to shipped. The key is just the primary key. The value is an envelope:

{
  "before": null,
  "after":  { "id": 1042, "status": "shipped", "amount_cents": 12990, "customer_id": 77 },
  "source": {
    "connector": "postgresql",
    "name": "shop",
    "ts_ms": 1792591391000,
    "snapshot": "false",
    "db": "shop",
    "schema": "public",
    "table": "orders",
    "txId": 88231,
    "lsn": 3456120880
  },
  "op": "u",
  "ts_ms": 1792591391412
}

• before is the old row: null here because the table uses the default replica identity. With REPLICA IDENTITY FULL it would hold "status": "paid" and the rest of the old row.
• after is the new row.
• op is the operation: c create, u update, d delete, r snapshot read.
• source is metadata: connector, database, schema, table, transaction ID and LSN.
• Two timestamps: source.ts_ms is when the change was committed in the database; the top-level ts_ms is when Debezium processed it. Subtract one from the other and you have capture lag for every single event (412 ms in this example).

A delete has after set to null, and by default it's followed by a tombstone: a message with the same key and a null value, so Kafka log compaction can eventually drop that key. And with the plain JSON converter and schemas.enable=true, every message carries its full schema alongside the payload. Avro or Protobuf with a schema registry sends a small schema ID instead.

Five common Debezium mistakes

1. Leaving plugin.name at the default. decoderbufs isn't installed on most servers, so the connector won't start.
2. Two connectors sharing one slot. That's silent data loss. One connector, one slot, one publication.
3. slot.drop.on.stop=true outside a dev environment. Stop the connector and the slot, with its position, is gone.
4. Expecting before to hold the old row without REPLICA IDENTITY FULL.
5. No heartbeat and no alert on how much WAL the slot is holding. That one can take down the database itself.

Recap

• Postgres needs logical WAL, a replication user and a publication you own.
• The connector needs pgoutput, its own slot, a fixed topic.prefix and heartbeats.
• On first start it snapshots as of a recorded log position, then streams from that exact position.
• Each table becomes a topic keyed by primary key, so ordering is per row.
• Each event carries before, after, source, op and ts_ms.

The CDC in Production series

• Part 1: Change Data Capture (CDC) Explained (article · video)
• Part 2: Debezium Postgres to Kafka (this post · video)
• Next, Part 3: The Silent CDC Bug: Data Loss With No Errors (video)
• Coming up: Part 4 (deletes, tombstones and the idempotent MERGE), Part 5 (schema changes), Part 6 (the WAL slot trap)

---

If this was useful, the video version has the animated diagrams for every step: watch it on YouTube. Subscribe to Pipeline Whiteboard for more diagram-first data engineering explainers. Next up, in Part 3: the bug that loses data without a single error, and how reconciliation catches it.