DEV Community

SoftwareDevs mvpfactory.io
SoftwareDevs mvpfactory.io

Posted on • Originally published at mvpfactory.io

Change Data Capture Without Debezium: Wiring Postgres WAL Directly to Your Event Bus

#webdev #programming 
---
title: "CDC Without Debezium — Wire Postgres WAL Directly to Kafka"
published: true
description: "Skip the JVM overhead. Build a lightweight CDC connector using pg_recvlogical and pgoutput in Go or Kotlin, with slot management, LSN checkpointing, and production-safe failure recovery."
tags: [postgresql, architecture, devops, api]
canonical_url: https://mvpfactory.co/blog/cdc-without-debezium-postgres-wal-kafka
---
Enter fullscreen mode Exit fullscreen mode

What You Will Build

By the end of this workshop, you will have a working Change Data Capture pipeline that reads row-level changes directly from Postgres's Write-Ahead Log and publishes them to Kafka or Redis Streams — without Debezium, without Kafka Connect, and without a JVM dependency in sight.

Why Skip Debezium?

Debezium is the industry default for CDC, but most teams do not realize they are signing up for an entire platform. When your scope is focused — single database, a handful of tables — the operational cost outweighs the benefit fast.

Here is the comparison that made me rethink my defaults:

Approach Memory Dependencies Startup
Debezium on Kafka Connect 512MB–2GB JVM Kafka Connect + ZooKeeper 10–30s
DIY Go connector 20–60MB Postgres + Kafka/Redis only <1s
DIY Kotlin connector 80–150MB Postgres + Kafka/Redis only 2–4s

For lean infrastructure or microservices where one service owns one schema, the DIY path is legitimate production engineering — not premature optimization.

Prerequisites

  • Postgres 10+ (pgoutput is built in — no extensions needed)
  • Go 1.21+ or Kotlin with coroutines
  • A running Kafka or Redis Streams instance
  • pglogrepl for Go: go get github.com/jackc/pglogrepl

The Architecture

Postgres exposes row-level changes through replication slots. The pgoutput plugin serializes them into a binary protocol your connector reads over a standard replication connection.

Postgres WAL → Replication Slot (pgoutput) → Connector (Go/Kotlin) → Kafka / Redis Streams
Enter fullscreen mode Exit fullscreen mode

Step 1 — Create the Replication Slot and Publication 

SELECT pg_create_logical_replication_slot('my_cdc_slot', 'pgoutput');

CREATE PUBLICATION my_pub FOR TABLE orders, inventory;
Enter fullscreen mode Exit fullscreen mode

Use pgoutput over wal2json. It is binary, faster, and ships built into Postgres — no extension installation required.

Step 2 — Connect and Start Replication in Go 

conn, _ := pgconn.Connect(ctx, os.Getenv("DATABASE_URL"))

sysident, _ := pglogrepl.IdentifySystem(ctx, conn)
log.Printf("System ID: %s, LSN: %s", sysident.SystemID, sysident.XLogPos)

err := pglogrepl.StartReplication(ctx, conn, "my_cdc_slot", sysident.XLogPos,
    pglogrepl.StartReplicationOptions{
        PluginArgs: []string{
            "proto_version '1'",
            "publication_names 'my_pub'",
        },
    })
Enter fullscreen mode Exit fullscreen mode

Step 3 — Checkpoint LSN After a Durable Write

Let me show you a pattern I use in every project. Confirm the Log Sequence Number only after your event lands durably in Kafka — never before:

// Confirm LSN only after successful Kafka produce + flush
err = producer.Flush(5000) // wait for broker ack
if err == nil {
    pglogrepl.SendStandbyStatusUpdate(ctx, conn,
        pglogrepl.StandbyStatusUpdate{WALWritePosition: currentLSN})
}
Enter fullscreen mode Exit fullscreen mode

At-least-once delivery is the contract. Your downstream systems must be designed for it.

Step 4 — Handle Schema Evolution

pgoutput sends column data by position, not by name. ADD COLUMN is safe — new columns append to the end and existing offsets are preserved. DROP COLUMN and column reordering will break your decoder.

The fix: maintain a local schema cache keyed by relation OID. When a RelationMessage arrives, diff it against your cache and emit a schema-change event before the data event. Consumers that care can pause and migrate; consumers that do not can ignore it.

Gotchas

LSN checkpointing is where DIY CDC connectors fail in production. If your connector crashes without confirming its LSN, it replays from the last confirmed position. Duplicates are expected; data loss is not. Design consumers for idempotent writes.

Kafka going down. Buffer events locally — a SQLite file or Redis list works — and do not confirm LSN until Kafka recovers. Do not lose your WAL position because the broker is temporarily unavailable.

Postgres failover. Here is the gotcha that will save you hours: replication slots do not survive a primary switch by default. Either use the pg_failover_slots extension or rebuild the slot on the new primary and resume from your last checkpointed LSN. Plan for this before production, not during an incident.

Talk to your DBA before the first ALTER TABLE hits production. Cache relation OIDs, define what "column stability" means for your tables, and get explicit agreement upfront. Much easier to sort out before than to retrofit.

Conclusion

A lightweight CDC connector is a deliberate architectural choice. When your scope is focused, the operational simplicity of a 20MB Go binary beats a 2GB JVM stack every time.

The docs do not always surface this, but pgoutput + pglogrepl + disciplined LSN checkpointing gives you a connector you can trust in production. Nail those three pillars — at-least-once delivery, schema-change events, and a failover plan — and this pattern will serve you for years.

Resources:

Top comments (0)