Monitoring Airbyte's Postgres CDC Replication Slots Before They Take Down Your Source Database

Every Postgres CDC source works the same way under the hood: it opens a logical replication slot, and Postgres promises to retain every write-ahead log (WAL) segment since that slot's restart_lsn until the consumer confirms it has read past that point. That promise is exactly what makes CDC reliable — no gaps, no missed updates — and exactly what makes it dangerous. If the consumer stops reading and nobody notices, Postgres keeps its promise anyway. It holds the WAL. Forever, if you let it.

Airbyte's Postgres source is one of the most common ways teams stand up CDC without hand-rolling a Debezium deployment, which makes it one of the most common ways teams accidentally fill up their primary database's disk. This isn't a bug in Airbyte — it's a property of logical replication that Airbyte inherits, and it bites self-hosted OSS users far more often than Cloud users, because OSS gives you the rope and Cloud gives you a bit more of a safety net around it. If you're running Airbyte against a production Postgres instance and you don't have a monitor on pg_replication_slots, you have a landmine sitting under your database, and it goes off quietly.

How the slot actually stalls

A replication slot goes stale for reasons that have nothing to do with Postgres being unhealthy:

• The connection is paused or disabled in Airbyte, intentionally or because someone clicked the wrong toggle, but the slot Airbyte created during setup isn't dropped when a connection is merely paused — only when it's fully removed.
• The destination is down or rejecting writes (schema mismatch, warehouse quota, an expired credential), so the sync job fails after decoding WAL but before Airbyte checkpoints state, and the slot's confirmed_flush_lsn never advances.
• The sync cadence is long (say, a 6-hour or daily schedule) on a high-write table, so WAL accumulates for hours between confirmations even in the healthy case — this is normal, but it means your safety margin before a stall becomes a disk-fill incident is much smaller than a real-time consumer would give you.
• The Airbyte worker pod restarts mid-sync in a Kubernetes deployment and the new attempt doesn't resume cleanly, leaving an orphaned slot that nothing is actively reading from.

In every case, Postgres's pg_wal directory grows because it cannot recycle any WAL segment newer than the stuck slot's restart_lsn, regardless of how many other consumers (including physical replicas) are keeping up fine. One neglected connection is enough to take the whole instance down.

Detecting it before the disk does

The query that matters is a straight comparison between the current WAL position and each slot's retained position, converted to bytes:

SELECT
  slot_name,
  active,
  pg_size_pretty(
    pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  ) AS retained_wal,
  pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn) AS retained_bytes
FROM pg_replication_slots
ORDER BY retained_bytes DESC;

active tells you whether a process currently holds the slot open — but don't trust it alone. A slot can show active = true between sync attempts and still be retaining gigabytes of WAL because Airbyte's connector reconnects between runs without ever calling it a failure. The byte count is the number that predicts an incident; active only tells you whether something is currently attached.

Wire this into whatever you already use for scheduled checks — cron, a Lambda, a sidecar container next to your Airbyte deployment. A minimal version:

#!/usr/bin/env bash
set -euo pipefail

WARN_BYTES=$((5 * 1024 * 1024 * 1024))   # 5 GiB
PAGE_BYTES=$((20 * 1024 * 1024 * 1024))  # 20 GiB

psql "$DATABASE_URL" -Atc "
  SELECT slot_name || '|' || pg_wal_lsn_diff(pg_current_wal_lsn(), restart_lsn)
  FROM pg_replication_slots;
" | while IFS='|' read -r slot bytes; do
  if (( bytes > PAGE_BYTES )); then
    curl -s -X POST "$SLACK_WEBHOOK_URL" \
      -H 'Content-Type: application/json' \
      -d "{\"text\":\":rotating_light: slot '${slot}' retaining $(numfmt --to=iec ${bytes}) of WAL — paging\"}"
  elif (( bytes > WARN_BYTES )); then
    curl -s -X POST "$SLACK_WEBHOOK_URL" \
      -H 'Content-Type: application/json' \
      -d "{\"text\":\":warning: slot '${slot}' retaining $(numfmt --to=iec ${bytes}) of WAL\"}"
  fi
done

Run it every five minutes. The two-tier threshold matters: a slot that's merely behind schedule (warn) is a different problem than a slot growing unbounded with no active consumer (page). Conflating them either trains your team to ignore the alert or wakes someone up for a sync that was always going to catch up on its next scheduled run.

Stopping the bleed automatically, not just alerting on it

Alerting tells a human there's a problem. On a small team, "a human sees the Slack message at 2am" is not a remediation plan. The more useful move is a circuit breaker: when a slot's retained WAL crosses the page threshold, pause the offending Airbyte connection via the API before the disk fills, rather than waiting for a person.

Airbyte's public API exposes connection status directly:

curl -s -X PATCH "https://api.airbyte.com/v1/connections/${CONNECTION_ID}" \
  -H "Authorization: Bearer ${AIRBYTE_API_TOKEN}" \
  -H "Content-Type: application/json" \
  -d '{"status": "inactive"}'

Pausing the connection doesn't drop the replication slot — Airbyte leaves the slot in place so a resumed sync can pick up where it left off without a re-snapshot. That's exactly the behavior you want here: pausing stops the connector from retrying against a broken destination, but it doesn't discard the checkpoint. If you need to actually reclaim disk immediately, that requires dropping the slot itself:

SELECT pg_drop_replication_slot('slot_name');

Do this only as a last resort, and only after confirming the connection is paused first — dropping a slot that's actively mid-sync will corrupt Airbyte's internal state tracking for that stream. Dropping the slot also means the next sync for that connection has lost its CDC position and Airbyte will fall back to a full initial snapshot, which is the exact expensive, disruptive operation you were trying to avoid by monitoring in the first place. The auto-pause is meant to buy you time to fix the actual problem — a stuck destination, a failed credential — not to be the fix itself.

What to actually configure

Two settings changes prevent most of this class of incident before it starts:

• Set wal_sender_timeout conservatively rather than disabling it — a slot with no active walsender for longer than this gets flagged by monitoring faster than a passive byte-count check alone would catch.
• If you're running Postgres 13+, set max_slot_wal_keep_size as a hard backstop. This caps how much WAL a single slot can force Postgres to retain; past that cap, Postgres invalidates the slot rather than filling the disk. An invalidated slot forces a re-snapshot on the next sync, which is a bounded, visible failure — far better than an unbounded disk-fill that takes the whole instance offline for every connection, not just the misbehaving one.

Neither setting replaces the monitor. max_slot_wal_keep_size is the backstop for when your monitoring itself fails; the monitor is what lets you fix the real cause — a paused connection, a broken destination — before you're forced into that backstop and eating a full resync. Teams that skip the monitor and rely on the Postgres-side cap alone tend to discover the problem only when a large table's next sync triggers an hours-long re-snapshot at the worst possible time, with no warning that it was coming.

CDC through Airbyte is easy to set up and easy to forget about, which is precisely why it needs a monitor watching the one number — retained WAL bytes per slot — that predicts the outage before it happens.