CDC Is Not a Feature — It's the Foundation Nobody Talks About Until It Fails
Change Data Capture keeps showing up in architecture diagrams as a box between a database and a Kafka topic. It looks simple. It almost never is.
There's a reason teams reach for CDC late in the game — usually after they've tried polling, batch exports, or webhook-from-the-app approaches and found them wanting. By then the data model is already messy and the production pressure is real. Getting CDC right retroactively is a very different project from designing for it upfront.
What CDC Actually Does (Beyond the Marketing)
At its core, CDC reads the database transaction log — the binlog in MySQL, the WAL in Postgres — and converts row-level change events into a stream. Every insert, update, and delete becomes a message with a before/after state, a timestamp, and a transaction ID.
This is genuinely powerful for a few reasons:
- You capture intent, not just state. A price field going from 101.50 to 99.00 is a different signal than seeing 99.00 in a snapshot.
- You get ordering guarantees within a transaction, which matters a lot when two tables change atomically.
- You're not polling, so you're not hammering the primary under load or missing changes between intervals.
But the log is an internal implementation detail of the database engine, not a public API. Which means it changes, has quirks, and requires careful handling.
The Failure Modes Nobody Warns You About
Log retention gaps
CDC connectors track their position in the replication log. If the connector goes down long enough — or falls too far behind — the database will rotate and purge the log entries it hasn't read yet. When the connector comes back, those changes are gone. This is not recoverable without a snapshot re-seed, which typically requires locking or pausing writes.
In financial or market data contexts, where event ordering and completeness matter more than throughput, this failure mode is quietly catastrophic. Missing 20 minutes of order book updates isn't a data quality issue — it's a correctness issue.
Schema evolution with no coordination
The binlog stores raw byte offsets, not schema-aware messages. When you alter a column — rename it, change its type, drop it — your CDC consumer needs to know about it at exactly the same moment the change takes effect in production. Most teams solve this with a schema registry, but schema evolution in high-volume pipelines is still one of the most common causes of silent data corruption.
The "at least once" problem compounds
CDC guarantees that changes will be delivered, but not that they'll be delivered exactly once. Deduplication logic at the consumer is non-negotiable. The tricky part is that the deduplication key (transaction ID + offset) works fine for simple cases, but breaks down when you're joining CDC streams with other event sources that have their own ID spaces.
Where CDC Gets Interesting in Real-Time AI and Agentic Pipelines
Most CDC discussions stop at "get data out of the database and into Kafka." That's solved infrastructure at this point. The harder question is what happens downstream.
Stateless consumers can't reason about change
An LLM or ML model consuming CDC events has no memory of what came before unless you build that explicitly. A sequence of three price updates might mean reversion, momentum, or a fat-finger error — and distinguishing them requires seeing the full sequence in order, with latency that doesn't destroy the signal.
This is where a lot of "real-time AI" pipelines quietly fail. The data is technically real-time, but the model is consuming it as if each event is independent. The result is a system that reacts to noise rather than signal.
Rolling context windows need durable stream state
If you want a model to maintain a view of recent entity state — a customer's last five interactions, a trading instrument's recent volume profile — you need somewhere to materialize that rolling context. CDC gives you the raw events; you still need the infrastructure to maintain the derived state, serve it low-latency, and keep it consistent with the source as changes flow in.
That derived state layer is where most teams underinvest. It's also where the real-time AI story actually lives.
Designing for CDC from Day One
A few patterns that make life easier:
Use surrogate keys, not natural keys, as primary keys. Natural keys change. When they do in a CDC stream, it looks like a delete and an insert — which breaks consumer logic that assumes identity is stable.
Log everything at the source, filter downstream. It's tempting to filter CDC events at the connector level to reduce volume. Resist it. Filtered-out events are gone. Consumer requirements change; you can't replay history you never captured.
Treat schema changes as deployments. Any column rename or type change that touches a CDC-consumed table should go through the same release process as a code change. Schema registry + migration tooling + consumer compatibility checks — all of it, every time.
Test lag recovery explicitly. Simulate connector downtime in staging. Know exactly what your log retention window is and build alerting around connector lag hitting 50% of that window. This is the failure mode that bites hardest in production and gets the least attention in pre-production testing.
The Invisible Becomes Visible When It Fails
The "plumbing" metaphor is accurate. CDC is infrastructure nobody thinks about when it's working. When it isn't, it's all anyone can think about — because every downstream system that depends on database state being reflected accurately is now wrong, and figuring out how wrong requires reconstructing history you may no longer have.
Getting it right isn't glamorous. But it's one of those investments that quietly enables everything downstream — real-time analytics, event-driven microservices, ML feature pipelines, agentic systems — to actually work as designed.
Top comments (0)