Change data capture gets sold as the fix for batch sync problems, and in a lot of ways it is. Streaming every row-level change as it happens instead of periodically re-scanning a whole table closes most of the gaps where batch jobs quietly drop data. But "most" isn't "all," and teams that treat a working CDC pipeline as proof they no longer need reconciliation are setting themselves up for a much harder debugging session later.
What CDC actually solves
Change data capture, using something like Debezium reading a database's write-ahead log, streams inserts, updates, and deletes as they happen, rather than relying on a scheduled job to notice what changed since the last run. This eliminates a whole category of batch-sync bugs: missed windows, race conditions between overlapping batch runs, and the delay between a change happening and a downstream system learning about it.
For teams migrating off nightly batch jobs, this genuinely fixes real, recurring problems. It's not marketing.
What it doesn't solve
CDC captures changes correctly assuming the pipeline stays healthy end to end. It doesn't protect against a consumer that silently stops processing events partway through a deploy, a schema change upstream that the CDC connector doesn't know how to map, or a network partition that causes the stream to resume from the wrong offset after recovery. In every one of those cases, the CDC pipeline can look completely healthy, connected, streaming, no errors in the logs, while actually missing a chunk of changes.
The failure mode is the same one that plagues batch pipelines: quiet, not loud. CDC changes where the failure can happen, not whether it can happen.
Why "no errors in the log" isn't the same as "no drift"
A CDC consumer that crashes and restarts from a stale offset will happily keep processing new events without ever raising an exception. It just permanently skipped whatever changed during the gap. Unless something is comparing the actual state of the source and destination independently of the stream itself, that gap is invisible until someone notices a specific record is wrong.
This is exactly the blind spot reconciliation exists to close. It doesn't care how the destination got its data, streamed via CDC or batched overnight. It just checks whether the destination currently agrees with the source, which is the one thing a healthy-looking pipeline log can't actually confirm.
What a minimal reconciliation layer looks like alongside CDC
You don't need to duplicate the whole CDC pipeline's logic. A lightweight, independent job that periodically samples records from both source and destination and compares them catches the offset-gap and schema-drift failure modes without needing to understand the streaming pipeline's internals at all. Row counts on a schedule are a reasonable starting point; checksum comparison on a sample is a stronger one.
The key property is independence. If the reconciliation job reads its expected state from the same stream the CDC pipeline consumes, a bug in that stream can take both the pipeline and its own safety check down together.
The pattern in practice
Teams running Kafka-backed CDC pipelines often assume the stream's own delivery guarantees are equivalent to a correctness guarantee. They're not the same thing. Delivery guarantees describe what happens to a message once it's in the stream. They say nothing about whether the consumer correctly processed every message it should have received, or whether the connector correctly captured every change from the source in the first place.
Pairing streaming CDC with periodic, independent reconciliation gets you both properties: the low latency and reduced batch-window risk of CDC, plus the closed-loop verification that confirms the destination actually matches reality rather than just trusting that it does.
Offset management is where most of this risk concentrates
If there's one part of a CDC pipeline worth understanding deeply before trusting it fully, it's how offset tracking and recovery actually work. Every CDC connector has to record its position in the source's change stream so it knows where to resume after a restart. When that offset store gets corrupted, reset incorrectly, or simply lags behind due to a slow consumer, the connector can resume from a point that skips real changes, and it will do so without any indication that anything's wrong.
This is worth testing deliberately rather than trusting the documentation's description of the failure mode. Kill the consumer mid-stream in a staging environment, force a restart, and verify independently, via reconciliation, not via the pipeline's own logs, that no changes were actually lost during the gap. Teams that skip this test tend to discover the failure mode for the first time in production, which is a considerably more expensive way to learn it.
Schema evolution is the other common blind spot
CDC connectors read a structured representation of each change from the source's write-ahead log or equivalent, and that representation assumes a stable schema. When a column gets added, renamed, or has its type changed upstream, the connector has to either adapt automatically or fail loudly. In practice, many configurations do neither: they silently drop the new field, or coerce it into an unexpected type, and continue streaming without an error. A reconciliation check that compares actual field values, not just record presence, is what catches this specific failure, because row counts and even basic connectivity checks look completely normal the entire time it's happening.
A note on connector configuration defaults
Most CDC connectors ship with default snapshot and recovery behavior tuned for the common case, not for correctness guarantees under every failure scenario. It's worth reading through your specific connector's configuration options for offset storage backend, snapshot mode, and recovery policy rather than accepting the defaults blindly, since some default configurations trade a small amount of correctness risk for simpler operations, which may or may not be the right tradeoff for the specific table you're streaming.
Document management systems like MongoDB, which expose change streams as a first-class feature, have similar considerations around resume tokens as relational databases have around log sequence numbers. The underlying risk, a consumer resuming from the wrong point after an interruption, is the same regardless of which database technology sits underneath the CDC layer.
What good monitoring for a CDC pipeline actually includes
Consumer lag is the metric everyone monitors, and it's necessary but not sufficient. A consumer that's caught up on lag can still have silently missed a window of events during a prior outage, and lag alone won't show that. Pair lag monitoring with periodic, independent reconciliation, and you get both halves of the picture: whether the pipeline is keeping up in real time, and whether what it's delivered so far actually matches the source.
The takeaway
CDC is a real improvement over batch sync for most of the problems it targets. It is not, on its own, a substitute for checking whether your data is actually correct. 137Foundry covers the broader case for treating reconciliation as a continuous, independent check, not a one-time audit, in our guide to why data sync jobs pass every test but still drift from the source, which walks through the same principle applied to batch pipelines as well.
For the streaming side of this, Debezium's own documentation is worth reading closely, particularly the sections on offset management and recovery behavior, since that's where the gap this article describes tends to actually originate. For the messaging layer underneath most CDC setups, Apache Kafka's documentation on consumer offsets and rebalancing covers the mechanics behind the recovery scenario described above in more technical depth.
Top comments (0)