Every engineering team has a version of this story. You spend three sprints tuning Kafka partition counts, debating consumer group strategies, and benchmarking Spark streaming configs. The architecture review doc runs twelve pages. Someone draws a beautiful diagram with arrows and boxes and latency budgets.
Then a microservice starts serving stale data. A dashboard shows totals that stopped updating six hours ago. A downstream system missed a batch of deletes and is now cheerfully reporting records that no longer exist. The post-mortem reveals the culprit: nobody ever seriously designed the layer that feeds all those carefully architected systems in the first place.
That layer is Change Data Capture. And it deserves a seat at the grown-ups table.
What CDC Actually Does (And Why It's Harder Than It Sounds)
The concept is almost insultingly simple. CDC watches your database's transaction log, the internal record your database already keeps of every write operation, and emits a structured event for every insert, update, or delete. That's it.
But "simple in concept" doesn't mean "trivial to operate." The transaction log was never designed to be a public API. It's format varies by database engine, changes across major versions, and behaves differently under replication lag, schema migrations, or high write throughput. Tools like Debezium, AWS DMS, and Google Datastream abstract most of this complexity, but they don't eliminate the decisions you still have to make about event ordering guarantees, schema evolution handling, and exactly-once delivery semantics.
Here's a minimal example of what a Debezium event actually looks like when a row gets updated:
{
"op": "u",
"before": {
"id": 1042,
"status": "pending",
"amount": 250.00
},
"after": {
"id": 1042,
"status": "completed",
"amount": 250.00
},
"source": {
"ts_ms": 1718021400000,
"db": "orders",
"table": "transactions"
}
}
Notice what's in there: the full before and after state, the operation type, a timestamp, and source metadata. This is not a simple row dump. It's a structured changelog that downstream systems can use to reconstruct state, audit changes, or trigger workflows, without polling, without full-table scans, without drift.
The Full-Table ETL Trap
The alternative most teams default to is scheduled ETL jobs. Pull the whole table every fifteen minutes, compare it to what you had before, figure out what changed. This works until it doesn't, and the failure modes are genuinely ugly.
Deletes are invisible. If a row gets removed from the source table, your next ETL run just... won't include it. Unless you've built explicit tombstone logic, your downstream system never finds out. Soft deletes with an is_deleted flag are a common workaround, but they require every application team to follow the convention, every time, forever. That's a governance problem masquerading as a technical solution.
High-volume tables become a resource problem. A table with tens of millions of rows, queried in full every fifteen minutes, adds meaningful load to your source database. At some point that load interferes with production traffic, and then you've made your data pipeline into a reliability risk for the system it's supposed to be observing.
Latency compounds. Fifteen-minute polling gives you data that's already fifteen minutes old before it starts processing. Add transform time, load time, and downstream propagation, and "real-time dashboard" starts meaning something embarrassing.
CDC sidesteps all of this because it works at the event level, not the snapshot level. Every change is captured at the moment it happens, including deletes, including partial updates, including the exact sequence of operations if you need it.
Why It Gets Ignored Until Something Breaks
CDC doesn't feel exciting to design. It's infrastructure, not product. The engineers who set it up correctly leave no visible fingerprints because everything just works. The engineers who set it up poorly also leave no visible fingerprints, until a production incident happens and suddenly everyone is staring at the pipeline architecture trying to figure out where the missing data went.
This is how you end up in a post-mortem where someone says "we never really defined ownership of the CDC layer" and nobody disagrees.
The Kafka topology got ownership. The Spark jobs got ownership. The CDC connector that feeds both of them was configured by whoever had time that week and has not been meaningfully revisited since.
What Treating CDC as a First-Class Concern Actually Looks Like
It means including CDC connector configuration in your infrastructure-as-code, not in a Confluence doc that describes what someone did manually in 2022. It means defining explicit schema evolution policies before you need them, not when a migration breaks your consumers. It means monitoring connector lag and offset commit health the same way you monitor application error rates.
It means deciding, upfront, whether your CDC events are the system of record for downstream state or an input to something that needs its own reconciliation logic. That distinction changes how you build every consumer.
Real-time systems are only as reliable as the data movement layer beneath them. CDC is that layer for most modern pipelines. The teams that figure this out before something breaks ship more reliable systems and spend fewer Saturdays in incident calls.
The ones that figure it out after have a much better post-mortem document.
Top comments (0)