It was 3:15 AM on a Tuesday when the PagerDuty alerts started screaming. Our Bronze-to-Silver processing job, which usually took forty minutes, had been running for six hours. The cluster was pegged, the spill-to-disk metrics were off the charts, and we were locking out the downstream BI dashboard for our compliance team. By the time I manually killed the job and truncated the Silver table to perform a full-refresh, we were three hours behind our SLA. That incident cost the firm roughly $40,000 in regulatory reporting penalties and a very awkward meeting with the CTO. The culprit? A "simple" merge operation on a 5TB table that finally hit a partition skew tipping point.
Why I chose this topic: I spent three years watching engineers treat Delta tables like static files, only to be surprised when their jobs collapsed under volume. I’m writing this because incremental processing isn't a "nice to have" anymore—it’s the only way to keep your sanity in production environments.
Most of you are currently standing at a crossroads. You have a massive Bronze (raw) layer, and you need to feed a Silver (cleansed) layer. You’re either running full-table overwrites, which is a financial death trap as your data grows, or you’re hacking together custom watermarks that break the second a schema changes or a late-arriving record shows up. You’re choosing between "simple but expensive" and "complex but fragile."
The contenders
You have three primary ways to move data from Bronze to Silver in the Delta ecosystem.
First, the Full Overwrite. You read the entire source, apply your logic, and df.write.mode("overwrite") the target. It’s clean, it’s idempotent, and it’s mathematically guaranteed to bankrupt you if your data volume doubles annually.
Second, the Custom Watermark. You track a max_processed_timestamp in a control table, filter your source by event_time > last_processed, and perform a merge into the target. It’s the "classic" approach. It works until you have an update to a record that happened three days ago, which your watermark filter will conveniently ignore.
Third, Change Data Feed (CDF). You enable delta.enableChangeDataFeed = true on your Delta table. Delta Lake then tracks every row-level change (inserts, updates, deletes) in a hidden log. You treat the table like a stream, consuming only the deltas.
Photo by Michael Evans on Unsplash
The cost of doing business
Let’s talk about the bill. Full Overwrites are the most expensive because you pay for the compute to shuffle the entire dataset for every run. If you have 10TB of data and only 1GB changed, you are paying for the other 9.999TB of sorting and shuffling.
Custom Watermarks feel cheaper, but the hidden cost is the merge operation. When you run a MERGE INTO, Spark has to perform a full shuffle of the target table to identify the files containing matching keys. If your Silver table is partitioned by date, and your updates are scattered across two years of data, a single MERGE will force Spark to rewrite every single partition in your table.
CDF is the clear winner here. Because CDF gives you a stream of only the changes, your Silver layer transformation becomes a pure append-only operation if you design it right. You are reading a few hundred megabytes of deltas instead of terabytes of static data. Your compute clusters can be 1/10th the size, and your job duration drops from hours to minutes.
Ops burden and the "missing update" problem
The real nightmare in production is the "missing update." With Watermarks, if a record arrives late or an upstream system triggers a retroactive correction, your job will never see it unless you implement a "look-back" window—which basically forces you back into a partial full-refresh, re-processing data you’ve already handled.
CDF solves this by design. Because CDF captures the transaction log, it doesn't care if an update comes in two hours late. It will show up in the change feed as a new version of the row. You don't have to write complex logic to handle late-arriving data; you just process the stream.
However, CDF comes with an ops tax. You must manage table properties. If you’re using Databricks or Delta OSS, you need to ensure delta.enableChangeDataFeed is set at table creation. If you try to enable it on a massive, pre-existing table, you’re looking at a rewrite of that table's metadata, which can be a blocking operation. You also need to manage your delta.deletedFileRetentionDuration and delta.logRetentionDuration. If your job fails for three days and your retention is only two days, you’ve lost your pointer to the stream. You’ll be doing a full rebuild anyway.
Photo by Jacek Dylag on Unsplash
Failure modes and recovery
The failure mode for Full Overwrites is simple: the job takes too long and times out. Recovery is easy but painful—you just rerun it.
The failure mode for Watermarks is silent data corruption. If your control table update fails after the merge finishes but before the commit metadata is written, you might double-process or lose data on the next run. Tracking the state of your pipeline becomes a distributed systems problem that you probably aren't qualified to solve (neither am I).
CDF has a cleaner failure mode. Since it’s integrated with Spark Structured Streaming, you get checkpointing for free. If the job dies, it restarts from the exact offset in the transaction log. You don't have to guess where you left off. The failure mode isn't "data corruption"; it's "the job stopped." You restart it, it catches up, and you move on.
What I'd pick, and why
I’ve reached a point where I refuse to build a production pipeline that doesn’t use CDF. It is the only way to move from "managing data" to "managing streams."
If you’re sitting on a massive, legacy Silver layer, don't try to migrate it all at once. Start by enabling CDF on your new Bronze ingestions. Build a "Silver-Incremental" table alongside your old "Silver-Legacy" table. Once you prove the latency gains, point your downstream BI tools to the new table and drop the old one.
The caveats:
First, check your storage costs. CDF creates extra files (the change data). It increases your storage footprint by about 10–15%. In the world of cloud storage, that’s pennies compared to the hundreds of dollars an hour you’re burning on massive Spark clusters.
Second, be careful with schema evolution. If you add a column to your Bronze table, your Silver job needs to handle that. When using CDF, you’re often casting and transforming in a stream. If a schema change isn't backwards compatible, your streaming job will crash. You need to implement schema evolution (spark.databricks.delta.schema.autoMerge.enabled = true) on your write operations, but even then, test your Silver logic in a dev environment first.
Third, don't over-partition. A common mistake is partitioning by customer_id or transaction_id. If you have a high-cardinality column, your partitions will be too small (the "small file problem"). Stick to partitioning by date or region, and let Delta’s Z-Ordering handle the performance optimization.
Stop fighting the infrastructure. The tools to handle incremental data are already built into the Delta protocol. If you’re still doing full refreshes, you’re not an engineer—you’re a janitor cleaning up after a bad architecture. Stop the bleeding, enable the feed, and save your sleep.
Top comments (0)