03:14 AM. Tuesday. The PagerDuty alert hits my phone with the specific, soul-crushing vibration reserved for production database outages.
"Pipeline billing_reconciliation_prod is failing to commit."
I roll out of bed, laptop open before my eyes fully adjust. The Databricks Delta Live Tables (DLT) dashboard is glowing a frantic, pulsating red. In financial services, a reconciliation failure at 3 AM isn't just a technical glitch; it’s a compliance incident waiting to happen. If those records don’t hit the Gold layer by 06:00, the downstream BI tools report $0 revenue for the previous day.
Here is the kicker: 92% of DLT pipeline failures in production are not caused by code bugs, but by "silent" data quality violations that we explicitly told the system to ignore. We treat DLT expectations as suggestions, not gates, and that is exactly why we were currently staring at a dead pipeline.
What we saw
The logs were screaming about ExpectationViolationException. Specifically: EXPECTATION_VIOLATED: transaction_id IS NOT NULL.
We had an expectation defined in our DLT pipeline:
@dlt.expect_or_drop("valid_transaction_id", "transaction_id IS NOT NULL").
The symptoms were classic. The pipeline was stuck in a retry loop. Because we used expect_or_drop, the data was simply disappearing into the ether. Or so we thought. The real issue wasn't the drop; it was the volume. A malformed upstream batch from a third-party payment gateway had pushed 400,000 records, 95% of which were missing the transaction_id.
The pipeline wasn't just dropping data; it was hitting the DLT threshold for "excessive failure rates." When more than 50% of your records fail an expectation, DLT’s internal state machine panics. It flags the pipeline as unhealthy and stops the ingestion.
My first instinct was the false lead: "Someone changed the schema upstream." I wasted forty minutes digging through information_schema.columns and checking if our schema evolution settings were set to rescue. They were. The schema was fine. The data, however, was fundamentally broken. We had treated the expectation as a garbage disposal, but the garbage was too big for the pipe.
Photo by Winston Chen on Unsplash
Root cause
The root cause was our reliance on expect_or_drop for critical financial records. In a regulated environment, you cannot "drop" data and pretend it never existed. If a record fails, it needs an audit trail.
We were using:
@dlt.table
@dlt.expect_or_drop("valid_amount", "amount > 0")
def transactions():
return spark.readStream.table("bronze.raw_transactions")
When the amount column arrived as a string literal or a null value, the record vanished. Because DLT doesn't natively surface the dropped records into a secondary "quarantine" table without explicit orchestration, we lost total visibility. We were blind. We had a gate that blocked the flow but left the offending data trapped in the cloudFiles source directory, causing the micro-batch to fail repeatedly because the same corrupted file was being re-processed every single minute.
The offending mechanism was the lack of a "Quarantine Sink." We had configured our pipeline to be a binary filter: valid data goes to Silver, invalid data goes to /dev/null. In banking, /dev/null is a compliance nightmare.
Photo by Markus Spiske on Unsplash
The fix
I didn't need to fix the data—I couldn't control the upstream vendor. I needed to fix the plumbing.
I refactored the pipeline to stop using expect_or_drop for anything that triggered a production reconciliation. Instead, I moved to a "Split and Quarantine" pattern. We stopped filtering at the DLT expectation layer and moved to an explicit staging pattern.
I redefined our ingestion logic to use a two-step process:
# The explicit quarantine pattern
@dlt.table
def validated_transactions():
return dlt.read("raw_transactions") \
.withColumn("is_valid", col("transaction_id").isNotNull() & (col("amount") > 0))
@dlt.table(name="silver_transactions")
def silver():
return dlt.read("validated_transactions").filter("is_valid = True")
@dlt.table(name="quarantine_transactions")
def quarantine():
return dlt.read("validated_transactions").filter("is_valid = False")
By doing this, we turned the "gate" into a "sorter." The pipeline no longer fails when the upstream vendor sends garbage; it simply diverts the garbage into quarantine_transactions. We then set up a separate alerting mechanism—not on the pipeline status, but on the row count of the quarantine table. If the quarantine table grows by more than 100 rows in an hour, that is when the pager goes off.
We gained observability. Instead of a crashed pipeline at 3 AM, we had a healthy pipeline that was successfully segregating bad data, and a ticket in Jira detailing exactly which transaction_ids were malformed.
What we changed so it never happens again
We stopped using expectations for data quality control and started using them for data quality monitoring. There is a massive, often misunderstood difference.
First, we implemented a "Dead Letter Queue" (DLQ) pattern for all DLT pipelines. Every single pipeline now has a corresponding quarantine table. If the data isn't clean enough to enter the Gold layer, it is moved to a locked-down, restricted-access table where our Data Stewards can inspect it.
Second, we moved away from expect_or_drop entirely. We now use expect_or_fail only for catastrophic data issues (e.g., if the primary key column is missing in 100% of rows). For everything else, we use custom logic to flag records as "valid" or "quarantined." This keeps the pipeline running, keeps the metrics accurate, and keeps the auditors happy.
Third, we automated the "re-drive" process. We built a small Python utility that allows Data Stewards to update the quarantine records—fixing typos or missing metadata—and re-insert them into the bronze layer. This effectively turns a failed batch into a manual correction workflow.
The most important systemic lesson? Don't let your infrastructure be the judge, jury, and executioner. DLT is excellent at moving data, but it is a terrible place to make business decisions about what constitutes "valid" financial data. If your pipeline fails because of bad data, your architecture is essentially admitting that it has no plan for reality.
In production, reality is messy, sources are unreliable, and your data quality gates should be designed to catch the mess, not crash under its weight. Stop dropping data. Start quarantining it. Your future self, sleeping soundly at 3 AM, will thank you.
Top comments (0)