Every data pipeline that runs long enough eventually hits the same failure, and it's rarely the one teams spend the most time defending against. It's not the crash, the timeout, or the malformed record that trips a parser. It's the run that completes cleanly, reports success, and quietly processes nothing at all, because something upstream changed in a way that looked like "no work to do today" instead of "something is broken."
Why This Failure Mode Is So Easy to Miss
Most pipeline monitoring is built around catching exceptions: a job dies, an alert fires, someone gets paged. That model works well for the failures it was designed for. It has nothing to say about a job that runs to completion without a single exception, because from the process's point of view, nothing went wrong. An API returned a 200 with an empty payload. A query executed successfully and returned zero rows. A file existed and was empty. Every one of these is a legitimate response, not an error, which means none of them trips a typical alert.
A Concrete Version of This Failure
Picture a nightly job pulling "records updated since the last successful run" from a source system. If a timezone conversion bug pushes that filter's timestamp slightly into the future, the query legitimately returns zero rows every single night, forever, until someone notices the downstream table hasn't grown. The job isn't malfunctioning by any definition the code itself can check. It's asking a well-formed question that happens to have a broken premise, and a well-formed question with zero results looks identical to a genuinely quiet day.
The Fix Requires an External Reference Point
The job itself has no way to distinguish "genuinely nothing happened" from "something's broken and returned nothing." That distinction can only come from outside the job's own logic, a baseline of what a healthy run typically produces. A rolling median of row counts over the trailing week, with an alert threshold set at some fraction of that median, catches the overwhelming majority of these failures without requiring anything close to a full anomaly-detection system.
recent_counts = get_last_n_runs(job_name, n=7)
baseline = statistics.median(recent_counts)
if today_count < baseline * 0.2:
alert(f"{job_name} processed {today_count} rows, expected roughly {baseline}")
That's the entire mechanism in its simplest form. It doesn't need to be smarter than this to catch the failure class that matters most.
Watermarks Catch a Different Shape of the Same Bug
Row counts catch a job that processed far less than usual. They don't catch a job that keeps reprocessing the same static window, technically producing a plausible row count on every run while never advancing. Logging a watermark, the last timestamp or ID successfully processed, and alerting when it stalls closes that second gap. Together, a row-count floor and a watermark check cover both the job that goes quiet and the job that's stuck spinning in place.
An observability layer like Sentry can be configured to treat an anomalous response shape, not just a thrown exception, as an event worth capturing, which is useful for catching the upstream side of this: the API or database that started returning empty results without ever technically erroring.
Message Queues Have Their Own Version of This
If your pipeline is built around a message queue rather than a batch query, the equivalent failure looks like consumers that stay connected, report healthy, and simply stop receiving messages because a topic subscription silently detached or a partition assignment shifted. Apache Kafka exposes consumer lag metrics specifically to catch this, and treating rising, unexplained lag as an alertable condition, the same way you'd treat a thrown exception, closes a very similar blind spot in a different architecture.
"The bug that scares me most isn't the one that throws. It's the one that returns a perfectly valid empty result to a perfectly valid question, because nothing about that looks wrong from inside the code." - Dennis Traina, founder of 137Foundry
Data Validation as a Complementary Layer
Beyond volume and watermark checks, validating the shape and plausibility of the data itself, not just its quantity, catches a related class of failure: a job that produces a normal-looking row count but with corrupted or nonsensical values inside those rows. The discipline of data validation as a formal practice predates modern pipelines by decades, and the same underlying principle applies just as directly to a scheduled automation job as it does to a form submission.
Why This Fails Code Review Even When It Shouldn't
Part of why this bug survives so long in production is that it's genuinely hard to catch in code review. A reviewer reading the query or the API call in the pull request sees correct, well-formed code. Nothing about the diff itself is wrong. The bug only manifests against real production data, at a specific point in time, in a way that no amount of careful reading of the code in isolation would reveal. This is a category of bug that unit tests with synthetic data also tend to miss, since the test fixtures rarely include the exact edge case (an expired token, a shifted timezone, a stale cache) that triggers the failure in production.
That's a strong argument for treating volume and freshness monitoring as a required production safeguard rather than something a sufficiently careful review process should have caught. Some categories of bugs are just not visible from the code alone.
How Long These Failures Typically Run Before Discovery
Across the client pipelines we've audited, silent failures in this category have run anywhere from a few days to several months before anyone noticed. The variable that predicts discovery time isn't the severity of the underlying bug, a completely broken filter and a subtly wrong one both look identical from outside. It's whether anything downstream of the pipeline was being actively, carefully watched. A dashboard checked daily by someone who knows what normal looks like catches this faster than one glanced at occasionally. A metric nobody has a strong intuition for can go wrong for a long time without anyone noticing the absence of change.
That's precisely why an automated volume check outperforms human vigilance here: it doesn't get tired, doesn't get used to a number being wrong, and doesn't need to remember what normal looked like three weeks ago.
Building This Into New Pipelines by Default
The teams that handle this well don't treat volume checks as an afterthought bolted onto a job after an incident. They treat "and produced a plausible amount of output" as part of a job's contract from the day it ships, alongside error handling and retry logic. That's the standard 137Foundry applies on client automation work, because retrofitting this check after a silent failure has already run for weeks is a much bigger job than building it in from the start.
A Quick Checklist Before You Ship the Next Job
Before marking a new scheduled job as production-ready, it's worth running through a short list: does it log a count of what it processed, is there a baseline it can be compared against after a couple of weeks in production, does something alert if that count falls far below expectation, and does something separately alert if the job stops running at all. Four short questions, and a job that can answer yes to all four has closed most of the gap this article describes.
The Takeaway
The failure mode described here doesn't announce itself, which is exactly what makes it worth defending against deliberately rather than reactively. A rolling baseline, a watermark check, and validated response shapes together close most of the gap between "the job didn't crash" and "the job actually worked." For a deeper walkthrough with more failure examples, see the full guide on catching silent pipeline failures.
Top comments (0)