A production incident, broken down using a five-step failure framework: symptom, root cause, blind spot, fix, principle.
The symptom
A scheduled batch job completed successfully. No errors, no failed retries, no alerts. The next morning, downstream reports were wrong not missing data, just incorrect values, in a way that initially looked like a data quality issue rather than a code issue.
Where we looked first (and why it was wrong)
Because the job showed as successful, the first hour of investigation focused on the reporting layer and the data pipeline. That was the wrong layer. The job hadn't failed it had run correctly, just against the wrong time window.
Root cause
A hardcoded UTC offset had been written into the job's config years earlier, before daylight saving was ever a relevant factor for this process. It had never been revisited since.
The blind spot
This is the step most postmortems skip. The real question isn't just "what broke," it's "why did nobody catch this." In this case: the config hadn't changed in years, so it had quietly fallen out of everyone's mental model of "things that could break." Stability had been mistaken for safety.
The fix
Two parts. The immediate fix replaced the hardcoded offset with a proper timezone-aware calculation. The more important fix added a synthetic check that verifies the job executed within the correct local time window, not just that it executed at all closing the gap between "ran" and "ran correctly."
The principle
If a system's correctness depends on time, don't just trust the clock. Verify the calendar too. Time-based assumptions in long-lived scheduled jobs are one of the most common invisible failure classes in enterprise systems, precisely because they're stable for years right up until they aren't.
This is the first post in a recurring series where I break down real production incidents using this same five-step structure. If you've hit a similar time-based bug, I'd genuinely like to hear how it surfaced for you.
Top comments (0)