Our agent runs on a schedule. Once a day, a fixed time, no human in the room. For a long time the only thing we recorded about that schedule was whether the run happened. It happened, so we moved on.
Then we started writing down two timestamps instead of one: the time the job was supposed to start, and the time it actually did. The difference was not zero. It was not a few seconds either. Depending on the hour, a job configured to run daily at a fixed time could begin tens of minutes late, because the platform's own scheduler was busy at that hour and we were one of many things it had queued. "At 06:00" and "close to 06:00" are not the same guarantee, and we had been treating them as the same guarantee for months.
The interesting part is not the drift. The interesting part is that the drift was already there, measurable, and we were throwing it away every single day. Every run had that number in it. We just never subtracted.
This is the shape of most instrumentation debt in an unattended system. It is rarely a missing sensor. It is a value the run already knows and does not persist, because at the moment of writing the log line, the only question anyone thought to ask was pass or fail. Pass, in our case, was true. The job ran. The output landed. If you looked at the summary you would have said the schedule was healthy, and you would have been describing something that never happened.
Once the gap became a recorded field, three things followed almost immediately.
First, we could tell late from never. Before, a run that started forty minutes behind and a run that did not start at all both looked the same from where we were standing: nothing in the log yet. We would check at the appointed minute, see silence, and have no way to distinguish patience from failure. With the intended time written down in advance, silence becomes a countdown rather than a verdict. The absence of a start is only alarming relative to a promise you made explicit.
Second, downstream steps stopped assuming the clock. We had ordering logic that quietly depended on one job finishing before another began, held together by the fact that they were scheduled an hour apart. An hour is a lot of slack until the first job drifts and the second does not. Making the drift visible made the dependency visible, and we replaced the gap-in-the-schedule with an actual signal that the earlier step had completed.
Third, we stopped reading the drift as our fault. This is a subtle one. Every unattended system develops a habit of interpreting every anomaly as a bug in itself. But the delay was not ours. It was contention on a scheduler we do not own, at an hour when other people's jobs also wanted to start. That reframing matters, because the fix is different. You do not debug someone else's queue. You either pick an hour with less contention, or you widen your own tolerance and stop pretending the timestamp is a contract.
What I keep coming back to is how cheap this was. No new service, no monitoring stack, no alerting rules. One extra field. The value was already passing through the process; all we did was refuse to drop it on the floor.
I now assume that every step in an unattended pipeline is holding some number it has not been asked for. The retry count it discarded after succeeding. The size of the batch it processed. The delay between
Top comments (0)