Our unattended agent has a daily slot. It is written down as a fixed time in the morning, it is configured as a fixed time in the morning, and for months I described it to people as "it runs at six." Then I started logging the delta between the time the job was scheduled for and the time the first line of work actually appeared, and the number was not zero. It was rarely zero. Depending on the day, the job started tens of minutes after the time on the schedule, because the platform's own scheduler was busy at that hour and we were one of many things it had queued up.
Nothing was broken. The job ran. The work got done. But "at six" and "close to six" are not the same guarantee, and I had built several things on top of the stronger version of that sentence without noticing I had assumed it.
The first thing that assumption leaked into was reasoning about time inside the run itself. An agent that wakes up and asks "has today's work already happened?" needs a way to answer that, and the cheap way is to compare against a wall clock boundary. If your job believes it starts at six, a window that opens at six is safe. If it actually starts at six forty, and some other check compares against a window that closed a moment ago, you get a run that concludes it is late for yesterday rather than early for today. The logic was never wrong in isolation. It was wrong about its own start time, which is a much harder thing to see in a code review, because the start time does not appear anywhere in the code. It lives in a scheduler configuration on someone else's machine.
The second leak was in how I read the logs. When a run appears in the record, the timestamp on the first line is the time it started, not the time it was supposed to start. If you never write down the intended time, the log has no way to tell you about drift. Every run looks punctual, because every run is stamped with the moment it actually woke up. You can stare at a month of that and conclude the schedule is perfectly reliable, when what you are really looking at is a month of the schedule reporting its own behaviour as the definition of correct. The only way out is to record both numbers and subtract them, which costs almost nothing and immediately turns an invisible property into a visible one.
The third leak was expectations set for humans. When I told people the slot fires at six, they built their own habits around it. Someone checking at five past six and finding nothing has to decide whether the job is late or dead, and there is no signal in the system that distinguishes those two states. A run that has not started yet and a run that will never start look exactly alike from the outside, which is the same shape of problem as every other ambiguity in unattended operation: two very different conditions producing one identical observation.
What I changed is small. The intended time is now carried into the run and logged next to the actual time. The drift is a number we can look at over weeks rather than a feeling. Anything downstream that reasons about "today" uses a tolerance window wide enough to survive a busy scheduler instead of a boundary that assumes precision nobody promised us. And when I describe the slot to someone now, I say it runs in the morning, usually within an hour of six, because that is the true statement and the tighter one was always a guess dressed as a fact.
The lesson generalises past cron. Every timing guarantee an unattended system relies on is somebody else's best effort until you measure it. The measurement is usually one subtraction. The assumption, unmeasured, will quietly shape logic three layers away from where it lives.
Top comments (0)