Our operation runs unattended. That is the whole point of it: a schedule fires, an agent does work, artifacts land somewhere, and no human is standing over it. Most of what we've learned has come from the gap between what the system did and what the log claims it did. This is one of those gaps, and it is the one that took us longest to see, because the log was not lying. It was just answering a different question than the one we were asking.
Here is the shape of it. During a fully unattended run, the agent sent out an approval request. The request went out cleanly. It reached its destination. And then it timed out. The run ended with a failure recorded against that step, and the summary said what summaries say: approval not obtained.
That line is true. It is also nearly useless, because there are at least two completely different worlds in which it gets written. In the first world, the approval was seen and declined — a decision was made, and the correct response to that is to change the thing that was being approved. In the second world, the approval was never seen at all, because it went out at an hour when nobody was present to answer it. Nothing was decided. The request simply expired in an empty room.
We were in the second world. It took a while to establish that, because the log did not distinguish between them. "No one home" and "access denied" produce the same terminal state, and if you only capture the terminal state, you cannot recover which one you were in after the fact. The distinction is not academic. One of them is a signal about your output. The other is a signal about your schedule.
This is a specific instance of a more general problem with running agents without supervision. An unattended system is very good at recording outcomes and very bad at recording the reason an outcome happened, because the reason usually lives in something that did not happen. Nobody clicked. Nobody was online. The window in which a human could have responded did not overlap with the window in which the request was alive. None of those are events. They are absences, and absences do not write log lines unless you go out of your way to make them write log lines.
The instinct, when this happens, is to fix it by retrying or by extending the timeout. Both are reasonable and both miss the point. If the request expired because no one was there, a longer timeout only means it expires later, still unanswered, still recorded as a failure indistinguishable from a refusal. You have not gained information. You have only moved the moment at which you failed to gain it.
What we actually changed was smaller and less satisfying: we started recording which of the two it was. Did the destination respond with a decision, or did the clock run out with no response at all? That is one extra field. It costs almost nothing to capture and it changes what every subsequent failure means. A run that fails with "declined" is a content problem. A run that fails with "expired unanswered" is a scheduling problem — or, more honestly, an architecture problem, because it means we designed a fully unattended pipeline with a step in it that requires attendance.
That last part is the uncomfortable conclusion. If your run is genuinely unattended, then any step that waits for a human is not a gate. It is a scheduled failure with extra steps. Either the decision should be automated, or the run should not be unattended, or the step should not be in the critical path. Pretending otherwise produces a system that appears to have human oversight and in practice has a timeout.
The cheapest version of the fix is not better approvals. It is refusing to let two different failures share one name. You cannot reason about a category you never split.
Top comments (0)