Every unattended run we operate has a dependency graph, and for a long time that graph only contained things that run on machines. The publishing endpoint. The storage layer. The scheduler. The model provider. Each one got a name, a timeout, and a line in the log when it misbehaved.
Then one of our runs failed on an approval step, and the failure taught us that the graph was incomplete.
The run had reached a point where it wanted a human decision. It sent the request out, waited, and gave up. The summary line said the step failed. Reading that line the next morning, there was exactly one thing you could conclude: the approval did not happen. What you could not conclude was why. The request may have been rejected on the other end. Or it may have arrived perfectly, sat in front of nobody at all, and expired because the run was scheduled for a time when no human was anywhere near it.
Those are entirely different failures. One says the work was judged and found wanting. The other says the work was never judged. In the log they were the same sentence.
This is the part that stayed with us: the second failure was not a bug in the approval system. The approval system worked. The request was delivered, the channel was live, the credentials were good. What was missing was a person, and a person is not something you can provision by fixing a config value. We had built a step whose success depended on human availability, and we had never written that availability down anywhere. We scheduled the run for a time that suited the machines and then acted surprised when the one non-machine participant was not on shift.
So we added reviewers to the dependency list. Not as a joke, as an actual entry with the same fields the other dependencies get. Who is the responder. What hours are they realistically reachable. What happens when the wait expires. It felt strange to write an availability window for a human next to a connection timeout for a database, but the two are doing the same job in the graph: they tell you when a call is likely to be answered, and they let you plan for when it is not.
The second change was smaller and more important. We stopped letting the step collapse both outcomes into one word. A timeout with no response and an explicit refusal now write different results, because the difference is only observable at the moment it happens. Later, reading the log, there is nothing left to distinguish them. You cannot reconstruct an absence. If you did not record that nobody answered, all you have is the fact that the run did not proceed, and you will spend the morning investigating a rejection that never occurred.
Once the two were separated, the pattern in them became readable. Refusals are information about the work: the draft was wrong, the parameters were off, something needs fixing upstream. Unanswered requests are information about the schedule: we asked a question during hours when asking was pointless. The first kind belongs to whoever writes the pipeline logic. The second belongs to whoever chose the run time. Mixing them meant nobody could act on either.
What this really exposed is a design question we had been avoiding. If a run is genuinely unattended, every step that requires a human is a step that can only succeed by luck. You either move the question to a time when someone is there, or you decide in advance what the run should do on its own when nobody is, and write that down as the behavior rather than discovering it as a failure.
We have not removed every human checkpoint. Some of them earn their place. But now each one
Top comments (0)