Every agent stack has one place where a sentence gets classified as work or as a question. Almost nobody designs the miss. Mine missed like this: the planner's trigger heuristic saw a schedule word or a workflow word, built a plan, emitted a plan card, and returned. The comment above that path said the worst case was a card the user ignores. The comment was wrong. The worst case is a card the user ignores with nothing underneath it, because the sentence that fired the heuristic was never answered.
The sentence that proved it was "I set up an automation yesterday, what did it do?" That is a question about an automation, not a request for one. It got a plan card and no answer, and the only escape was to rephrase the question around the trigger word. Making a user reverse-engineer your keyword list to get an answer is a bug, even when every individual component behaved as written.
The exit is the ordinary path with the offer suppressed for one turn
I built this into Vodou, where workflows are plan-first by design: you describe multi-step work in a sentence and get resolved tools, parallel steps, and an approval gate before anything sends. Plan-first is the right default. It just needs a way out when the plan was a wrong guess.
The fix is one button on the plan card, "Just answer it", backed by one flag: ChatOptions.skipGraphOffer, threaded through every entry point the offer can reach (the REST /chat endpoint, the web socket message frame, and the panel's send path). The button re-sends the same sentence through the ordinary chat path with the offer suppressed for that turn only. It is deliberately not a special answering path. The answer is whatever it would have been had the guess not fired. The flag is never persisted; the next sentence gets the offer again. And an explicit /workflow command ignores the flag entirely, because the user named that intent, and a failure there should be reported, not sidestepped. The one structural change this forced: the plan event now carries the original sentence, because a button has to have something to re-send.
The wrong guess, and the way out
Eleven of the twelve commits were what the button flushed out
The feature was one commit. Shipping it took twelve, across 29 files in two days, because putting a real exit on the plan card meant driving the plan flow on every surface, and every surface was quietly wrong.
Telegram first. Every plan arrived twice, and the second copy was corrupted: it read get_cry_info and was missing a step. I assumed a chunked resend. It was not. The plan was 1,718 characters against a 3,900 limit, one dispatch, one forward, and the stored reply already contained two plans. The channel fan-out rendered every graph event as text so surfaces without cards could see them, and the plan's text echo carried no marker saying it was an echo. The same disease hit the approval ask one event later: the numbered menu is the text echo of the ask exactly as the plan text is the echo of the plan, and card surfaces showed both. Three emitters now flag their echoes, and a test pins all three, because one unflagged path silently brings the duplicate back on whichever surface hits it.
The plan was also built from the wrong words. A user asked a 14-word question about cpu and memory, and both steps resolved to a channel-status tool. The recipe author had been handed the entire wrapped channel turn, roughly 700 characters of untrusted-channel framing around the sentence, and the word "channel" in the wrapper out-ranked "cpu" in the tool catalog. The author now gets the unwrapped human words, the same unwrap our recall path had used for a year for the same reason. And a channel had no button at all, so a plan offered on Telegram was built and unreachable. The plan text now ends "Reply run to run it", and a bare "run" enters through the same driver path as the button, so the approval gate is enforced by the same code either way.
"2 no add it to #alpha-testing" reached a model as leftover words
Then the one that scared me. Live, a user answered "2 no add it to #alpha-testing" to "post the summary to slack? 1. Yes 2. No". A model then replied, in a confident table, that the approval gate was now OFF and future posts would go to #alpha-testing automatically. It had changed nothing. The run was closed, the real recipe was still gated, nothing had posted. A model narrated a safety change that did not happen, which is worse than the change itself, because the user now believes the gate is off and cannot see that it isn't.
The log said user selected option 2: No. The match was correct. Three defects lined up behind it. First, option 2 has no steps, so the answered run returned an empty result, and the chat dispatcher treats an empty workflow result as "nothing happened" and forwards the user's original message to a model. Second, an unmatched reply re-showed the menu only for skill-defined workflows; an ad-hoc graph fell through to return null, which the dispatcher reads as "not a menu reply", which also means a model. Third, the matcher itself used a substring test that would select option 1 for any reply containing a "1". The fixes: a zero-step answer returns a verbatim acknowledgement in a shape the dispatcher streams as-is, every parked menu re-shows itself, and the matcher got exact rules. The regression test is public, in MCP-servers/Vodou-Console/src/__tests__/workflow-choice-b16.test.ts. Two smaller ones from the same sweep: a declined gate had been leaving its run record running forever, and the answer endpoint logged nothing on success, so proving the fix meant proving a negative from the database alone.
Handled-empty and not-handled must be distinct values before any model fallback
The transferable class is not "validate replies harder". It is this invariant: in any dispatcher whose fallback is a model, "handled with empty output" and "not handled" must be represented by different values. If a handler can signal both with the same null or empty result, then every legitimately empty outcome (a No answer, a declined gate, a cancel) hands the user's raw words to the most fluent component in your system, which will do what fluent components do: answer them. This is checkable, not aspirational. Either your gate handler has an explicit "handled, nothing to run" value that short-circuits the model, or it does not, and you can go read the function and find out in five minutes.
Two probes to run on your own router this afternoon
Nothing here needs our stack. First, the stuck-run probe, against whatever table holds your workflow runs:
SELECT id, status, created_at
FROM runs
WHERE status = 'running'
AND created_at < datetime('now', '-1 day');
Passing looks like zero rows, or rows with a live worker attached. Failing looks like rows whose approval was declined weeks ago and whose status never moved, which means your decline path exits before it closes the record.
Second, the leftover-words probe. Trigger any flow that parks on a yes/no gate, then answer with a matched option plus extra words:
curl -s localhost:8080/chat -H 'content-type: application/json' \
-d '{"message": "2 no but also send it to the other channel"}'
Then count model invocations for that turn in your trace. Passing is an acknowledgement of option 2 and zero model calls. Failing is a fluent reply about your own system's configuration, and if you get one, do not skim it: check whether anything it claims actually changed. Mine claimed a safety setting had flipped. It had not.
The plan-or-answer line is in every framework; the miss path is in none
The classification itself is table stakes. It appears verbatim as a prompt in a minimal two-node agent example: "If it's a coding task, output a step-by-step PLAN. If it's a question, just ANSWER it." The LangGraph literature covers pausing for human input when risk warrants it, and recent work on agentic abstention studies when an agent should stop acting entirely. What none of it designs is the recovery when the classifier fired wrong and the human is looking at the wrong artifact: the exit that re-enters the ordinary path instead of adding a repair path. And Anthropic's guidance on writing tools for agents argues for evaluating tools against real tasks; the router in front of the tools deserves the same treatment, because my trigger heuristic had never once been evaluated against a fixture set of questions about automations.
The detector still guesses on keywords
Still true after all twelve commits: the heuristic fires on trigger words, so a question about a workflow still gets a plan card first, and the button makes the wrong guess cost one click instead of a rephrase. That is a cheaper miss, not a smarter classifier. The suppression is also strictly per-turn, so a user who asks two questions about automations clicks the button twice. Phase 1 of workflows is what shipped (the docs are public in docs/workflows.md); the later phases are open.
If you want the plan-first version of this instead of building it, Vodou on vodou.ai gives you workflows that show resolved tools, parallel steps and an approval gate before anything sends, with the exit described here already wired in.
Source: A model told my user the approval gate was off. It wasn't. by Chad Priest, from Building Vodou in Public.
Top comments (0)