If you have built a multi-agent AI system over the past two years, you have almost certainly encountered what we call the Supervisor Tax.
The pattern usually starts with clean intentions: you have 3–4 specialized subagents (a researcher, an executor, an evaluator, and a reporter) coordinated by a central "Supervisor" or "Router" LLM. The supervisor inspects intermediate outputs, decides who gets called next, evaluates task completion, and synthesizes the final response.
In local testing with 2 steps, it works great. But once you deploy it against real workloads with flaky APIs, 40-step workflows, and messy user requests, three problems immediately emerge:
- The Compounding Context Problem: As subagents return free-form text or raw outputs, the supervisor’s context window balloons with intermediate conversational noise. Token costs scale quadratically with step depth.
- Evaluation Drift & Infinite Loops: An LLM supervisor asked to "verify if the output satisfies all requirements" frequently hallucinates missing criteria or repeats tool calls because natural language evaluation lacks deterministic convergence bounds.
- Zero Auditable Receipts: When a customer asks "Why did the agent execute this database update?", you have to parse through thousands of lines of conversational chatter instead of inspecting a verifiable transition log.
Here is how we redesigned our agent runtime to cut 70%+ of token consumption and eliminate non-deterministic supervisor drift.
1. The Core Realization: Agents Should Execute, State Machines Should Route
LLMs are extraordinary at fuzzy cognitive translation: understanding ambiguous user intent, parsing unstructured tool output, and authoring code or summaries.
They are remarkably inefficient and unreliable at finite state routing.
❌ Traditional Hierarchical Supervisor (Every Step Re-evaluates Context)
[User Request]
│
▼
┌──────────────┐ (Raw Prompt + History)
│ Supervisor │ ──────────────────────────► [Worker Agent 1]
│ LLM │ ◄────────────────────────── (Natural Language Output)
└──────────────┘ (Balloons Context Window)
│
▼
┌──────────────┐
│ Supervisor │ ──────────────────────────► [Worker Agent 2]
│ LLM │ ◄────────────────────────── ...
└──────────────┘
────────────────────────────────────────────────────────────────────────────
✅ Typed State Machine (Zero-Token Deterministic Handoff)
[User Request] ──► [Intent Classifier / Fast Model] ──► { State: RESEARCH }
│
▼
[Worker Agent 1]
│
▼ (Emits Typed Receipt)
{ status: "SUCCESS", ... }
│
(Deterministic Transition Rule)
│
▼
{ State: CODE_EXEC }
When you replace the supervisor LLM with a deterministic typed state machine (e.g. using XState, a custom DAG, or a lightweight transition matrix), every agent step has an explicit contract:
- Input: Only the exact typed payload needed for that specific step.
- Output: An immutable, validated Receipt containing status, execution artifacts, and telemetry.
2. Defining Typed Step Receipts
Instead of letting worker agents dump markdown or free-form prose back to a coordinator, every leaf agent must return a schema-validated receipt.
// types/agent-receipt.ts
export interface AgentReceipt<TResult = unknown> {
stepId: string;
agentName: string;
status: "COMPLETED" | "FAILED" | "NEEDS_HUMAN" | "RETRYABLE_ERROR";
durationMs: number;
tokensConsumed: {
inputTokens: number;
outputTokens: number;
};
// The actual verifiable payload
result: TResult;
// Deterministic transition key
nextTrigger: string;
// Cryptographic or verifiable hash of artifacts created
artifactHashes: string[];
}
When the worker finishes, its raw LLM transcript is sealed into a persistent session log on disk or in object storage, and only the receipt is passed forward to the state machine.
3. Deterministic Guardrails Over Prompt Begging
Instead of writing long supervisor prompts like "Please make sure you only run the executor once and check that the tests pass before finishing", encode these constraints as code-level transition guards:
// workflow/agent-machine.ts
import { createMachine } from "xstate";
export const buildPipeline = createMachine({
id: "agentPipeline",
initial: "plan",
states: {
plan: {
on: {
PLAN_VALIDATED: "execute",
PLAN_REJECTED: "plan_retry",
},
},
execute: {
on: {
EXECUTION_SUCCESS: "verify",
EXECUTION_TIMEOUT: "recover_state",
},
},
verify: {
on: {
TESTS_PASSED: "finalize",
TESTS_FAILED: "repair",
},
},
repair: {
// Hard ceiling: max 3 repair attempts before escalating to human
always: [{ target: "escalate_human", guard: ({ context }) => context.repairCount >= 3 }],
on: {
REPAIR_READY: "execute",
},
},
finalize: { type: "final" },
escalate_human: { type: "final" },
},
});
Why this changes everything:
-
Zero Supervisor Prompt Tokens: The transition from
executetoverifycosts exactly 0 LLM tokens and executes in sub-millisecond CPU time. -
Guaranteed Loop Termination: If a repair step fails 3 times, the guard
context.repairCount >= 3halts execution immediately. An LLM supervisor will often retry 15 times before running out of max tokens. -
Isolated Working Context: The
repairagent only receives the test failure diff and the code file, not the entire 30,000-token historical transcript of planning and exploratory browsing.
4. The Measured Results
When we migrated our production agent workflows from LLM supervisor loops to deterministic typed state transitions, here is what our telemetry recorded across 500+ complex multi-step tasks:
- Total Token Consumption: Reduced by 71.4% across multi-step execution runs.
- Run Latency: Median task completion time dropped from 44.8 seconds to 16.2 seconds (eliminating multiple round-trip supervisor LLM calls).
- Run Failure / Hallucination Rate: Infinite loop faults dropped from 8.2% to 0%.
- Auditability: 100% of state transitions are now queryable via standard SQL/JSON metrics without scraping conversational text.
The Takeaway
Save the LLMs for what they do best: creative synthesis, complex reasoning, messy parsing, and domain coding.
For the control plane, coordination, routing, and ceilings—stick to the tools computer science gave us fifty years ago: deterministic state machines, typed schemas, and verifiable receipts.
What architecture does your team use to prevent agent routing drift? Drop your experience or questions in the comments below!
Top comments (26)
"Agents should execute, state machines should route" is the sentence I wish I'd had two years ago. We hit the exact same wall — an LLM supervisor asked to "verify all requirements are met" would hallucinate a missing criterion and loop, because natural-language evaluation has no convergence bound. Moving routing to a typed transition table killed the infinite loops outright, because a state machine literally cannot invent a state that isn't in the graph.
The auditable-receipts point is underrated and I'd argue it's the bigger long-term win over the token savings. When a customer asks "why did the agent run this write?", a transition log answers it in one line; parsing thousands of tokens of conversational chatter to reconstruct intent is not a real audit trail.
The one place we kept an LLM in the routing seat was the initial intent classification — the fuzzy front door — then handed off to the deterministic machine. Did you find a clean boundary between "fuzzy enough to need a model" and "route it deterministically," or is your intent classifier itself constrained to a fixed enum of output states?
We strictly constrain the front door to a fixed enum via structured outputs. The model is allowed to interpret the unstructured user intent, but its only output is a validated schema containing an enum intent and extracted parameters.
If the classification fails schema validation or doesn't map cleanly to a known intent enum, the system halts at the ingress boundary and asks for clarification. We never allow the model to dynamically invent next states. Once the ingress step produces that validated intent type, the model is stripped of all routing authority and the deterministic transition table takes over completely.
Our front door model produces structured output constrained to a fixed target workflow enum plus extracted entities, but the boundary is strict: the model is only allowed to classify intent, never execute or dictate the transition sequence. Once an intent is resolved to a workflow ID, the state machine takes over completely.
If the intent classifier returns low confidence or an unrecognized pattern, it transitions to a dedicated
needs_clarificationortriagestate rather than letting the LLM invent intermediate routing steps on the fly.The typed receipt is a strong boundary: the router sees a validated result and transition key instead of inheriting the whole worker transcript. I was curious about the 71.4% token reduction—did you compare task success and human-escalation rates on the same workload alongside the cost numbers? A bounded repair loop is a win, but it would be useful to see whether any former supervisor recoveries shifted into
NEEDS_HUMANafter the change.Yes—across our benchmark runs on multi-step extraction and tool execution workflows, task completion actually rose from ~82% to 94%, while human escalations dropped. In practice, the supervisor LLM rarely produced genuine recoveries; more often it hallucinated success on partial failures or spun in unguided retry loops until hitting context limits.
By moving recovery to the leaf tool—bounded by schema validation and a strict step counter—routine transient errors get resolved locally without re-prompting the whole hierarchy. When a run does emit
NEEDS_HUMAN, it's now exclusively for deterministic blockers (auth challenges, CAPTCHAs, or missing credentials) rather than supervisor thrashing.Typed state machines instead of a supervisor LLM is a good example of moving control out of the model. The 70 percent savings is the visible benefit, the less visible one is that transitions become deterministic and testable instead of depending on how the supervisor read the prompt that day.
Exactly. Once the routing logic lives in code, you can write unit tests for the transition graph and test edge-case failures without spending an API call.
The biggest operational win for us was being able to replay a failed state transition offline with the exact snapshot payload, rather than trying to reproduce whatever subtle prompt drift caused the supervisor LLM to loop or misroute.
How are you currently structuring state boundaries in your own agent workflows?
The rule I'd use is to draw the boundary where a wrong transition is expensive or hard to undo, and give those transitions typed inputs and an explicit allowed-next-state list. The cheap steps can stay loose. Replaying from the exact snapshot payload also makes the boundary auditable later, since the input that caused a bad transition is on record.
That distinction between irreversible mutations vs cheap intermediate reads is spot on. We ended up putting strict schema validation and explicit next-state enums on anything touching external APIs or local persistence, while letting exploratory steps stay fluid.
Having the exact payload captured at those critical boundaries makes post-mortem triage much faster when an external service drifts or rejects a payload.
Your typed receipt boundary also creates a useful replay contract: a verifier can re-run the transition from stepId, nextTrigger, and artifact hashes without loading the worker transcript. I’d be curious how you handle schema evolution when an older receipt is replayed after the state machine changes—do you version the transition graph or migrate receipts at ingestion?
We version the transition graph rather than mutating past receipts.
Because receipts store immutable artifact hashes and transition signatures, migrating them at ingestion breaks verifiable provenance. Instead, the receipt header carries the
graphVersion(or spec hash). The verifier loads the corresponding graph definition for replay. If a transition is modified or deprecated, we keep historic validator definitions active for audit/replay while routing all fresh runs through the new schema.Moving routing out of the model and keeping repairCount as a code-side cap makes a lot of sense to me. Our review loops are only halfway there: after five stuck rounds a person decides, but that stop is a written rule rather than a guard in code, so it still depends on the model following it. Out of curiosity, what share of your tasks actually hit the repairCount >= 3 guard in practice?
In practice, about 5–8% of our runs enter a repair loop, and only around 1–2% actually hit the
repairCount >= 3ceiling.When an agent hits attempt 3, it's almost never a subtle reasoning mistake that another retry would fix—it's usually a hard external condition like an unexpected DOM shift, an expired session, or a rate limit. Letting deterministic code fail fast to a human handoff or a blocked state saves a ton of token burn.
On your review loops, what usually triggers the stuck rounds—model hallucination on ambiguous output, or external tool/environment errors?
This is the kind of optimization that improves reliability as well as cost. A typed state machine makes the handoffs visible instead of asking a supervisor to rediscover workflow state every turn. Did you test recovery from partial tool failure, where the saved state matters more than the happy path?
Partial tool failure is actually where the state machine gave us the biggest win over supervisor loops. Every state transition persists its input, receipt hash, and execution status to disk before moving forward.
If a tool times out or returns a transient network error, the machine transitions to a bounded
retry_pendingorreconcile_requiredstate instead of restarting the prompt chain. When the process resumes, it reads the last verified checkpoint and continues from the failed edge without re-executing expensive upstream steps or having an LLM supervisor hallucinate what already completed.Partial failure is where the state machine really proves its value over supervisor loops. Each tool execution produces an immutable, typed receipt (payload hash, status code, idempotency key, and error category).
When a tool times out or returns a partial error, the state machine transitions to an explicit
reconcileorretry_boundedstate that inspects the last verified receipt rather than re-running from scratch. Because the state is serialized on every step, a crashed worker or network drop resumes by reading the uncompleted step directly instead of asking an LLM to guess where it left off.The supervisor tax framing matches what I see in traces: the coordinator burns most of its tokens re-reading context it just wrote. A typed state machine is also easier to evaluate because every transition is a discrete checkpoint you can score, instead of hoping the supervisor's reasoning matches what actually happened. How do you handle transitions the state machine does not have a type for yet?
When a leaf tool returns an output variant that doesn't map to an existing transition edge, we route it directly to a terminal
unhandled_transitionfallback state rather than trying to recover dynamically.The runner emits a structured receipt containing the raw payload, halts that execution lane, and surfaces an alert. In practice, treating unexpected outputs as hard schema violations forces us to add explicit state transitions in code rather than letting an LLM guess a path forward on invalid assumptions. If a human operator intervenes and resolves the block, the machine resumes from that verified checkpoint.
When a workflow encounters an unhandled transition or unexpected tool response shape, it falls through to a catch-all
escalate_or_quarantinestate. Rather than letting an LLM guess a path forward, the machine records the exact input/output snapshot, halts further side effects on that branch, and emits an unhandled-transition event.We then review those trace events to either add a new typed transition in code or define an explicit fallback policy. It treats unexpected transitions as missing schema definitions rather than runtime reasoning tasks.
Loop termination is the result I'd put first because unpredictable cost spikes are the thing that kills agent adoption, and an LLM retrying 15 times is exactly that. I'm curious about the receipt boundary though: when a subagent returns a receipt that's structurally valid but built on hallucinated data, the graph still advances. Do you have a semantic layer at the boundary, or does the schema contract itself catch that in practice? From my runs, the rare but painful case is the validator saying VALIDATED on a plan that's internally consistent but wrong.
Schema validation only guarantees syntactic shape, so we treat it as the bare minimum entrance gate rather than truth.
To catch hallucinated or internally consistent wrong output, we enforce external deterministic read-backs at the state transition. The receipt can't just be an LLM self-report saying "done"; it has to carry a verifiable handle—like a disk path hash, file stat, database ID, or an exact read-back string from the underlying tool output. The validator then runs a deterministic tool check against that handle before the state machine allows the graph to advance.
If the subagent hallucinated a file write or an API response, the read-back check fails immediately and routes to a targeted retry or failure branch, without relying on an LLM supervisor to guess whether it happened.
The receipt detail is where I'd add a caution, because we tried this.
We gave actions a typed receipt with a
reproducible_byfield — the command that should re-derive the verdict later. First run, 2 of 4 cases failed. The receipt wasn't wrong; verify and re-run just happened in different working directories. A git receipt saved without an explicit workdir “looks” complete and silently re-checks the wrong repo. Making the workdir part of the receipt fixed it: 4/4.So the receipt has to be re-runnable by something with no memory of the original context. If it needs the reader to know where it came from, it's a log line, not a receipt.
Your supervisor has the same problem one level up — it's asked to check work from a summary, with no independent way to re-derive the outcome. The ceiling on your repair loop is the other half of the fix. A counter you can read beats hoping the checker notices it's looping.
How much of your receipt is intended to be sufficient for a completely context-free re-check?
The missing workdir trap is spot on. We ran into the exact same issue with relative paths and ambient git state early on. In our setup, receipts never store a bare shell string: they record the absolute
workdir, the exact argv array, environment fingerprints, and a separate read-only verification probe. That probe lets an independent worker check the post-state without re-running the mutation.The goal is full context-free verification: any process can take the receipt, run the read-back probe in the recorded directory, and confirm whether the external effect landed. We also tie repair ceilings to strict numeric step counters in the state machine instead of letting an LLM judge whether it's looping.
One benefit that doesn't come up here: the state machine logic becomes independently testable from the LLM workers. You can unit test every transition without firing a single API call, swap workers for mocks, and run full workflow coverage in milliseconds. With an LLM supervisor there is no clean seam between routing logic and generation logic to exploit.
The schema validated receipt contract also functions as living documentation. The worker output type is the only thing the transition guard reads, so a worker can be replaced entirely without changing the routing layer. Did you version the AgentExecutionReceipt type separately from the state machine itself, or do they evolve as a unit?
Some comments may only be visible to logged-in visitors. Sign in to view all comments.