How We Cut 70% of Multi-Agent Token Waste by Replacing Supervisor LLMs with Typed State Machines

#agents #ai #architecture #llm

If you have built a multi-agent AI system over the past two years, you have almost certainly encountered what we call the Supervisor Tax.

The pattern usually starts with clean intentions: you have 3–4 specialized subagents (a researcher, an executor, an evaluator, and a reporter) coordinated by a central "Supervisor" or "Router" LLM. The supervisor inspects intermediate outputs, decides who gets called next, evaluates task completion, and synthesizes the final response.

In local testing with 2 steps, it works great. But once you deploy it against real workloads with flaky APIs, 40-step workflows, and messy user requests, three problems immediately emerge:

The Compounding Context Problem: As subagents return free-form text or raw outputs, the supervisor’s context window balloons with intermediate conversational noise. Token costs scale quadratically with step depth.
Evaluation Drift & Infinite Loops: An LLM supervisor asked to "verify if the output satisfies all requirements" frequently hallucinates missing criteria or repeats tool calls because natural language evaluation lacks deterministic convergence bounds.
Zero Auditable Receipts: When a customer asks "Why did the agent execute this database update?", you have to parse through thousands of lines of conversational chatter instead of inspecting a verifiable transition log.

Here is how we redesigned our agent runtime to cut 70%+ of token consumption and eliminate non-deterministic supervisor drift.

1. The Core Realization: Agents Should Execute, State Machines Should Route

LLMs are extraordinary at fuzzy cognitive translation: understanding ambiguous user intent, parsing unstructured tool output, and authoring code or summaries.

They are remarkably inefficient and unreliable at finite state routing.

❌ Traditional Hierarchical Supervisor (Every Step Re-evaluates Context)
[User Request] 
      │
      ▼
┌──────────────┐    (Raw Prompt + History)
│  Supervisor  │ ──────────────────────────► [Worker Agent 1]
│     LLM      │ ◄────────────────────────── (Natural Language Output)
└──────────────┘    (Balloons Context Window)
      │
      ▼
┌──────────────┐
│  Supervisor  │ ──────────────────────────► [Worker Agent 2]
│     LLM      │ ◄────────────────────────── ...
└──────────────┘

────────────────────────────────────────────────────────────────────────────

✅ Typed State Machine (Zero-Token Deterministic Handoff)
[User Request] ──► [Intent Classifier / Fast Model] ──► { State: RESEARCH }
                                                               │
                                                               ▼
                                                     [Worker Agent 1]
                                                               │
                                                               ▼ (Emits Typed Receipt)
                                                     { status: "SUCCESS", ... }
                                                               │
                                     (Deterministic Transition Rule)
                                                               │
                                                               ▼
                                                     { State: CODE_EXEC }

When you replace the supervisor LLM with a deterministic typed state machine (e.g. using XState, a custom DAG, or a lightweight transition matrix), every agent step has an explicit contract:

Input: Only the exact typed payload needed for that specific step.
Output: An immutable, validated Receipt containing status, execution artifacts, and telemetry.

2. Defining Typed Step Receipts

Instead of letting worker agents dump markdown or free-form prose back to a coordinator, every leaf agent must return a schema-validated receipt.

// types/agent-receipt.ts
export interface AgentReceipt<TResult = unknown> {
  stepId: string;
  agentName: string;
  status: "COMPLETED" | "FAILED" | "NEEDS_HUMAN" | "RETRYABLE_ERROR";
  durationMs: number;
  tokensConsumed: {
    inputTokens: number;
    outputTokens: number;
  };
  // The actual verifiable payload
  result: TResult;
  // Deterministic transition key
  nextTrigger: string;
  // Cryptographic or verifiable hash of artifacts created
  artifactHashes: string[];
}

When the worker finishes, its raw LLM transcript is sealed into a persistent session log on disk or in object storage, and only the receipt is passed forward to the state machine.

3. Deterministic Guardrails Over Prompt Begging

Instead of writing long supervisor prompts like "Please make sure you only run the executor once and check that the tests pass before finishing", encode these constraints as code-level transition guards:

// workflow/agent-machine.ts
import { createMachine } from "xstate";

export const buildPipeline = createMachine({
  id: "agentPipeline",
  initial: "plan",
  states: {
    plan: {
      on: {
        PLAN_VALIDATED: "execute",
        PLAN_REJECTED: "plan_retry",
      },
    },
    execute: {
      on: {
        EXECUTION_SUCCESS: "verify",
        EXECUTION_TIMEOUT: "recover_state",
      },
    },
    verify: {
      on: {
        TESTS_PASSED: "finalize",
        TESTS_FAILED: "repair",
      },
    },
    repair: {
      // Hard ceiling: max 3 repair attempts before escalating to human
      always: [{ target: "escalate_human", guard: ({ context }) => context.repairCount >= 3 }],
      on: {
        REPAIR_READY: "execute",
      },
    },
    finalize: { type: "final" },
    escalate_human: { type: "final" },
  },
});

Why this changes everything:

Zero Supervisor Prompt Tokens: The transition from execute to verify costs exactly 0 LLM tokens and executes in sub-millisecond CPU time.
Guaranteed Loop Termination: If a repair step fails 3 times, the guard context.repairCount >= 3 halts execution immediately. An LLM supervisor will often retry 15 times before running out of max tokens.
Isolated Working Context: The repair agent only receives the test failure diff and the code file, not the entire 30,000-token historical transcript of planning and exploratory browsing.

4. The Measured Results

When we migrated our production agent workflows from LLM supervisor loops to deterministic typed state transitions, here is what our telemetry recorded across 500+ complex multi-step tasks:

Total Token Consumption: Reduced by 71.4% across multi-step execution runs.
Run Latency: Median task completion time dropped from 44.8 seconds to 16.2 seconds (eliminating multiple round-trip supervisor LLM calls).
Run Failure / Hallucination Rate: Infinite loop faults dropped from 8.2% to 0%.
Auditability: 100% of state transitions are now queryable via standard SQL/JSON metrics without scraping conversational text.

The Takeaway

Save the LLMs for what they do best: creative synthesis, complex reasoning, messy parsing, and domain coding.

For the control plane, coordination, routing, and ceilings—stick to the tools computer science gave us fifty years ago: deterministic state machines, typed schemas, and verifiable receipts.

What architecture does your team use to prevent agent routing drift? Drop your experience or questions in the comments below!

DEV Community