Reliable AI Agent Control Flow: Keep the State Machine Out of the Prompt

Picture the failure that keeps me up at night. An agent reports that a job failed. The job did not fail. The work went through cleanly, every field extracted, the output sitting right there, correct. And the agent routed itself to the error state and stopped, calm as anything, as if it had done its job. There is no diff to look at. The code did not change. The config did not change. The transition that misfired lives in neither place. It lives in the prompt, as a few lines of English telling the model which state may follow which, and somewhere upstream the model that reads those lines got quietly updated. The machine you deployed is not the machine you are running. Nobody touched it. It drifted out from under you.

I have not had that exact night land on me, and I am writing this so that it never does. But it is not a hypothetical I had to strain to imagine, because the ingredients are sitting in plain sight in agent codebases everywhere. Once you see the shape of it, you stop being able to unsee it, and I landed on a boundary I now treat as non-negotiable: the state machine that governs which step runs next belongs in ordinary code, not in a prompt. State machines are deterministic by definition. LLMs are not. Putting safety-critical control flow inside a model's instruction-following behavior is putting load-bearing logic on a non-load-bearing surface. The fix is mundane. Keep the machine in code and let the model work inside the steps.

I want to be clear up front that none of the pieces here are mine to claim. State machines are decades old. The failure modes are the ordinary properties of stochastic systems. What I am offering is the case for a boundary, drawn before the bill comes due, not an invention of mine.

The pattern is everywhere, and it is seductive

Open a handful of orchestration repos and you find the same prompt fragment within minutes. It tells the model it is an agent operating as a state machine, injects the current state as a template variable, lists the legal transitions in plain English, and asks the model to emit the next state. Idle to processing on a new task, processing to complete or error, error back to idle once acknowledged. The shape is identical every time, and I understand why, because the first time I reached for a quick agent loop, a version of it is what my hand wanted to write.

The appeal is real, so let me be honest about it before taking it apart. It is compact: a working state machine in one prompt block, no imports, no transition table, no boilerplate. A junior engineer reads it and immediately understands what the agent is doing, and adding a state is a one-line edit. There is also a real intelligence argument: a model can judge ambiguous situations a hand-branched conditional would get wrong, and a code guard would need its own classifier for that judgment. And it composes naturally, since the window holding the task instructions also holds the machine's rules. One prompt, one call, and it feels elegant.

These are not imaginary benefits. The pattern persists because it works, for a while. The trouble is what that while turns into.

The first crack: the machine drifts when the model does

A state machine has a contract: given a state and an input, the next state is fixed, and that determinism is the entire value. Without it you have a stochastic function that occasionally returns a state name.

Models above zero temperature are openly non-deterministic: same state, same input, different transitions across runs. But even pinned at zero, the deeper problem should scare you: determinism there is an artifact of one model checkpoint, and checkpoints change, sometimes silently. A provider fine-tunes, ships an alignment update, or reformats the system prompt, and your machine shifts with it.

That is the whole mechanism behind the opening scene. A transition that landed on complete for a year starts occasionally landing on error, not because anything failed but because a new checkpoint weights the error-handling prose differently. The job processes. The agent believes it did not. No diff, no commit, no config change to inspect, because the thing that moved moved upstream and invisibly. You cannot pin a model version forever on most hosted APIs, and no test catches the model changing its mind. The machine you shipped is not the machine you are running six months later.

The second crack: instructions collide in one context

The machine's rules and the task's content share a single context window, and the model is asked to honor both at once with no enforced wall between them. That produces a failure no prompt engineering fully clears.

First, the attention problem. In long conversations or deep agent loops, the machine's rules drift toward the edge of the model's effective attention while fresh task content sits front and center. Constraints near the top of a sixteen-thousand-token context are weaker than the content filling the bottom, and the rules quietly demote themselves to suggestions.

Worse is the semantic collision. What happens when the task content itself contains your state names. A document that says move this ticket to the error state: is that content to process, or an instruction to execute. The model has to guess. Careful naming shrinks the collision surface but cannot remove it, because any vocabulary shared by your machine and your task domain is a path for an unintended transition. The model has no privileged parser for instructions versus content. One context window, read all at once.

The third crack: nothing leaves a trace

A code-based machine has a callstack. You can break on every transition, log the state, the trigger, the inputs, and the result with one decorator, and replay a whole sequence offline with no network call. When something breaks, you have a trace.

A prompt-based machine makes its decision inside a forward pass. You see the output token, next state error, but you do not see why. The attention weights that produced it are not inspectable, and the model's state tracking is not a data structure you can query but an emergent property of the activations that leaves nothing behind.

This is precisely where the opening scene becomes unfixable. When an agent reaches an unexpected state at step fourteen, you need to know one thing: did it receive the wrong input, or make the wrong transition on correct input. Those are different bugs. In code you read the log and answer in seconds. In a prompt-based system you re-run the workflow, vary the temperature, and try to reproduce. Sometimes you cannot, because the model update that caused it already rolled back. The bug was real in production and gone from your debugger, and you cannot fix a failure you cannot observe.

The turn: make the machine data, and the model a worker inside it

The fix is not clever, and that is the point. Instead of describing the machine in prose and letting the model emit the next state, you represent it as data and let plain code decide transitions. It reduces to three pieces.

First, an explicit, closed set of states. A small enumeration, not a free-text vocabulary. The legal states are fixed and listable, and nothing outside the set can ever appear, because no token-generation step can invent a fourth one.

Second, the transition table, held as data: a mapping from each state to the states allowed to follow it, the single source of truth for the machine's shape. Idle may only advance to processing, processing may resolve to complete or error, complete is terminal, error returns to idle. Because it is a data structure and not a paragraph of instructions, every legal move is explicit, the whole machine is readable in one glance, and adding or removing an edge is a change you can diff, review, and test, not a reweighting of prose the model reads differently after the next update.

Third, a single transition function that consults the table. Given the current state and a desired target, it refuses the move if the target is not allowed or a guard fails, and otherwise returns the new state. It never touches a model, so you can test it exhaustively, every legal edge, every illegal one, every guard outcome, with no network.

The model still has a contained job. Inside a given state the workflow calls it to do content work, extract the entities or generate the output, and it returns a structured result. Plain code decides which transition to request, and the function rules on whether it is legal. The model is a reasoning oracle invoked at a node, never the dispatcher that decides which node runs next. And because every transition flows through that one function, each is logged with the state, the target, the guard result, and a timestamp, so when something fails at step fourteen you have the complete, replayable trace the prompt-based version can never give you.

The rule, and the line it draws

The whole decision collapses to one rule. If the behavior must be deterministic, it belongs in code. If it needs language understanding, in a prompt.

In practice, code owns the skeleton: the states, the transition table, the guards, the error and recovery paths, the loop-termination conditions, the timeouts and retry counts, the hard ceiling on steps. The prompt owns the language work: generating output for a human, pulling structured data out of unstructured text, classifying content into the categories your guards consume. The model decides what to say at a step, never which step to go to.

One test makes the line concrete. Could a determined adversary steer the agent by injecting text into the task content. In a prompt-based machine, often yes, because content and transition instructions share a context. In a code-based machine the model's output is a classification your code consumes, so injecting move to complete changes the text, not the transition logic. The moment you ask a model to emit a state name your system treats as a routing instruction, you have handed control flow to a stochastic process.

Write the code version first

The prompt-as-state-machine pattern will keep showing up because it is trivially easy on day one. Twenty lines, working, demos beautifully. The failure modes only surface under operating conditions: a model update, a long context, an adversarial input, an edge case never in the demo. And the debt compounds. Every state you add makes the transition prose harder to follow, and every feature crossing a state boundary adds another collision surface. A few months in, the prompt is a few hundred tokens of machine logic nobody understands and everybody is afraid to touch.

Rewriting a prompt-based machine into a code-based one is always possible and always painful. The tests you should have written do not exist, the transitions that seemed obvious are underspecified, and you trace edge cases back through model outputs just to understand them. The closed set of states takes five minutes, the table ten, the tests twenty, and you spend those minutes either way. The only choice is whether you spend them now, calmly, or later, after the bug has been live long enough to hurt.

So stop putting state machines in prompts. Define the states explicitly, make the transition table the single source of truth, and test it with no model in the loop. The model belongs inside the nodes, doing the content work it is good at, not governing which node runs next. The agent in the opening scene, calmly reporting failure on a job that processed perfectly, is not a model problem. It is what happens when you ask the model to be the one thing it can never be, which is sure.