Originally published on Loop & Retry — field notes on building LLM agents that survive production.
Here's the most useful thing I can tell you about agent architecture: most of the time, don't build one. The task in front of you probably has a known set of steps, and a thing with a known set of steps is a pipeline, not an agent — building it as an agent buys you nondeterminism, latency, and a token bill you didn't need, in exchange for flexibility you're not going to use.
This post is the decision I make before writing any orchestration code: does this task actually need an agent, or is it a fixed workflow wearing a costume? I'll define the line precisely, show the arithmetic on what autonomy costs, build one task both ways so the difference is concrete, and end with the checklist I actually run down.
First, a definition that does real work
The word "agent" has been stretched to mean "anything with an LLM in it," which makes the design question impossible to reason about. So here's the distinction I use, and it's the one that matters for cost and reliability:
- A workflow is an LLM (or several) orchestrated through control flow you wrote. The code decides what happens next. The model fills in the steps; the sequence is fixed.
- An agent is an LLM that decides its own control flow. It's a model in a loop with tools, and the model — not your code — chooses which tool to call next and when to stop.
That single property — who owns the control flow — is the whole decision. When your code owns it, you can read the path, test the path, and bound the cost of the path. When the model owns it, you've traded all three away for the ability to handle situations you couldn't enumerate in advance. Sometimes that trade is exactly right. Usually the situations were enumerable and you just hadn't written them down yet.
The agent tax
Autonomy isn't free, and the cost isn't abstract. Four things get worse the moment the model owns the loop.
Token cost goes quadratic. This is the one people underestimate. In an agent loop, each step re-sends the entire conversation so far — system prompt, tool schemas, and every prior turn and tool result. If each step adds roughly a constant amount of context, then step k sends about k units, and N steps send 1 + 2 + … + N ≈ N²/2 units total. A 10-step agent doesn't cost 10× a single call; the input side costs closer to 50×. A single structured call sends the context once. This is exactly the cost model I explored in how agents with long trajectories compound their cost.
Put numbers on it. Say your base context (system prompt + tool schemas + input) is 4,000 tokens, and each step appends ~800 tokens of assistant reasoning and tool output. A one-shot call reads 4,000 input tokens. A 10-step agent reads 10×4000 + 800×(0+1+…+9) = 40,000 + 36,000 = 76,000 input tokens for the same job — 19× the reads, before you count a single output token. Prompt caching claws some of this back for the stable prefix, but the part that grows every step — the transcript — is exactly the part caching helps least.
Latency is serial. Every tool call in an agent loop is a round trip: model → tool → model → tool. Ten steps is ten sequential model calls plus ten tool executions, and you can't parallelize a sequence where step k depends on the result of step k−1. A workflow with known structure can fan out independent calls concurrently; an agent that discovers its plan one step at a time cannot.
The failure surface is the whole trajectory. A single call fails in one place. A ten-step agent can go wrong at any step, and — worse — a wrong-but-plausible intermediate result poisons every step after it. When it fails you're not debugging a function, you're debugging a path that was different last time. These failures are often silent — the agent completes, returns exit zero, and the result is wrong. (This is exactly why grading an agent means grading a trajectory, not an output — I wrote a whole post on why that's hard.)
You can't unit-test control flow you don't own. assert route(ticket) == "billing" is a test. There is no clean assertion for "the agent will, across runs, choose a reasonable sequence of tool calls," because the sequence is a distribution, not a value. You can evaluate it statistically over a suite, but you've left the world of cheap deterministic tests — and you left it voluntarily.
None of this is an argument against agents. It's an argument for making sure you're buying something with it.
Continue reading: the rest of this post — including the sections that follow — is on Loop & Retry: read the full article
Top comments (0)