Your agent didn't "hallucinate a wrong action." It called a tool that timed out, retried without an idempotency key, charged the customer twice, lost its scratchpad on the third hop, and then produced a confident summary of a state that no longer existed. None of that is an intelligence problem. It's the same class of problem a payments team or an orchestration team has been fighting for decades.
The moment you give a model tools, memory, and a loop, you have stopped building a prompt and started building a distributed system — one where every step is a network call to a flaky, slow, non-deterministic service. The good news: that's a solved-ish discipline. The bad news: most agent code ignores all of it.
TL;DR — An agent is a control loop over unreliable remote calls. Its real failure modes are distributed-systems failures: non-idempotent retries, timeouts and partial failure, lost or corrupted state, and zero observability. Fix those with the boring tools — idempotency keys, timeouts, durable state, tracing, trajectory tests — and the "AI reliability" problem mostly dissolves.
An agent is a control loop over calls that fail
Strip the branding and an agent is a while loop: ask the model what to do, execute a tool, feed the result back, repeat until done. Every arrow in that loop crosses a network boundary to something that can be slow, down, or lying — the LLM API, a search endpoint, your own microservices, a database.
That is the exact shape of a distributed system: independent components, unreliable links, no shared clock, partial failures. The literature is unambiguous about what happens if you pretend the network is reliable. An agent that assumes every tool call succeeds instantly and exactly once is a demo, not a system.
Retries need idempotency, or you double-charge the customer
The first thing everyone adds when agents flake is a retry. The second thing everyone learns — usually in production — is that retrying a non-idempotent action is how you send two emails, create two tickets, or move money twice.
The model can't see this. It calls charge_card, the call times out on the response (but succeeded on the server), the loop retries, and now there are two charges. The fix is not a smarter prompt — it's the same one Stripe hands every integrator: an idempotency key. Every side-effecting tool takes a caller-supplied key; the downstream service dedupes on it. Reads can retry freely; writes retry safely. This is a plumbing decision, and it belongs in your tool layer, never in the model's judgment.
Timeouts and partial failure are the default, not the edge case
Ask engineers what happens when a tool call hangs and most agent frameworks answer "it hangs." No timeout, no deadline budget, no cancellation — one slow dependency and the whole agent sits there burning tokens and wall-clock.
Borrow the service playbook: put a timeout on every call, propagate a deadline across the whole run (this task has 30 seconds total, not 30 seconds per hop), and decide explicitly what a timed-out step means. Did it fail, or did it succeed and you just didn't hear back? That ambiguity — success-but-no-ack — is the hardest problem in distributed systems, and your agent hits it constantly. Handle it on purpose, or the model will improvise on top of a state it can't observe.
State is the hard part — where does the agent's memory actually live?
The conversation buffer is not your state. The moment an agent does anything durable — files a refund, updates a record, kicks off a job — the truth lives out in the world, and your in-context "memory" is a cache of it that goes stale the instant the process restarts.
Treat agent state like you'd treat any workflow's state: make it durable and resumable. Persist each committed step so a crashed run can resume instead of redoing side effects. Separate what's been decided from what's been executed. This is why durable-execution engines are quietly eating the agent-infra space — they bring exactly-once-ish workflow semantics to a loop that otherwise loses its mind on the first restart. If your agent can't survive a process kill mid-run without repeating actions, you have a state bug, not a model bug.
You can't debug what you don't trace
When a service misbehaves, you open a distributed trace and follow the request. When an agent misbehaves, most teams open a wall of print statements and squint. Same problem, worse tooling.
Log the trajectory: every model decision, the exact tool inputs and outputs, latencies, token counts, retries, and the state at each step — correlated by a run ID, ideally as real spans. Then a wrong outcome becomes a traceable incident ("it retried search four times, each timed out at 5s, then answered from stale context") instead of a vibe. You cannot improve what you cannot see, and "the AI is unreliable" is what unreliability looks like when you have no traces.
Test the trajectory, not just the output
A unit test asserts an output. An agent needs more, because the same final answer can come from a clean two-step path or a flailing nine-step one that happened to land. Build a golden set of tasks and assert on the path: did it call the right tools, in a sane order, without redundant or destructive steps, inside the step and cost budget?
Track those metrics over time so a bad model swap or a broken tool shows up as a regression on a dashboard, not as a slow bleed of angry users. This is the unglamorous part everyone skips, and it's the line between an agent you can ship and a party trick you can demo.
The reframe that fixes most of it
Next time your agent "goes rogue," don't reach for a bigger model or a cleverer prompt first. Ask the distributed-systems questions:
- Are my side-effecting tools idempotent, with keys, so retries are safe?
- Does every call have a timeout and the run a deadline?
- Is my state durable and resumable across a restart?
- Can I pull the full trace of any run?
- Do I have trajectory evals that catch a bad path before users do?
The teams shipping reliable agents aren't the ones with the smartest model. They're the ones who noticed they'd built a distributed system — and engineered it like one.
Your model is probably fine. Go engineer the loop around it.
Top comments (0)