Does Your Agent Suck, or Is Your State Layer Just Missing?

Everyone's building agents. Almost nobody's building the thing that makes agents work. It's not the model. It was never the model.

I spent eleven months at a company that shall remain unnamed building retrieval pipelines for a product that never shipped, and the single most useful thing I learned there is that ninety percent of "the AI is broken" tickets are actually "we never wrote down what the system knows." Same bug, every time, across four teams, in three different companies.

So. If you're running an agent in production and it keeps forgetting things, contradicting itself, or confidently doing the wrong task twice in a row, read this before you go shopping for a bigger model. Bigger models fix reasoning. They don't fix memory. Those are different problems and only one of them costs you money per token.

The thing agents actually need

An agent is a loop. Model plus tools plus state. That's it, that's the whole architecture, everything else is dressing.

Most people build the loop and the tools and then let the state be "whatever's in the context window." Which works until turn nine, at which point the context is a landfill: tool outputs, half-finished reasoning, an error message from four minutes ago, the user's original request buried somewhere in paragraph two.

It's just a data structure problem. That's the part that annoys me. We talk about agents like they're magic, and then the actual failure is that nobody defined a schema.

Here's the shape I use now, and I've used it on three systems that survived contact with real users.

Layer 1: Facts (mutable, authoritative)

A key-value store for the things that are true right now. Current user, current task, current ticket ID, whatever. Small. Written by tools, never by the model. If the model wants to change a fact it calls a tool and the tool writes it.

This is the layer that fixes contradiction errors. When the model "remembers" the wrong order ID, it's not remembering, it's reading stale text out of the transcript. Give it a fact store and it stops guessing.

Layer 2: Events (append-only, immutable)

Log every tool call, every result, every decision, in order. Append only. Never edit.

You need this for three reasons and all of them are boring: debugging, replay, and audit. When an agent does something insane on Thursday, the event log is the only reason you'll know why. I've watched teams spend two days reproducing a bug that a 200-line event log would've explained in ninety seconds.

Layer 3: Scratch (summarizable, disposable)

The agent's working notes for the current task. Plan, sub-goals, what it's tried. This layer is allowed to be lossy. Summarize it aggressively. Throw it away at task boundaries.

This is what you feed back into context instead of the raw transcript, and it's the single highest-leverage change you can make. Instead of 40,000 tokens of history, you pass 800 tokens of "here's where we are and what we've ruled out."

The context assembly step nobody writes

Between the state and the model there's a function. It takes your state layers and produces a prompt. Half the agent codebases I've looked at don't have this as a named thing. It's just string concatenation scattered across four files.

Name it. Make it one function. Have it emit a deterministic prompt given the same state. Now you can test it.

Then do the obvious thing: test it. Given this state and this user message, does the assembly produce something a human would consider sufficient? That's a unit test. It's not an eval, it doesn't need a model, it just checks that the right facts made it in and the stale ones didn't.

I've caught more real bugs with that one test than with any eval harness I've ever built.

On retrieval, briefly, because everyone asks

Retrieval isn't memory. Retrieval is search. If your agent has to search for something it should already know, you've got a state design problem, not a retrieval problem.

The pattern that works: retrieve once, at task start, into the fact layer. Then the agent works from facts, not from a search index it queries on every turn. If it needs to search mid-task, that's a tool call, and the result goes into events and possibly facts.

What doesn't work: dumping the top twenty chunks into context every turn. Feels thorough. Produces the exact landfill you were trying to escape.

Cost, since we're being practical

Token spend on an agent loop scales with turns times context size, and context size is the variable you control. Compressing scratch from 30k to 1k tokens doesn't save ten percent. It saves an order of magnitude on long tasks, because you're not re-sending the same growing pile every turn.

I rewrote one internal agent's assembly step and its per-task cost dropped from about $0.71 to about $0.06. Same model. Same tools. Same tasks. The only change was that we stopped pasting the transcript into the prompt.

Nobody talks about this because it's not a model announcement. It's a database schema. But it's the difference between a demo and a product.

The uncomfortable part

Agent quality is mostly engineering, and engineering is not exciting, and that's why the field keeps skipping it.

We keep waiting for a model that remembers. A model that plans. A model that doesn't hallucinate the order ID. And each release gets closer, and people build the same broken loop on top of it and blame the model again.

The model will get better. Your state layer will still be missing.

Write the schema. Name the assembly function. Log the events. Compress the scratch. Four things, maybe a week of work, and your agent stops being embarrassing.

Everything is just layers.