Your LLM has no memory. Your application had better have one.

Intro

Every LLM tutorial has the same shape. You send a list of messages, you get a reply, you append the reply, you send the list again. It works, it demos well, and it quietly teaches you the wrong mental model.

That loop is not state management. It is the absence of it, dressed up as a feature.

The APIs behind LLM platforms are stateless. Each request is independent of the last one. The model does not "remember" your conversation, it re-reads whatever you hand it, every single time. Meanwhile the thing you are building, a support flow, a code migration, an approval process, a multi-step agent, is stateful by nature. Somebody started it, it is halfway done, and it has to survive interruptions, retries and crashes.

That gap between a stateless processor and a stateful process is where production systems break. Here are the three places I see it break first.

1. "Just send the history" stops scaling

The tutorial answer to "how does the model know what happened earlier?" is to resend the whole transcript. It is fine for ten turns. At a hundred turns you are paying for the same tokens over and over, latency creeps up, and you start hitting context limits at the worst possible moment.

The usual patch is summarization: compress old turns into a paragraph and carry on. That helps with size but introduces a subtler problem. The summary is now the model's opinion of what mattered, and nothing in your system can verify it.

Imagine a multi-step refund workflow (illustrative scenario, not a real case). At turn 4 the user confirms the order ID. At turn 30 the summary says "user wants a refund for a recent order." The ID got compressed away, and the model now confidently picks the wrong one.

The fix is to stop treating the transcript as the state. Facts that the process depends on belong in a structured object your code owns:

@dataclass
class RefundState:
    order_id: str | None = None
    reason: str | None = None
    amount_confirmed: bool = False
    step: str = "collect_order"   # explicit position in the workflow

def build_prompt(state: RefundState, recent_turns: list[Message]) -> list[Message]:
    # The model gets the current state as input, plus only the turns it needs.
    return [system_prompt(), state_message(state), *recent_turns[-6:]]

Now the context you send is small, deterministic and reviewable. The model reads state, it does not store it.

2. The user interrupts halfway through

Real users do not wait politely for a long-running process to finish. They close the tab, change their mind, or send "actually, cancel that" while three tool calls are still in flight.

If the only record of progress is "whatever the model said last," you cannot answer basic questions. Which steps already ran? Which are safe to abandon? Does the half-finished action need to be rolled back?

A workflow that can be interrupted needs explicit steps with explicit statuses, persisted outside the model:

class Step(Enum):
    PENDING = "pending"
    RUNNING = "running"
    DONE = "done"
    CANCELLED = "cancelled"

# Persisted per run, updated by your code, never inferred from model output.
run.steps["reserve_inventory"] = Step.DONE
run.steps["charge_card"] = Step.RUNNING

When the interrupt arrives, your code reads the run record, decides what a cancellation means at this point, and tells the model the outcome. The model is not asked to work out where it was.

3. The API call fails in the middle of a transaction

Retries are where stateless design bites hardest. If a request times out after your tool executed but before you saw the response, resending it can execute the tool twice. Charge the card twice. Send the email twice. Open two tickets.

This is an old distributed systems problem, and the old answers apply: idempotency keys, an append-only event log, and recovery by replay.

def execute_tool(run_id: str, step_id: str, call: ToolCall):
    key = f"{run_id}:{step_id}"
    if (result := store.get_result(key)) is not None:
        return result                      # already ran, do not run again
    result = tools[call.name](**call.args, idempotency_key=key)
    store.save_result(key, result)
    return result

Notice that none of this depends on the model behaving well. That is the point. A model can be asked to "remember not to repeat itself," and it will still eventually repeat itself.

The pattern underneath

All three failures come from the same mistake: letting the conversation double as the system of record. Once you separate the two, the design gets much less mysterious.

• State lives in your application: structured, persisted, versioned, recoverable.
• The model is a stateless processor. It receives the relevant state, produces a proposed next action, and your code validates and applies it.
• The transcript is a log for humans and for debugging, not the source of truth.

An LLM workflow is really a state machine with a language model on top of it.

Build it or lean on the platform?

Some providers now offer conversation or thread objects that manage history for you. They are convenient for prototypes and for simple chat. Before relying on them for a business process, ask a few questions:

• Can you inspect and export the state in a form your own code can reason about?
• Can you resume, replay or roll back a run after a failure?
• Can you move to another model or provider without losing the process?
• Who is accountable when the stored history and reality disagree?

If the answers are "not really," keep the state in your own store and treat the platform's memory as a cache at best. Managed history is fine as an optimization. It should not be where correctness lives.

Closing thought

If you rely on the model to remember the state of the conversation, your system will eventually break, and it will break in the least reproducible way available. Keep your application code as the source of truth and let the model do what it is good at: processing what you hand it, one request at a time.

Where does your team keep workflow state for multi-step LLM interactions: your own database, an event log, or the provider's thread objects? And what made you pick it?

Comments

[Max Quimby]

The refund example nails the subtle failure: summarization quietly turns facts into the model's opinion of what mattered, and nothing downstream can verify it. We ran into this on multi-step pipelines where an ID confirmed at turn 4 got compressed out by turn 30, and the model happily picked a plausible-but-wrong one. Your RefundState dataclass is the fix we landed on too — the process-critical facts live in a typed object the code owns, and the transcript is demoted to "recent color," not source of truth. One thing I'd add for anyone adopting this: make the state object the thing you persist and replay, not the message list. When a run crashes and resumes, rebuilding from a durable state object is deterministic; rebuilding from a re-summarized transcript is not. The mental-model line — "that loop is the absence of state management dressed up as a feature" — is exactly why so many demos survive to production and then fall over at turn 100. Curious whether you version the state schema, since workflows outlive their own field definitions.

[Edward Sinclair]

Treating the message history as the application state works for demos, but falls apart with retries, failures, branching workflows, and long-running agents. Explicit state management is what makes these systems production-ready.