Mayur Agnani

Posted on Nov 4 Edited on Nov 4 Originally published at theaicommit.com

How to Send Claude Only the New Messages in a Long Agent Loop

#ai #agents #performance

Engineer's view. This is a sticky session. A load balancer pins a user to a server that keeps their session in memory, and the app breaks when that server restarts. A message thread is the API keeping your agent's conversation in memory. Treat it as a cache, or one restart loses the agent's work.

TL;DR. Claude can now hold an agent's conversation on its side, so each turn sends only what is new. The server can forget it at any time, so you keep the full copy and send it again when asked.

Explain Like I'm 5

You are telling a friend a long story over the phone, one piece a day. Every day you start again
from the very beginning, because you are not sure they remember. The story gets longer, so the
calls get longer.

Then your friend says: "I will remember. Just tell me the new part." That is much faster.

But your friend forgets if you wait too long between calls. So you keep the whole story written
down. When your friend says "sorry, I forgot," you read it all out again, and carry on.

The Problem

You have shipped this bug before, in a system with no model in it.

A web app kept each user's cart in the memory of one server. A load balancer sent the user back to
that server every time. It was fast, because nothing had to be loaded. Then a deploy restarted the
server, and every cart on it vanished. The fix was not a longer-lived server. It was writing the
cart to the database and treating memory as a cache.

An agent loop has the opposite habit today. The Messages API is stateless, so every turn sends the
whole conversation: the system prompt, every tool definition, and every file the agent has read so
far. Turn 40 of a coding agent uploads turn 1 to 39 again. The API then validates and tokenizes
all of it again before the model starts. In my simulated 40-turn loop, with a 6 KB file read per
turn, that came to 5.6 MB of uploads for one task.

Claude's API now has a beta called message threads. The server keeps the conversation, and later
turns send only what is new. It is the cart in memory again, with a five-minute timer on it.

The fix: send only the new messages on a thread, keep your own full copy of the conversation,
and write the replay path before the happy path.

The Fix: Keep the Conversation on a Thread, and Keep Your Own Copy

What does the server actually keep?

The first request is the one you send today, plus "thread": {"type": "create"}. The API stores
the conversation, your system prompt, your tools and Claude's reply under the response id.

Each later request sets the thread type to continue, with previous_message_id set to that id.
It carries only the messages since that response, usually the tool_result blocks the agent just
produced. It must leave system and tools out. Sending either one, even as null, is a 400.

In my simulated loop, the uploads fell from 5.6 MB to 256 KB. That is 4% of the bytes.

Can I change the model or the thinking setting mid-thread?

No. The thread stores a fingerprint of its fixed fields: model, thinking, tool_choice,
output_config, the top-level cache_control and the set of beta headers, among others. A
continue request with a different value gets a 400 with thread_fingerprint_mismatch. To change
one, start a new thread with the full conversation.

When does it forget?

Five minutes after its last request, and it is deleted within an hour. It also forgets for reasons
you are not told. A turn written by a fallback model is never stored. Nor is a reply whose tool
call was cut off by max_tokens. Every one of these comes back as the same 404.

Does a thread make the tokens cheaper?

No. Pricing does not change, and a thread does not cache your prompt. Prompt caching still needs
cache_control, exactly as without a thread. The thread saves upload and processing time, not
input tokens.

What This Means for You

When this matters. You run an agent loop on Claude: a coding agent, a support bot that calls
tools, a research agent. It matters most when the loop runs many turns quickly and the tool
results are large, because each turn then repeats the most. It matters little for a three-turn
chat.

How it affects you. Without a thread, each turn re-uploads everything, and the time to the
first token creeps up as the conversation grows. With a thread, a careless client gets a new
failure instead. It throws away its message list because "the server has it." Then the first 404
loses the agent's whole task. Gaps decide how often that happens. A loop whose tools return in
seconds almost never expires. One where a person approves each tool call is different. If they
take about four minutes on average, my simulation replayed 29% of turns.

What to do about it. First, with no setup: log the size of each request body your agent loop
sends, and the gap between turns. Plot both over one long task. That tells you how much a thread
would save and how often it would expire.

Next, switch one loop to the SDK's thread helper, which handles the replay for you. Keep writing
the conversation to your own store as well.

Then, if you build requests yourself, write the client below.

Implementing It

The change.

Three parts of the agent change. Your team may own all of them.

Part 1, the loop that calls the model. With the Python SDK's helper, this is the whole change.
The helper keeps the conversation in memory, adds the beta header, and replays once on a 404:

thread = client.beta.messages.thread(
    model="claude-opus-5-5", max_tokens=4096,
    cache_control={"type": "ephemeral"},      # prompt caching still has to be asked for
    system=SYSTEM, tools=TOOLS,
)
reply = thread.send({"role": "user", "content": task})
while reply.stop_reason == "tool_use":
    reply = thread.send({"role": "user", "content": run_tools(reply)})

The helper's copy dies with the process. If the task must survive a restart, save
thread.messages after each turn.

Part 2, the client, if you build requests yourself. Append to your copy first, send only the new
messages, and replay on the three codes that mean "rebuild it":

REPLAY = {"thread_not_found", "thread_already_continued", "thread_continue_in_flight"}

def send(self, *new):
    self.messages.extend(new)                 # our copy is the record
    full = lambda: self.create(self.messages, thread={"type": "create"},
                               system=SYSTEM, tools=TOOLS)
    if self.prev_id is None:
        r = full()
    else:
        try:                                  # no system, no tools on a continue
            r = self.create(list(new), thread={"type": "continue",
                                               "previous_message_id": self.prev_id})
        except anthropic.APIStatusError as e:
            details = (e.body.get("error") or {}).get("details") or {}
            if details.get("error_code") not in REPLAY:
                raise                         # a fingerprint mismatch is our bug
            r = full()
    self.messages.append({"role": "assistant", "content": r.content})
    self.prev_id = r.id

Two more rules belong in that function. After stop_reason: "refusal", never continue. Drop the
refused turn and start a new thread. After a response with a fallback_message entry in
usage.iterations, the next turn must also be a create. code_example.py has both, plus a fake
server that runs the documented rules.

Part 3, the streaming handler. The id arrives in message_start. Continue from it only after
message_stop, or the next request can get a 404. If the stream drops before message_stop, send
the same continue request once more. If that returns thread_already_continued or
thread_continue_in_flight, replay. The server may have saved the turn you never received.

Getting a field wrong fails loudly, which is the good news. A continue request that still carries
system is a 400. A changed thinking setting returns thread_fingerprint_mismatch, and the
message names the field. A fifth cache_control breakpoint returns thread_unsupported_request.
The quiet failure is the client that never replays. It works in every short test, then loses the
task the first time someone takes a coffee break.

How you know it worked.

Log three numbers per task. The bytes per request should stay flat after turn 1 instead of
growing. The share of turns that replayed should match your gaps. My simulation predicts close to
0% for a fast loop and about 30% at four-minute gaps. Finally, cache_read_input_tokens should
still be large on every turn. If it drops to zero, the top-level cache_control is missing.

Then kill a thread on purpose. Wait six minutes before the next turn, and check that the task
finishes with the same answer.

When a Message Thread Is the Wrong Tool

When turns are minutes apart. A person who approves each step, or a job that waits on a slow
build, will expire the thread often. Each replay costs a failed request first, so at eight-minute
gaps about half of all turns pay an extra round trip.

When the conversation is short. A three-turn chat re-sends little, so a thread adds code and saves
nothing.

When you need a feature threads do not support. That includes Message Batches, token counting,
compaction on demand, and most context editing. Organizations with zero data retention, HIPAA
readiness or inference hooks cannot use threads at all.

When you want the server to be your database. A thread lasts minutes, and there is no API to list,
read or delete one. If you want stored history, keep it yourself.

My numbers come from a simulation of the documented rules, not from timing the live API. Measure
your own loop before you trust them.

Three questions before you switch:

How many turns does a typical task take, and how large is each tool result?
What is the longest gap between two turns, including people and slow tools?
Where does your own copy of the conversation live if the process dies?

Glossary

message threads — a beta where Claude's API keeps a conversation in memory, so later turns send only new messages.
fingerprint — the stored values of a thread's fixed fields; a continue request must match them.
replay — sending the full conversation again in a new create request after the server forgot it.
fallback model — a different model the API answers with when the requested one cannot; its turns are never stored.
prompt caching — reusing already-processed prompt text at a lower input price, requested with cache_control.

DEV Community