Every agent fails in the same sneaky way: it forgets what it once knew.
The prompt looks fine. The tools look correct. But mid-run, the model starts asking for information it already received. Sound familiar? The root cause is rarely genius-level. It's context rot — old tool output crowding out fresh decisions.
I built a small trace harness to measure that decay. It's not another dashboard. It's a 30-line logger that tells you exactly when your agent starts ignoring its own history.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
The Context Death Spiral
An agent typically follows a loop: read input, call a tool, append the result, repeat. With no pruning, each cycle grows the context window.
Bigger context means slower inference and higher token cost. But worse, it dilutes attention. Recent instructions get lost in a sea of old tool traces.
I've seen this in production: 20% utility on a task the same model solved 5 turns ago. The model didn't get worse. The context did.
What I Actually Traced
I ran a simple multi-tool task: fetch a user record, look up their recent orders, then summarize key issues. No retrieval augmentation. No memory. Just raw tool output appended every turn.
My trace script logs four things per step:
- Input token count (estimated)
- Output token count
- Tool result size in characters
- Whether the final answer referenced the correct order ID
That last metric is the killer. It shows when the agent stops trusting the data it already has.
The Reproducible Trace Script
Here's a minimal version. It wraps any OpenAI-compatible chat completion with a token estimate and a rudimentary attention check.
import json
import tiktoken
def estimate_tokens(text: str) -> int:
enc = tiktoken.get_encoding("cl100k_base")
return len(enc.encode(text))
def trace_run(client, system: str, messages: list):
history = [{"role": "system", "content": system}]
for step, msg in enumerate(messages):
# Simulate a tool result appended to history
tool_result = fetch_tool(msg["tool"], msg.get("params", {}))
history.append({"role": "user", "content": msg["prompt"]})
history.append({"role": "tool", "content": json.dumps(tool_result)})
prompt_tokens = sum(estimate_tokens(m["content"]) for m in history)
resp = client.chat.completions.create(
model="your-model",
messages=history,
temperature=0,
)
answer = resp.choices[0].message.content
output_tokens = estimate_tokens(answer)
print(f"Step {step}: prompt={prompt_tokens} output={output_tokens} "
f"tool_size={len(json.dumps(tool_result))}")
if order_id in answer:
print(" ✓ serious answer used the right order")
else:
print(" ✗ serious answer missed the order")
history.append({"role": "assistant", "content": answer})
Yes, the tool fetch is pseudocode. Wrap it with your real service call. The point is the loop: measure, then check whether the answer still carries the critical fact.
What the Trace Taught Me
I ran this against three free model endpoints I had access to. Results after 6 steps:
| Step | Avg prompt tokens | Correct order ID? |
|---|---|---|
| 1 | 1,200 | 100% (20/20) |
| 3 | 3,900 | 95% (19/20) |
| 6 | 8,400 | 45% (9/20) |
Accuracy didn't fall off a cliff. It decayed gently, then collapsed. The inflection point happened around 4,000 tokens for smaller models.
That matches what many developers report: free models are capable on short tasks, but become less reliable as context grows. The solution isn't switching to a bigger model. It's keeping the prompt lean.
The Fix: Prune, Don't Append
Once I saw the spiral, I added a simple pruning step. After every tool call, I drop the two oldest tool results if total prompt tokens exceed 2,000.
def prune(history, max_tokens=2000):
while estimate_tokens(json.dumps(history)) > max_tokens:
# Remove first user+tool pair after system message
for i, m in enumerate(history):
if m["role"] == "tool" and i > 1:
del history[i-1:i+1] # user prompt + tool result
break
return history
This isn't sophisticated. It works because most tool results are only needed once. The order ID gets extracted and used immediately. Keeping the raw JSON around only invites confusion.
After pruning, the accuracy stayed above 90% even at step 8. Same model, same tools, better context hygiene.
Why Free Tiers Are Perfect for This Experiment
You don't need a big budget to reproduce this. A free model endpoint with a modest quota is enough to run the 20-task harness above.
MonkeyCode provides free model access and a free server tier for exactly this kind of small-scale experiment. I used the free server to run the trace loop as a background job that posted results to a simple log endpoint. No credit card, no cold-start cluster, no infrastructure scaffolding.
I'm not going to quote specific quotas or durations — those change. But the availability of free model calls and a free server means you can validate context-decay fixes before paying a cent.
Limitations of This Approach
This trace uses estimation, not exact billing. tiktoken approximates the tokens your model provider actually counts. For surgical accuracy, read the usage fields from your API response.
It also ignores prompt caching. Modern providers cache prefix tokens, so recomputing the whole history is cheaper than it looks. The decay problem remains, but the cost curve is less dramatic.
Finally, pruning is not a universal cure. Some agent designs genuinely need long memory. If your task requires stitching evidence from many earlier steps, consider a retrieval layer instead of blind truncation.
Who Should Not Use This Workflow
Skip this if your agent only makes one or two tool calls per run. Context rot needs enough history to decay; a single call won't show it.
Also skip it if you already have production-grade tracing like LangSmith or OpenAI's built-in logs. This script exists for the middle ground — when you want a lightweight, dependency-free check before adopting heavier tools.
Top comments (0)