I finally cut AI context bloat by 80% and the fix was embarrassingly simple

A few months ago I watched an agent do something deeply cursed.

It had already fetched the data.
It had already parsed the data.
It had already stored the useful fields.

And then it pasted the entire raw tool response into the next model call anyway.

Full JSON blobs. Logs. Metadata. Useless fields. Everything.

The next step got slower.
The step after that got more expensive.
Then the model started doing that annoying thing where it answers the easy part of the task and quietly ignores the important part.

That was the moment the bug became obvious:

Most “context window problems” are not model problems.
They are state management problems.

Once you see that, you start spotting it everywhere in n8n flows, LangGraph agents, MCP tool loops, and custom OpenAI-compatible orchestration code.

The actual fix

Stop replaying raw tool output into every prompt.

Keep full results in structured state outside the model.
Send only:

• a short summary
• the exact fields needed for the next step
• or the exact evidence slice if verification matters

That one change usually does more for agent quality than switching from one frontier model to another.

The anti-pattern

Here’s the bad version:

result = weather_api.get_forecast("Austin")

messages.append({
    "role": "user",
    "content": f"Use this API response to decide whether to send an alert:\n{result}"
})

If result contains 200 lines of hourly forecast data, provider metadata, timestamps, debug fields, and trace IDs, you just turned your prompt into a junk drawer.

Now the model has to:

• find the relevant fields
• ignore the irrelevant ones
• remember prior steps
• plan the next action
• not get distracted by noise

Models can do this.
They are just not especially efficient at it.

The better pattern

Store the full result in app state.
Pass only the fields the next step needs.

result = weather_api.get_forecast("Austin")
state["weather_raw"] = result

weather_summary = {
    "temperature_f": result["current"]["temp_f"],
    "conditions": result["current"]["condition"],
    "wind_alert": result["alerts"].get("wind", False)
}

messages.append({
    "role": "user",
    "content": f"Decide whether to send an alert using this data: {weather_summary}"
})

Same workflow.
Smaller prompt.
Less distraction.
Better odds the model focuses on the decision instead of re-parsing junk.

Prompt size is not the same as prompt quality

A lot of builders confuse these two ideas:

• the model can fit a long context
• the model should receive a long context

Those are not the same.

A giant prompt is often just a badly managed state store wearing a trench coat.

Even when GPT-5, Claude Opus, Grok, Qwen, or Llama can technically accept the full history, performance still gets mushier as stale content piles up. Latency rises. Cost rises. Attention gets diffused.

This is why long-running agents often feel smart for 5 steps and weirdly flaky by step 20.

The 3 patterns, ranked

Approach	What happens in practice
Raw tool output in prompt	Biggest prompt growth. Highest distraction. Latency and cost climb every step.
Summarized tool result in prompt	Usually the best balance. Smaller prompt footprint, but bad summaries can hide edge cases.
Structured state outside prompt	Best default for multi-step agents. Small recurring prompts, cleaner reasoning, better orchestration.

My opinionated take: structured state outside the prompt should be the default.

If you are building anything more serious than a toy bot, replaying raw tool output into every step is lazy architecture.

What should live where?

This is the mental model I wish more agent builders started with:

1. Short-term conversational state stays attached to the thread.
2. Long-term facts and tool results live outside the prompt in a database or memory store.
3. The next model call gets only the smallest useful slice of that state.

That’s it.

Not complicated. Just disciplined.

Real examples

LangGraph: checkpoint state, don’t replay transcripts

LangGraph gets this right.

Instead of stuffing previous tool output back into every prompt, persist graph state and resume by thread.

from langgraph.checkpoint.postgres import PostgresSaver

DB_URI = "postgresql://postgres:postgres@localhost:5432/postgres?sslmode=disable"

with PostgresSaver.from_conn_string(DB_URI) as checkpointer:
    graph = builder.compile(checkpointer=checkpointer)

That is the grown-up version of memory.

The model gets what the next node needs.
Postgres keeps the rest.

n8n: trim memory on purpose

n8n is refreshingly practical here.

If you are building AI workflows in n8n, you already have the tools to avoid context bloat:

• Simple Memory
• Redis
• Postgres
• Xata
• Zep
• Motorhead
• Chat Memory Manager

The existence of Chat Memory Manager should tell you everything.

If a workflow platform ships a node specifically to inspect and reduce memory, then “just append to chat history forever” is obviously not the best pattern.

MCP: tool output is app state first, prompt material second

MCP makes this cleaner because tools already have structure.

Example tool definition:

{
  "name": "get_weather",
  "description": "Get current weather information for a location",
  "inputSchema": {
    "type": "object",
    "properties": {
      "location": { "type": "string" }
    },
    "required": ["location"]
  },
  "outputSchema": {
    "type": "object"
  }
}

That should push you toward a simple pattern:

• call the tool
• store the full structured result
• pass only the needed fields into the next prompt

For example:

tool_result = {
    "temperature": "72F",
    "conditions": "cloudy",
    "wind_alert": False,
    "hourly": [...],
    "provider_metadata": {...},
    "trace_id": "abc123"
}

state["weather_result"] = tool_result

prompt_payload = {
    "temperature": tool_result["temperature"],
    "conditions": tool_result["conditions"],
    "wind_alert": tool_result["wind_alert"]
}

The prompt becomes a working note, not a warehouse.

Prompt caching helps, but it does not fix bad context hygiene

This part gets misunderstood a lot.

Anthropic prompt caching is useful when you keep reusing the same stable prefix.

That means things like:

• system instructions
• tool definitions
• fixed documents
• repeated scaffolding

It does not magically make a sloppy prompt good.

Claude caching only helps when the prefix is identical, and cache entries last 5 minutes by default. That’s great for repeated prefixes. It does nothing to make a giant raw tool dump more relevant.

If anything, caching can hide bad architecture for a while because the bill hurts less before quality problems become obvious.

Example:

import anthropic

client = anthropic.Anthropic()
response = client.messages.create(
    model="claude-opus-5-5",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an AI assistant tasked with analyzing literary works.",
    messages=[
        {
            "role": "user",
            "content": "Analyze the major themes in Pride and Prejudice."
        }
    ],
)

Use caching for stable prefixes.
Do not use it as an excuse to keep shoving junk into prompts.

A quick before/after refactor

Here’s what this looks like in a normal agent loop.

Before

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    prompt = f"""
    User request: {user_request}

    Search results:
    {search_results}

    CRM data:
    {crm_data}

    Decide what to do next.
    """

    return llm(prompt)

This works at first.
Then search_results gets huge.
Then crm_data gets huge.
Then every step drags around more baggage.

After

def run_step(state, user_request):
    search_results = search_docs(user_request)
    crm_data = fetch_customer_record(state["customer_id"])

    state["search_results_raw"] = search_results
    state["crm_data_raw"] = crm_data

    prompt_payload = {
        "user_request": user_request,
        "top_matches": [doc["title"] for doc in search_results[:3]],
        "customer_tier": crm_data["tier"],
        "open_tickets": crm_data["open_tickets_count"],
        "billing_status": crm_data["billing_status"]
    }

    prompt = f"Decide what to do next using this state: {prompt_payload}"
    return llm(prompt)

This version is less fragile.

And when the model needs exact evidence later, you can pull the raw state back in selectively.

When should you keep raw output in the prompt?

There are real exceptions.

Sometimes the next step genuinely needs exact source material.

Keep raw output in-context when you need:

• legal or compliance citations
• exact stack traces for debugging
• code review on full diffs
• extraction from messy text
• verification against original evidence

But even then, pass the exact slice that matters.
Not the entire upstream universe.

The rule I use now

This is the heuristic that cleaned up most of my agent workflows:

• If information is needed for reasoning, summarize it.
• If it is needed for retrieval, store it.
• If it is needed for verification, pass the exact slice.

That rule has saved me more latency, more cost, and more agent quality than swapping models ever did.

Why this matters even more when you pay per token

This problem gets expensive fast in production.

If your agent runs 24/7 in n8n, Make, Zapier, OpenClaw, or a custom loop, every extra blob you keep replaying becomes recurring cost.

That’s one reason teams end up obsessing over token usage dashboards instead of fixing the architecture.

A cleaner state design reduces prompt bloat at the source.

And if you are using an OpenAI-compatible endpoint with flat monthly pricing, this gets even nicer because you can optimize for agent quality and throughput instead of constantly asking whether one more tool call is going to spike the bill.

That’s the part I like about Standard Compute: it works as a drop-in OpenAI API replacement, so you can keep your existing SDKs and agent code, but stop thinking in per-token panic mode while you clean up orchestration.

The architecture fix still matters.
You just get to apply it without token anxiety hanging over every experiment.

Practical checklist

If your agent feels slower and dumber over time, check these first:

• Are you pasting raw tool output into every next prompt?
• Are you storing canonical results outside the model?
• Are you summarizing tool results before the next reasoning step?
• Are you passing exact evidence only when verification is required?
• Are you using prompt caching only for stable prefixes?
• Are you trimming memory in n8n or your orchestration layer?
• Are you treating the model like a reasoning engine instead of a database?

If you fix just those, you will probably get a bigger improvement than another model bake-off.

I went looking for a model fix.

What I found was an architecture fix.

Most “context window problems” are really just us using chat history as a storage layer.

Once you stop shoving raw tool output back into the next prompt, agents get faster, cheaper, and noticeably less confused.

Not because the models changed.

Because we finally stopped making them carry our state management mistakes.