We kept arguing about GPT vs Claude and missed the 2x long-context cost blowing up our agent runs

A thing finally clicked for me while debugging an agent trace:

we were spending way too much time arguing about model choice, and not enough time looking at the giant pile of context getting replayed on every step.

Not user intent.
Not fresh instructions.
Just baggage.

Old tool output. Debug blobs. Previous assistant replies. Summaries of summaries. Random thread history nobody trimmed.

That stuff is easy to ignore because it accumulates one harmless-looking decision at a time.

But in a lot of agent workflows, especially tool-heavy ones, that is where the bill gets weird.

The hidden cost wasn't the model switch

The contrarian take:

For a lot of agent pipelines, repeated input context matters more than the headline per-token rate.

OpenAI's current pricing makes this painfully obvious. Long-context input is priced at 2x short-context input across the lineup, and long-context output is 1.5x higher too.

One concrete example:

• gpt-5.6-terra short-context input: $1.00 / MTok
• gpt-5.6-terra long-context input: $2.00 / MTok
• gpt-5.6-terra short-context output: $6.00 / MTok
• gpt-5.6-terra long-context output: $9.00 / MTok

Same model. Same app. Same team.

The only difference is that one request is dragging around more history.

That means a bloated thread can become a pricing decision all by itself.

What this looks like in real agent code

A lot of memory bugs don't look like bugs.

They look like normal code:

from langchain.agents import create_agent
from langgraph.checkpoint.memory import InMemorySaver

agent = create_agent(
    model="google_genai:gemini-3.6-flash",
    tools=[get_user_info],
    checkpointer=InMemorySaver(),
)

thread_config = {"configurable": {"thread_id": "1"}}

Nothing scary there.

But if that thread stays alive across multiple tool-heavy runs, the agent can keep pulling old state into new calls unless you explicitly trim or summarize it.

And once tools are involved, each turn can add:

• system instructions
• user messages
• assistant replies
• tool calls
• tool outputs
• intermediate state
• summaries
• more summaries later

A simple chatbot grows slowly.
A workflow agent grows like a log file nobody rotates.

The biggest source of junk: tool output

This is the one I see most often.

A tool returns a huge JSON payload, and someone decides to pass the whole thing back into the next model call "just in case."

Example:

{
  "customer_id": "cus_123",
  "tickets": [... 400 records ...],
  "recent_orders": [... 80 records ...],
  "audit_log": [... 1200 events ...],
  "crm_notes": "very long string..."
}

Then the next step asks something tiny like:

Should we escalate this support ticket?

That decision probably needs 10 facts, not 10,000 tokens.

A better pattern is:

raw_result = fetch_customer_context(customer_id)
store_raw_result(raw_result)

summary = {
    "customer_id": raw_result["customer_id"],
    "open_ticket_count": len(raw_result["tickets"]),
    "recent_order_count": len(raw_result["recent_orders"]),
    "has_refund_last_30_days": has_recent_refund(raw_result),
    "priority_signals": extract_priority_signals(raw_result),
}

messages.append({
    "role": "tool",
    "content": json.dumps(summary)
})

Keep the raw data outside the prompt.
Pass only the facts needed for the next decision.

Observability data is not prompt data

LangSmith traces are useful.
OpenTelemetry spans are useful.
Debug logs are useful.

That does not mean they belong in your next model call.

I've seen teams accidentally turn observability into recurring token spend by mirroring traces back into prompts so the model is "fully informed."

That usually does two bad things at once:

1. increases cost
2. makes the model worse

Too much stale context doesn't just cost more. It distracts the model.

If your agent keeps seeing old tool results, outdated instructions, and irrelevant history, it starts anchoring on the wrong stuff.

So this is not only a pricing problem.
It's also an accuracy problem.

Doesn't prompt caching solve this?

No.

It helps, but it does not make sloppy context free.

OpenAI still charges for cached input and cache writes.
For gpt-5.6-terra:

• cached input short-context: $0.10 / MTok
• cached input long-context: $0.20 / MTok
• cache write short-context: $1.25 / MTok
• cache write long-context: $2.50 / MTok

Anthropic has the same basic story with different numbers. Claude Opus 4.6 pricing includes:

• input: $5 / MTok
• output: $25 / MTok
• 5-minute cache write: $6.25 / MTok
• 1-hour cache write: $10 / MTok
• cache hit/refresh: $0.50 / MTok

Google Gemini also treats caching as a paid feature. Gemini 3.8 Flash in the paid tier is:

• input: $1.50 / MTok
• output: $3.75 / MTok
• context caching: $0.15 / MTok
• cache storage fee starts in 2027

So yes, cache stable instructions and repeated reference material.

But don't confuse:

• discounted repetition

with:

• no consequence

The pricing differences that actually matter

Model pricing matters.
Model quality matters.

If Claude Opus 4.6 solves a planning task that Claude Haiku 4.5 keeps fumbling, paying more can be completely rational.

If GPT-5 handles tool use or reasoning better for your workload, same story.

But if your workflow is replaying irrelevant history on every step, even the cheaper model becomes an expensive habit.

Here's the cleaner way to think about it:

Model / pricing view	Key numbers
OpenAI short vs long context	gpt-5.6-terra: short input $1.00/MTok, long input $2.00/MTok; short output $6.00/MTok, long output $9.00/MTok; cached input $0.10/$0.20; cache writes $1.25/$2.50
Anthropic Claude Opus 4.6	Input $5/MTok, output $25/MTok, 5-minute cache writes $6.25/MTok, 1-hour cache writes $10/MTok, cache hits/refreshes $0.50/MTok
Anthropic Claude Sonnet 5	Input $2/MTok, output $10/MTok
Anthropic Claude Haiku 4.5	Input $1/MTok, output $5/MTok
Google Gemini 3.8 Flash	Input $1.50/MTok, output $3.75/MTok, context caching $0.15/MTok, batch pricing is 50% lower

If you only compare vendors, you'll debate forever.

If you inspect your traces, you'll ask the better question:

why is this workflow paying to remember things it no longer needs?

5 practical fixes that reduce context without breaking the workflow

You do not need a PhD in memory architecture for this.

You need rules.

1. Stop replaying raw tool output by default

If Salesforce, Jira, GitHub, Discord, or your internal API returns a giant object, do not feed the whole thing back into the model unless the next step actually needs it.

Summarize first.
Store the raw payload elsewhere.

2. Split working memory from audit memory

Your prompt is not your log sink.

Keep detailed traces in LangSmith, OpenTelemetry, Datadog, ClickHouse, BigQuery, or wherever you want.

But only pass the model what it needs for the next decision.

Working memory and audit memory should be different structures.

3. Summarize at step boundaries

After every major tool call, compress the result into:

• facts
• decisions made
• unresolved questions
• next action

Example:

def compress_tool_result(result: dict) -> dict:
    return {
        "facts": extract_facts(result),
        "decisions": extract_decisions(result),
        "open_questions": extract_open_questions(result),
        "next_action": suggest_next_action(result),
    }

This is one of the easiest ways to keep multi-step agent runs from ballooning.

4. Reset threads more often

A lot of workflows should start fresh more often than they do.

If an n8n flow or Make scenario is really a new job, give it a new thread.

Don't let today's invoice lookup inherit yesterday's support escalation history just because both touched the same account.

5. Keep stable instructions stable

Large reusable instructions can benefit from caching.

But volatile task state should stay separate.

A small message list is often healthier than a giant rolling transcript:

messages = [
  {"role": "system", "content": "You are a helpful assistant."},
  {"role": "user", "content": "Summarize the latest support ticket and decide whether it needs escalation."}
]

response = model.invoke(messages)

Simple beats clever more often than agent builders like to admit.

Quick way to audit your own agent

If you want a fast sanity check, log token counts and payload sizes per step.

For example:

def log_prompt_stats(step_name: str, messages: list[dict]):
    total_chars = sum(len(m.get("content", "")) for m in messages if isinstance(m.get("content"), str))
    print({
        "step": step_name,
        "message_count": len(messages),
        "approx_chars": total_chars,
    })

Or if you're debugging a workflow locally, dump request bodies and inspect them:

jq '.messages | length' request.json
jq '.messages[].role' request.json
jq -r '.messages[].content' request.json | wc -c

You will usually find one of these:

• repeated system instructions
• giant tool payloads
• stale conversation turns
• duplicated summaries
• debug data that should never have been in the prompt

What if you actually need long context?

Sometimes you do.

Codebase-wide reasoning, contract analysis, compliance workflows, and deep research can genuinely need large windows.

I'm not saying long context is fake.
I'm saying default long context is lazy architecture.

If you need it, use it deliberately.
Route to it selectively.
Budget for it.

Treat expensive inference modes like an operational choice, not an accidental side effect of thread sprawl.

The weird part: oversized context can make agents dumber

This was the part that surprised me most.

I expected bloated context to cost more.
I didn't expect it to so reliably make agents worse.

But once you see it, it's obvious.

The agent starts:

• anchoring on stale instructions
• revisiting old tool results
• answering the question from three turns ago
• slowing down because every call carries too much baggage

So yes, model choice matters.

But if your memory policy is a mess, benchmarking GPT-5 vs Claude Opus 4.6 vs Grok 4.20 is a little like comparing race cars while towing a dumpster.

My practical takeaway

Before you switch models, inspect what your agent is dragging into every call.

Cut stale turns.
Summarize tool output.
Separate traces from prompts.
Reset threads more often.
Use long context only when the task actually needs it.

Most teams do not have a model problem first.

They have a context hygiene problem with a pricing symptom.

One more thing: this is exactly why flat-rate AI compute is appealing

If you're running agents all day in n8n, Make, Zapier, OpenClaw, or custom workflows, per-token billing pushes teams into constant cost babysitting.

And the worst part is that a lot of the cost comes from architecture mistakes like context sprawl, not from doing more useful work.

That's a big reason products like Standard Compute are interesting: you get an OpenAI-compatible API, flat monthly pricing, and dynamic routing across models like GPT-5.4, Claude Opus 4.6, and Grok 4.20 without every extra tool trace turning into another finance discussion.

You still want clean prompts. Bad context can still hurt quality.

But removing per-token anxiety makes it much easier to build agents that run continuously without treating every long thread like a budget incident.

If you're building serious automations, that's a much better default than staring at token dashboards all week.