I thought prompt caching was free money until my agent stack broke it on every single call

I had an agent workflow that looked like a perfect prompt-caching candidate.

Same developer prompt. Same tools. Same task shape. Long static instructions. On paper, it should have gotten cheaper over time.

It didn’t.

Every request behaved like a fresh request.

That sent me down a rabbit hole, and the answer was annoyingly simple:

Most cases of prompt caching not working are not model problems. They’re workflow problems.

If you’re using GPT-5, Claude, or any OpenAI-compatible stack through n8n, Make, Zapier, OpenClaw, LangChain, or your own orchestration layer, there’s a good chance your workflow is quietly mutating the prompt prefix on every run.

That kills the cache.

The thing people get wrong about prompt caching

A lot of people talk about prompt caching like it works on intent.

It doesn’t.

It works on exact rendered prefix matching.

That means the provider is not asking:

“Is this basically the same prompt?”

It’s asking:

“Is the prefix exactly the same bytes/tokens up to the cache boundary?”

That distinction is everything.

OpenAI’s docs are pretty clear: caching depends on the full rendered context, including system instructions, developer messages, tool definitions, and conversation history.

Anthropic’s docs say the same thing in a slightly different way: prompt caching depends on matching prefixes and cache breakpoints, whether you rely on automatic behavior or explicit cache_control.

So if your app says “same prompt,” but your middleware injected one line before the stable prefix ends, you do not have the same prompt.

You have a cache miss.

What was actually changing in my "same prompt"

This was the frustrating part.

If I looked at the user-facing prompt in a debug UI, it looked identical.

But the actual payload sent to GPT-5 or Claude had tiny mutations all over it.

The usual cache killers

These are the ones I keep seeing in real agent stacks:

• timestamps injected into system or developer instructions
• trace IDs appended by middleware
• JSON keys serialized in a different order
• user/account policy wrappers inserted before the static prompt
• tool definitions rebuilt in a slightly different order
• conversation history reformatted between runs
• tool traces or debug banners added too early in the request

Any one of those can break a cache hit.

And if the mutation happens early, everything after it becomes effectively uncached too.

That’s why dynamic junk near the top of the prompt is so expensive.

OpenAI makes this easy to misunderstand

OpenAI says prompt caching is enabled by default on supported models, and cached input tokens can be discounted by up to 90%.

That sounds great, and it is great when your prompt prefix is stable.

But a lot of developers hear “enabled by default” and mentally translate it to:

“My agent is automatically saving money.”

That’s where things go sideways.

Same session does not mean same cache hit.

Same prompt template does not mean same rendered prefix.

Same tool list in your code does not mean same tool payload over the wire.

A very normal OpenAI failure mode

You call GPT-5 through the Responses API.

You keep the same developer message and the same tools.

Then your middleware adds this before the request goes out:

{
  "trace_id": "run_2026_09_22_14_03_18",
  "timestamp": "2026-09-22T14:03:18Z"
}

Or your tool schema gets regenerated with keys in a different order.

Your app thinks it sent the same prompt.

OpenAI did not receive the same rendered prefix.

That means no cache hit.

Anthropic has a different trap: time

Anthropic’s prompt caching is solid, but the default ephemeral cache lifetime is 5 minutes.

That matters a lot in multi-step automations.

If your workflow streams a long response, waits on Airtable, calls a webhook, then comes back for a follow-up request, you may have already burned most or all of the cache window.

Anthropic also offers a longer write tier, but most teams remember the feature and forget the timing.

A normal Claude failure mode

Picture an n8n or Make flow using Claude Opus:

1. Load a giant static instruction block
2. Send request
3. Wait on another system
4. Send follow-up request expecting reuse

If too much time passed, the cache entry is gone.

Nothing is broken. Your workflow just treated a 5-minute cache like a permanent asset.

Here’s the basic shape:

import anthropic

client = anthropic.Anthropic()

response = client.messages.create(
    model="claude-opus-4-1",
    max_tokens=1024,
    cache_control={"type": "ephemeral"},
    system="You are an assistant that follows a fixed review rubric.",
    messages=[
        {"role": "user", "content": "Review this PR summary."}
    ]
)

That works fine.

But if your automation is slow, or your prefix changes before the cache breakpoint, the savings disappear.

How to tell if your stack is sabotaging prompt caching

Don’t inspect the pretty prompt template in your app.

Inspect the fully rendered request.

That means:

• the actual system/developer messages
• the exact tool definitions
• the exact conversation history
• any middleware-injected metadata
• the final JSON sent over the wire

If you’re building with Node, log the final request body before sending it.

import fs from "node:fs";

function logRenderedRequest(payload: unknown) {
  fs.writeFileSync(
    `./debug/request-${Date.now()}.json`,
    JSON.stringify(payload, null, 2)
  );
}

Then diff two supposedly identical requests.

diff -u debug/request-1.json debug/request-2.json

Or use jq to normalize and compare:

jq -S . debug/request-1.json > debug/sorted-1.json
jq -S . debug/request-2.json > debug/sorted-2.json
diff -u debug/sorted-1.json debug/sorted-2.json

If you see timestamps, reordered keys, changing wrappers, or tool definitions moving around, that’s your problem.

What actually works

Prompt caching is real.

It’s just extremely literal.

The pattern that works is boring and reliable:

• put stable static content first
• move volatile content later
• serialize structured data deterministically
• stop injecting per-run junk into the prefix

If you want a practical rule:

Audit the first 1,000 tokens of every request before you do anything else.

That’s usually where the damage is.

The 3 rules I trust now

1. Freeze the prefix

Keep system and developer instructions stable.

Do not inject timestamps, trace IDs, request IDs, or debug text before the cacheable block.

Do not rebuild tool definitions on every call unless you have to.

Bad:

{
  "system": "You are a support agent. Timestamp: 2026-09-22T14:03:18Z"
}

Better:

{
  "system": "You are a support agent."
}

If you need the timestamp, put it in a later user or metadata block that does not poison the static prefix.

2. Canonicalize structured data

If you send JSON blobs, tool schemas, or config objects, make them deterministic.

That means stable key ordering and stable serialization.

Example in Node:

function stableStringify(obj: unknown): string {
  return JSON.stringify(sortKeys(obj), null, 2);
}

function sortKeys(value: any): any {
  if (Array.isArray(value)) return value.map(sortKeys);
  if (value && typeof value === "object") {
    return Object.keys(value)
      .sort()
      .reduce((acc: Record<string, any>, key) => {
        acc[key] = sortKeys(value[key]);
        return acc;
      }, {});
  }
  return value;
}

Then reuse the same serialized output instead of regenerating equivalent-but-different objects every run.

3. Move volatility down

Put static instructions first.

Push dynamic state later.

That includes:

• user-specific wrappers
• retrieval results
• live tool traces
• changing conversation summaries
• run-specific debug metadata

This is one of the easiest ways to reduce context waste without making the agent dumber.

Practical comparison: OpenAI vs Anthropic prompt caching

Option	What matters in practice
OpenAI Prompt Caching	Enabled by default on supported models, but exact rendered prefix still has to match. Tool definitions, history, and developer messages all matter.
Anthropic Prompt Caching	Prefix matching still matters, but timing matters more because the default ephemeral cache lifetime is 5 minutes.
Workflow Design	Usually the real problem. Stable prefix first, deterministic JSON, and volatile content pushed later.

My take:

OpenAI’s default-on behavior is convenient, but it also makes people lazy. Teams assume the savings are automatic.

Anthropic forces you to think harder about cache boundaries and timing, which is annoying but often leads to better prompt architecture.

Neither vendor can save you from a messy orchestration layer.

Are cache misses always a bug?

No.

Sometimes the prefix really does need to change.

Examples:

• account-specific policy wrappers
• changing tool availability
• fresh retrieval results
• real conversation growth
• different safety or compliance constraints

That’s fine.

The problem is accidental cache breakage from stuff nobody meant to matter:

• hidden timestamps
• debug banners
• inconsistent JSON formatting
• middleware wrappers
• tool schemas generated in random order

That stuff is pure self-inflicted pain.

A quick checklist for debugging cache misses

If prompt caching “isn’t working,” here’s the checklist I’d use:

# 1. Capture the final request payload
# 2. Compare two runs that should match
# 3. Sort JSON keys before diffing
# 4. Check the first 500-1000 tokens for volatility
# 5. Check tool definitions for reordering
# 6. Check middleware for timestamps/trace IDs
# 7. For Anthropic, check elapsed time between requests

And the engineering checklist version:

• log final rendered requests
• diff identical runs
• canonicalize JSON
• pin tool schema versions
• move metadata out of the prefix
• keep long static instructions stable
• validate timing on multi-step Claude workflows

The fix was embarrassingly simple

Once I stopped treating prompt caching like magic and started treating it like a strict string-matching contract, the problem got much less mysterious.

We:

• pulled timestamps and trace metadata out of the prefix
• stabilized tool definitions
• made JSON serialization deterministic
• moved volatile wrappers later
• checked timing in multi-step Claude flows

After that, the “bad caching” problem looked a lot less like model behavior and a lot more like normal engineering.

That’s the main lesson.

If your agent stack keeps missing prompt cache hits, don’t start by blaming GPT-5, Claude, OpenAI, or Anthropic.

First assume your workflow is mutating the prefix in some stupid little way.

Because it probably is.

One more practical angle: cost predictability still matters even when caching works

Prompt caching helps, but it doesn’t fix the bigger budgeting problem for agent teams:

You still end up thinking about token economics all the time.

You’re still watching context growth, cache hit rates, retries, tool loops, and long-running workflows like they’re financial liabilities.

That’s a big reason teams running heavy automations are moving to flat-rate API setups like Standard Compute.

Standard Compute is a drop-in OpenAI-compatible API for agents and automations, with unlimited AI compute at a predictable monthly price. It works with existing OpenAI SDKs and fits neatly into n8n, Make, Zapier, OpenClaw, and custom workflows.

If your team is doing constant prompt/cost math just to keep agents running 24/7, predictable pricing is honestly as useful as prompt optimization.

Caching still matters. Good workflow design still matters.

But not having to panic about every token is a pretty nice upgrade too.

Comments

[TokenLat]

The "looks fresh every call" symptom is almost always a prefix-shift problem, not a model one. Agent loops quietly inject non-deterministic tokens — timestamps, tool-call IDs, prior-step output — ahead of your "static" instructions, so the cached block never stabilizes and you re-pay full price on every step. The cost you expected to amortize hides in the loop, not the first call. Do you track cache-hit rate per step, or only per request? That is usually where the miss lives.