My upstream raised cache-hit pricing 15x overnight. Here's what I changed.

On September 1 my upstream provider repriced one of the models I depend on. Here's the part of the price sheet that got my attention:

	Before	After
Input	¥1 / M	¥3 / M
Output	¥2 / M	¥9 / M
Cache hit	¥0.02 / M	¥0.30 / M

The output jump is 4.5x. Annoying, but it's the number everyone looks at, and honestly it's the one I expected.

The cache-hit jump is 15x. And that's the one that actually changed my bill.

Why cache hit mattered more than output

Most of my traffic reuses the same long system prompt. Thirty or forty lines of instructions, tool definitions, a few examples. That prefix is byte-identical across thousands of calls a day, so it was being served from cache at the old ¥0.02 rate — essentially free.

I had optimized for output tokens because that's what the pricing pages scream about. But when you do the arithmetic on my actual traffic:

• system prompt: ~2,400 tokens, cached, called 3,000 times/day
• user message + response: ~700 tokens total

At the old rates the cached prefix cost me roughly ¥0.14/day. At the new rates the same traffic costs about ¥2.16/day. The output increase added less than a third of that.

The lesson isn't "caching got worse." It's that I had no idea what my cache hit rate was, so I couldn't see this coming.

Read your own cache hit rate first

Most OpenAI-compatible endpoints hand you this for free in the usage object. Here's a real response from my gateway:

"usage": {
  "prompt_tokens": 98,
  "completion_tokens": 22,
  "total_tokens": 120,
  "prompt_tokens_details": {"cached_tokens": 0},
  "prompt_cache_hit_tokens": 0,
  "prompt_cache_miss_tokens": 98
}

That's a cache hit rate of zero. Which makes sense: it's a one-off say OK request with no shared prefix. But it also means every request like it pays full input price.

A small wrapper that logs the ratio:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://keheai.com/v1",
    api_key=os.environ["KEHEAI_KEY"],
)

def ask(messages):
    r = client.chat.completions.create(
        model="deepseek-chat",
        messages=messages,
        max_tokens=300,
    )
    u = r.usage
    hit = getattr(u, "prompt_cache_hit_tokens", 0) or 0
    total = u.prompt_tokens or 1
    print(f"cache hit: {hit}/{total} = {hit / total:.0%}")
    return r.choices[0].message.content

Run that over a few hundred real requests and you'll know where you stand. I was at 71%, which I'd have guessed was "pretty good" — it wasn't, and 29% of my input volume was paying full price.

The fix: move everything that changes to the end

Prefix caching only matches on a byte-identical prefix. One differing character early in the prompt and everything after it misses.

The usual suspects that quietly break your prefix:

# Bad: timestamp at the top poisons the whole prefix
messages = [
    {"role": "system", "content": f"You are a support agent. Today is {datetime.now()}."},
    {"role": "system", "content": TOOL_DEFINITIONS},
    {"role": "user", "content": question},
]

# Good: stable content first, volatile content last
messages = [
    {"role": "system", "content": TOOL_DEFINITIONS},
    {"role": "system", "content": "You are a support agent."},
    {"role": "user", "content": f"Today is {datetime.now()}.\n\n{question}"},
]

Other things that bite the same way:

• Request IDs or UUIDs in the system prompt. Put them in the user turn or a header.
• Shuffled tool definitions. If you build the tool list from a dict or a set, the order can change between calls. Sort it deterministically.
• A/B test flags baked into the system prompt. Keep the system prompt constant, vary the user turn.
• Few-shot examples loaded from a database in nondeterministic order. Sort them once, cache the string.

After reordering, my hit rate went from 71% to 94%. On the new pricing that's the difference between paying ¥0.30/M on 71% of my prefix volume versus 94% of it.

The second thing I changed: stop hardcoding prices

This is the one that actually stung, and it's not about caching.

I had a price table in a config file. It was wrong the moment the upstream changed their sheet, and I found out from a margin calculation, not from the provider. Upstream LLM pricing moves fast — I've seen a provider revise a sheet multiple times within two days.

If you resell or budget on top of these APIs, treat the price as a runtime value:

# Don't:
PRICES = {"deepseek-chat": {"in": 1.0, "out": 2.0}}

# Do: fetch it, timestamp it, alert when it moves
def check_price_drift(fetched, last_known, tolerance=0.05):
    drifted = {}
    for model, price in fetched.items():
        old = last_known.get(model)
        if old and abs(price["in"] - old["in"]) / old["in"] > tolerance:
            drifted[model] = (old["in"], price["in"])
    if drifted:
        alert(f"upstream price moved: {drifted}")
    return fetched

A 15x move on a line item you assumed was free is exactly the kind of thing that only shows up if someone is watching.

What I'd tell you to do today

1. Log prompt_cache_hit_tokens / prompt_tokens on real traffic for a day. One number.
2. If it's under 90% and your workload has a stable preamble, go hunting for whatever is mutating at the front of your prompt.
3. Put a timestamp on your upstream price sheet and diff it nightly.

None of this needs a new vendor or a new model. It's about measuring the line item that nobody puts on a dashboard.

I run a small OpenAI-compatible gateway, so I see these sheets up close — happy to answer questions about how the caching behaves on different model families.