Your Cache Key Is Not Your Prompt: Four Response-Cache Myths and a Collision Test

You put a cache in front of your model calls. The first week looks great. Then a ticket lands: same question, different answer. Sometimes.

Your prompt did not change. Your key did not change either. That is the bug.

Caching a model response is a claim about equivalence. Most teams never write that claim down. They hash a string and hope.

Here are four myths I keep hearing. Each one comes with a corrected mental model and a test you can run today.

Myth 1: "The prompt text is the cache key"

This is the most expensive myth. It feels obvious, so nobody reviews it.

A prompt string is only the input. The answer also depends on:

• the model identifier your endpoint actually resolves to
• decoding parameters such as temperature, top_p, max_tokens, stop sequences
• the system prompt template, which changes far more often than the version number
• the tool or function schema, even when no tool gets called
• the tenant, user, or document scope the answer was allowed to see

Hash the prompt alone and you get collisions between calls that must not share an answer. You will not see them in staging. You will see them in a support thread.

The corrected model: a key is a written equivalence claim

Two requests may share a cache entry only if a reviewer would say "these must produce the same answer." Write that sentence down before you write the hash.

A key builder you can actually test

The two small modules below are the whole harness. Copy them, then adapt the field names.

# cache_key.py
import hashlib, json

KEY_VERSION = "v3"

def template_sha(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()[:16]

def _canonical(obj) -> str:
    # sort_keys makes dict ordering irrelevant at every depth
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)

def cache_key(req: dict) -> str:
    material = {
        "key_version": KEY_VERSION,
        "tenant": req["tenant"],
        "model": req["model"],
        "template_sha": req["template_sha"],
        "params": req["params"],
        "tools": req.get("tools", []),
        "input": req["input"],
    }
    return hashlib.sha256(_canonical(material).encode("utf-8")).hexdigest()

Note the key_version field. When you change what counts as equivalent, bump it. Old entries then become unreachable instead of wrong.

hashlib and deterministic JSON serialization are the only primitives you need. If you prefer a standard for freshness and invalidation vocabulary, HTTP caching already has one; application-level response caches usually do not.

Four tests that catch real collisions

# test_cache_key.py
from cache_key import cache_key, template_sha

def base_request():
    return {
        "tenant": "acme",
        "model": "model-id-from-your-endpoint",
        "template_sha": template_sha("You are a terse assistant."),
        "params": {"temperature": 0, "max_tokens": 300},
        "tools": [],
        "input": "How do I rotate a log file?",
    }

def test_params_are_part_of_the_key():
    a = cache_key(base_request())
    r = base_request(); r["params"]["temperature"] = 0.9
    assert a != cache_key(r)

def test_template_edit_changes_key_without_a_version_bump():
    a = cache_key(base_request())
    r = base_request(); r["template_sha"] = template_sha("You are a verbose assistant.")
    assert a != cache_key(r)

def test_tenant_isolation():
    a = cache_key(base_request())
    r = base_request(); r["tenant"] = "other-customer"
    assert a != cache_key(r)

def test_key_is_stable_across_dict_order():
    a = cache_key(base_request())
    r = base_request(); r["params"] = {"max_tokens": 300, "temperature": 0}
    assert a == cache_key(r)

That last test matters more than it looks. Nondeterministic key order gives you a cache that silently never hits.

Myth 2: "A hit means the answer is still correct"

A TTL is a guess about staleness. It is not a proof of correctness.

Ask three questions before you trust a hit:

1. Did the input data change, or only the prompt text?
2. Did the model behind that identifier change?
3. Did a policy, price, or schema that the answer quotes change?

If the answer depends on live data, a time-based expiry is a coin flip. Prefer content-addressed invalidation: put the data version into the key, not just into the clock.

Myth 3: "Caching always cuts cost"

Caching moves cost around. It rarely deletes it.

Call shape	Cache it?	Why
Fixed template, temperature 0, stable inputs	Yes	Key is easy to defend
High temperature, creative output	No	No two answers are equivalent
Tool-using agent loop	Key per step, not per turn	Step inputs change constantly
Long-tail user prompts	Usually no	Hit rate near zero, review cost stays
Answers that quote live prices or quotas	Only with data version in key	Stale answer is a wrong answer

Count the misses too. A cache you must debug is not free.

Myth 4: "The cache is on my server, so it is mine"

An in-process dictionary on an ephemeral instance is not a cache. It is a coincidence.

• A restart empties it. Your hit rate becomes a function of uptime.
• Two instances hold two different views of the world.
• Stored payloads can outlive the request that created them unless you set a bound.

If the key omits tenant, the fastest path to a data leak is a cache hit.

A 20-minute verification workflow

Do this on a scratch deployment before touching production.

1. Write the equivalence sentence for one call path. One sentence, no hedging.
2. Implement cache_key with the fields that sentence names.
3. Run the four tests above. Add one test per new field you argued about.
4. Send the same request twice. Log hit / miss for both.
5. Change only the system template text. Confirm the second request misses.
6. Restart the instance. Measure how far hit rate falls.
7. Read one stored payload. If reading it makes you uncomfortable, encrypt or drop it.

For step 4 you need a place to send real calls cheaply. One option for that scratch environment is MonkeyCode: it offers free model access and a free server option, both as operator-supplied availability claims, not guarantees. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Measure elapsed time with a monotonic clock, never with wall time. Log these fields on every call:

• cache_key, truncated to 12 chars
• key_version and template_sha
• model as resolved, not as requested
• hit or miss
• elapsed_ms from time.monotonic()
• tenant

Six fields. That log answers almost every "why did it say that" question.

Limitations and who should skip this

This approach is not universal. Skip it, or get review first, if:

• you store raw prompts that contain personal or regulated data
• your answers must reflect live state with no version marker available
• you cannot name the model version your endpoint resolves to
• you run unsampled creative output and someone will still call a hit "wrong"
• your team has no owner for cache invalidation; an unowned cache rots quietly

And do not cache to win a latency argument you have not measured. Measure first. Then cache one path.

What to take away

A cache is a promise that two requests deserve one answer. Write the promise. Test the promise. Version the promise.

If your key is a hash of the prompt text, you have not made a promise. You have made a guess with a very fast lookup.

Run the four tests. If one fails, you just found a bug before a customer did.

Comments

[Alex Shev]

A response cache key needs to represent every input that can change the answer, not just the visible prompt string. Model version, system instructions, tools, retrieval corpus version, tenant, and relevant generation settings are all candidates. The collision test is a great idea because it turns that principle into an executable claim instead of an architectural hope.