I Thought the Model Drifted. My Cache Key Was Serving Tuesday.

Have you ever watched an LLM endpoint return a clean answer that belonged to a different prompt entirely? I spent forty-eight hours blaming sampling noise, temperature, and a free model that would not sit still. The request logs looked honest enough, and the health check on the box stayed green the whole time. The bug was quieter than that: a cache key that hashed the user message and ignored everything else that actually changes a completion.

I was trying to keep a small eval loop cheap, which is a very ordinary instinct. Free-model access is useful when you want overnight volume without treating every call as precious. I parked a thin HTTP wrapper on a free server, hashed each prompt, and stored the JSON body on disk so retries would not hammer the model. Does that sound reasonable? It did, until two different system prompts started colliding on the same key and I spent a day chasing "nondeterminism" that was just a hash.

I ran that wrapper against MonkeyCode's free model access on the free server option because I wanted a boring place to reproduce the cache bug, not a production SLA. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Nothing below depends on a named model, a quota, or a hardware claim. The lesson is the key function, and it still applies if you delete the product name from the stack.

What I walked into

The wrapper looked like every weekend cache I have written under time pressure. Incoming POST bodies were reduced to user_message, run through hashlib.sha256, and written under ./cache/<hex>.json. A hit returned the file. A miss called the model, then wrote the file. I even logged X-Cache: HIT so future-me would feel scientific.

That design has one attractive property and one fatal one. The attractive property is that identical user text becomes free after the first call. The fatal property is that user text is not the request. System prompt, temperature, stop sequences, tool schemas, and even a date injected into the preamble all change the meaning of "the same prompt." If those fields are missing from the key, the disk is not a cache. It is a costume party for old answers.

Hours 0–24: the wrong suspects

I did what I always do when a free model looks flaky: I assumed the model was the flaky part. Here is the ordered list of dead ends, because the order is the lesson.

1. I pinned temperature to 0 in the client and celebrated too early. Hits still disagreed with the live miss I forced with curl.
2. I added a request id header and grepped logs by id. The id was unique. The body on disk was not.
3. I tailed the process and watched wall-clock latency. Hits were 3–8 ms. Misses were hundreds of milliseconds. That only proved the cache was working as a cache, not that it was keyed correctly.
4. I compared pretty-printed JSON from two "identical" calls. The answers were fluent, on-topic, and for a slightly different instruction set. That is the meanest failure mode, because it does not look like an error.

A command that wasted a surprising amount of time:

# example: this proves latency, not correctness
curl -s -D - -o /tmp/a.json -X POST http://127.0.0.1:8080/v1/complete \
  -H 'Content-Type: application/json' \
  -d '{"user":"summarize this log"}'

Latency is a terrible oracle for cache identity. Fast and wrong still prints X-Cache: HIT.

Hours 24–48: the file that should not have existed

The break arrived when I stopped reading answers and started reading filenames. Two requests that I could distinguish in the access log were writing the same 64-character hex. Once you see a collision on disk, the model debate is over. The model never got a chance to drift, because the second call never reached it.

I dumped the cache directory and grouped by mtime. Several keys had been overwritten hours apart by payloads that only shared the user sentence. The system prompt had changed. The temperature had changed. In one case I had added a stop list and forgotten it, which is the kind of foot-gun you only notice when you diff the request objects instead of the prose.

# example: find keys that were overwritten rather than created
find ./cache -name '*.json' -printf '%T@ %p\n' | sort -n | tail -n 20
sha256sum ./cache/*.json | awk '{print $1}' | sort | uniq -d

If uniq -d prints anything, you do not have a model problem. You have an identity problem.

The artifact: a canonical key and a test that fails on purpose

The fix is not "stop caching." Caching completions on a free server is still a good idea when the key is honest. The fix is a canonical request document, stable JSON, and a hash over that document. The test below is meant to be copied. It is labeled as an example because I am showing the method, not a published benchmark.

# example cache identity helper — stdlib only
from __future__ import annotations

import hashlib
import json
from typing import Any

KEY_FIELDS = (
    "system",
    "user",
    "temperature",
    "stop",
    "tools",
    "response_format",
    "seed",
)


def canonical_request(payload: dict[str, Any]) -> dict[str, Any]:
    doc = {}
    for field in KEY_FIELDS:
        if field not in payload:
            continue
        doc[field] = payload[field]
    # Stable separators and sorted keys so logically equal dicts hash equal.
    return json.loads(json.dumps(doc, sort_keys=True, separators=(",", ":")))


def cache_key(payload: dict[str, Any]) -> str:
    blob = json.dumps(
        canonical_request(payload), sort_keys=True, separators=(",", ":")
    ).encode("utf-8")
    return hashlib.sha256(blob).hexdigest()


def naive_key(payload: dict[str, Any]) -> str:
    # the bug I shipped: hash the user sentence only
    return hashlib.sha256(payload["user"].encode("utf-8")).hexdigest()

And the collision test that would have saved the first day:

# example tests — run with: python -m pytest test_cache_key.py -q
from cache_key import cache_key, naive_key

BASE = {
    "system": "Return JSON with keys ok and reason.",
    "user": "Is this log line an error?",
    "temperature": 0,
    "stop": ["\n\n"],
    "tools": [],
    "response_format": "json",
    "seed": 7,
}


def test_naive_key_collides_when_system_changes():
    other = dict(BASE)
    other["system"] = "Return plain text. Never mention JSON."
    assert naive_key(BASE) == naive_key(other)  # documents the bug


def test_canonical_key_changes_when_system_changes():
    other = dict(BASE)
    other["system"] = "Return plain text. Never mention JSON."
    assert cache_key(BASE) != cache_key(other)


def test_canonical_key_changes_when_temperature_changes():
    other = dict(BASE)
    other["temperature"] = 0.7
    assert cache_key(BASE) != cache_key(other)


def test_canonical_key_is_stable_under_key_reordering():
    shuffled = {
        "user": BASE["user"],
        "seed": BASE["seed"],
        "stop": BASE["stop"],
        "tools": BASE["tools"],
        "system": BASE["system"],
        "response_format": BASE["response_format"],
        "temperature": BASE["temperature"],
    }
    assert cache_key(BASE) == cache_key(shuffled)

If you already have a wrapper, drop naive_key next to it and assert it fails. That failing test is the artifact. A passing vibe check on two chatty answers is not.

Decision table: what belongs in the key

I used this table while I was deciding which fields were "part of the request" and which fields were just telemetry. Steal the table. Do not steal my original key function.

Field	Put it in the key?	Why
user	Yes	This is the obvious part, and still not sufficient.
system	Yes	Instruction changes are the collision I actually hit.
temperature / top_p	Yes	Sampling knobs change the distribution you think you cached.
stop	Yes	Silent truncation later looks like a different model.
tools / function schema	Yes	Tool-capable prompts are not the same request as prose-only.
response_format	Yes	JSON-mode versus markdown is a semantic fork.
seed	Yes, if the API honors it	If it is ignored, the key should not pretend otherwise.
request_id	No	Identity of the call is not identity of the completion.
Wall clock / Date header	Only if the prompt is time-sensitive	Caching "today's on-call" overnight is how Tuesday leaks.
Auth token	Never the raw token	Hash a stable tenant id if isolation is required.
Server hostname	No	That belongs in logs, not in the lookup key.

A useful rule showed up while I filled the table. If changing a field can change the bytes you would accept as correct, it belongs in the key. If changing a field only changes how you observe the call, it belongs in the log line.

What broke beyond the hash

The incomplete key was the headline, but it was not the only break. Disk on a free server is a place, not a promise. I restarted the process once and still had files, then restarted the box and did not. That is not a model failure either. It is an assumption about persistence that nobody wrote down.

I also learned that pretty-printed JSON in the cache file is a gift to diffs and a tax on disk. Compact JSON is enough if the key is right. Pretty JSON is for the autopsy directory, which I now keep separate from the hot cache. Mixing them made grep noisy enough that I missed the collision for several hours.

One more break that still annoys me: I logged the user message and not the canonical document. When two hits shared a key, the log could not explain why. After the fix, each write stores key, canonical_request, created_at, and source (hit or miss). The completion body is the payload. The metadata is how you argue with yourself tomorrow.

# example record shape written next to the completion
record = {
    "key": cache_key(payload),
    "canonical": canonical_request(payload),
    "source": "miss",
    "created_at": "2026-09-03T12:00:00Z",  # example timestamp only
}

Notice the timestamp is labeled as an example. I am not claiming a production incident window. I am claiming that a cache without provenance will waste the next forty-eight hours too.

What I would repeat

I would still cache. I would still use a free model and a free server for an overnight eval loop, because that is a fine place to practice boring infrastructure. I would not start with the model card, the temperature, or a theory of drift. I would start with two requests that a human can tell apart, and I would demand two keys.

The repeatable sequence is short enough to keep on a sticky note.

1. Write naive_key and cache_key side by side so the bug has a name.
2. Add tests for system prompt, temperature, stop lists, and tool schemas.
3. Log the canonical document, not just the user sentence.
4. Keep the hot cache compact, and keep an autopsy directory for overwritten keys.
5. Treat X-Cache: HIT as a performance signal, never as a correctness signal.

Would I ship this as a platform cache for end-user traffic? No. This is a lab wrapper. Lab wrappers become production incidents when somebody adds tenants and forgets to put the tenant in the key. That is the same bug with a higher blast radius.

Limitations, and who should not use this

This approach does not do semantic caching. Two paraphrases of the same question still miss, and that is intentional. If you want similar-question reuse, you are in embedding territory, and you inherit a different class of wrong answers. I am not selling that upgrade here because I did not run it in this loop.

Do not use a disk cache of raw prompts if the prompts contain secrets, customer text, or anything you would not write to an unencrypted notebook. A free server is a convenient scratch box. It is not a confidentiality boundary. Hashing the canonical request protects you from collisions. It does not protect the bodies on disk from whoever can list the directory.

Skip this whole pattern if you are scoring model quality. Cache hits will make a flaky model look stable, which is how I lost the first day. Eval traffic should bypass the cache or should key on a run id that never repeats. Caching is for cost control after you trust the request identity, not for manufacturing a reassuring pass rate.

Also skip it if your API ignores seed or silently drops unknown fields. A key that includes theater knobs will fragment the cache without buying determinism. Read the live request echo, not the docs you remember from another vendor.

Field notes I am keeping

The model was not drifting in this incident. The cache was confident, fast, and wrong in a way that still looked like language. That is a nastier outage than a 500, because nobody pages on fluency.

If you already have a completion cache, run the naive-versus-canonical tests against last week's code before you tune another sampling knob. What field did your key forget? Mine forgot the system prompt, then the stop list, then the fact that the disk could vanish on reboot. I will repeat the tests. I will not repeat the forty-eight hours of arguing with a hash.