Postmortem: A Cached Verdict Passed a Diff the Model Never Saw

A verdict cache made the patch gate cheap and slow to notice. The gate kept returning green. The green belonged to an older diff.

Root cause in one line: the cache key covered the prompt and the model name. It never covered the code.

The gate itself worked. The lookup around it lied.

What the gate was supposed to do

• Send a diff, a rules file, and a review prompt to a model.
• Parse the reply into a structured verdict: pass, block, or unknown.
• Cache the verdict so repeated diffs skip the model call.
• Block the merge unless the verdict is pass for the exact revision under review.

Steps one, two, and four were correct. Step three quietly broke step four.

Timeline

1. T+0 — A caching layer is added to reduce review cost. Key = sha256(model + prompt).
2. T+0, same hour — Cache hit rate looks healthy. Review latency drops. Nobody asks what a hit proves.
3. T+9 days — The rules file is split into a system message. The user message now carries only the diff.
4. T+9 days — A context trimmer shortens the system message to fit the free tier window. It truncates silently and logs nothing.
5. T+11 days — A refactor changes behavior in three files. The gate returns pass in 40 ms. No model call occurs.
6. T+11 days — The pass was cached from a diff that touched none of those files.
7. T+14 days — A reviewer notices a passing badge on a diff that removes an authorization check.
8. T+14 days — The badge is traced to a cache entry whose stored diff hash differs from the reviewed revision.
9. T+15 days — Cache disabled. Review volume returns. The false pass is confirmed in 6 of the sampled entries.

Nothing in that list is exotic. No attacker. No outage. Only a key that was too small for the question.

The reproduction harness

Run this to see the collision without any model access. It is a unit test, not a benchmark.

import hashlib, json

def cache_key_v1(prompt: str, model: str) -> str:
    # The original key: cheapest fields only.
    return hashlib.sha256(f"{model}:{prompt}".encode()).hexdigest()

def envelope(diff_sha: str, files: list[str], system: str) -> dict:
    return {
        "model": "free-tier-route",
        "system": system,
        "diff_sha": diff_sha,
        "files": sorted(files),
        "prompt": "review this patch against the rules",
        "prompt_version": "v3",
    }

def test_collision_is_real():
    a = envelope("aaa111", ["src/auth/login.py"], "rules: v3")
    b = envelope("bbb222", ["src/billing/invoice.py"], "rules: v3")

    # Same prompt, same model, different code -> same key.
    assert cache_key_v1(a["prompt"], a["model"]) == cache_key_v1(b["prompt"], b["model"])

The assertion passes. That is the bug. A cache key that cannot separate two unrelated diffs is not a cache key. It is a random number generator with good manners.

Contributing factors

1. The key was built from the cheapest fields

The prompt was long and boring. It was also identical for every patch. Hashing it produced a stable key and zero information about the diff. Convenient fields are rarely sufficient fields.

2. Context trimming changed the input silently

The trimmer respected a token window and nothing else. When it cut the system message, the effective request changed. The key did not.

3. A cache hit was treated as evidence

A hit proved that some request had been reviewed. It proved nothing about this request. The pipeline collapsed two different claims into one green badge.

4. No revision binding

Entries stored a verdict. They did not store the revision the verdict belonged to. A stale verdict therefore had no way to announce itself.

5. Free tier fallback hid the cost signal

When the primary route was unavailable, a fallback route served the request. Both routes shared the same cache namespace. Failover changed the reviewer without changing the key.

The durable fix

Four changes, ordered by importance.

Fix 1: Content-addressed keys

def cache_key_v2(env: dict) -> str:
    parts = {
        "model": env["model"],
        "system": env["system"],          # post-trim, not pre-trim
        "diff_sha": env["diff_sha"],
        "files": env["files"],            # sorted upstream
        "prompt": env["prompt"],
        "prompt_version": env["prompt_version"],
    }
    blob = json.dumps(parts, sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(blob.encode()).hexdigest()

def test_collision_is_gone():
    a = envelope("aaa111", ["src/auth/login.py"], "rules: v3")
    b = envelope("bbb222", ["src/billing/invoice.py"], "rules: v3")
    assert cache_key_v2(a) != cache_key_v2(b)

Hash what was actually sent. Not what the caller intended to send.

Fix 2: Bind every entry to a revision

entry = {
    "key": key,
    "base_sha": git_head_sha(),
    "diff_sha": diff_sha,
    "created_at": now_iso(),
    "verdict": verdict,
    "raw_reply": raw_text,
}

A lookup with a mismatched base_sha is a miss, even on a key hit. Rebase invalidates the verdict by construction.

Fix 3: Cache transcripts, not verdicts

Store the raw model reply and the envelope hash. Recompute the verdict locally on every read. Parsing is cheap and deterministic. Caching the parse means caching a parser bug forever.

Fix 4: Fail closed on anything unknown

• Missing diff_sha → block.
• Unknown prompt_version → block.
• Truncated system message → block and log the trim.
• Empty model reply → unknown, never pass.

A gate that cannot prove a pass must refuse the patch. Refusal is a valid verdict. Silence is not.

Cache reuse rules

Condition	Reuse verdict?	Action
Key hit, same base_sha, same diff_sha	Yes	Return stored verdict
Key hit, different diff_sha	No	Re-review
Key hit, different provider route	No	Records are route-scoped
Key hit, entry older than TTL	No	Re-review
Truncation flag set	No	Re-review, then fail closed
Parse error on stored reply	No	unknown, block

Six rows. They fit on a sticky note. That is the point.

Where free model access fits the workflow

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate above needs many cheap review calls. Broad eval sweeps are exactly where a free model access tier helps. The operator describes MonkeyCode as an open-source project offering free model access and a free server option. Both claims come from the operator, not from independent testing here. Check the project's own documentation for current limits, supported models, and server capacity before relying on them.

The practical split used here:

• Free model access — prompt iteration, parser regression runs, and dry runs against the rules file.
• Free server option — hosting the stateless gate runner plus the content-addressed cache.
• Paid or self-hosted capacity — anything needing a durability or latency commitment the free tier does not promise.

The gate stayed the same shape across all three. Only the host and the route changed. That portability came from the key design, not from the provider.

Limitations and who should not use this

• No SLA, no capacity guarantee, and no permanence claim is made for any free tier.
• This workflow assumes a deterministic verdict parser. Free-text-only review does not fit.
• Teams without revision pinning should skip the cache entirely and re-review.
• Regulated pipelines needing audited third-party review should not route through a free tier.
• High-volume monorepos will outgrow a small single-host runner.

If the cache cannot be keyed correctly, delete it. A slow correct gate beats a fast wrong one.

Pre-flight checklist

1. Does the key include the post-trim system message and the diff hash?
2. Does every entry carry base_sha and diff_sha?
3. Does a provider switch change the namespace?
4. Does truncation force a re-review?
5. Does an empty reply block the merge?

Five questions. Each one catches a failure described above. Run them before adding the cache back. If a free tier fits that checklist, try the MonkeyCode gateway for the eval sweep and keep the gate logic yours.