Canary an AI Code Reviewer With Stale-Context Failures and a Replay Gate

The merge that passed because the reviewer saw a different diff

The pull request looked clean at 09:41, and the AI reviewer said "Approved" with a confidence I now understand was cosmetic. We merged it, the nightly build turned red, and the data race we had hunted for three days was introduced by a diff the reviewer never actually saw. The pipeline had built the prompt from a cache while CI was already evaluating a newer commit.

Was the model weak? No. The model was given stale context, and every downstream verdict system blindly trusted the output. A review verdict is not an opinion; it is a derived value, and derived values are only as good as their input snapshot.

The invariant your review gate is missing

A review verdict is trustworthy only when it is replayable: given the same diff SHA, the same rules file, and the same model version, the gate must be able to reproduce the exact same decision. If you cannot replay a verdict, you cannot audit it; if you cannot audit it, you cannot evaluate the reviewer. Most teams treat the AI reviewer as a black box and the verdict as an opaque string, and that is precisely the mistake.

The verdict is a data-flow output, and data-flow outputs need a stable input contract, a detector for stale or truncated inputs, and a decision that says reject, replay, or compensate. Once you see it that way, the whole design problem changes from prompt tuning to protocol design.

Declared assumptions

Before we go further, let me be explicit about the constraints I am assuming.

1. You control the prompt builder and can snapshot the exact prompt text.
2. The model endpoint is external, so its availability, latency, and sampling are not under your control.
3. CI can provide a monotonic diff SHA for the head commit being reviewed.
4. Duplicate delivery is possible and must never double-record or double-act.
5. The gate is advisory or blocking by configuration, but it always logs the snapshot.

If any of these assumptions does not hold on your stack, the design below needs adaptation before it gives you safety.

Data flow: from push to verdict

Here is the system I draw in my head. Push → CI → diff fetcher → prompt builder → model gateway → verdict queue → gate → branch status. The critical property is not model quality; it is that every arrow in this flow carries a snapshot identifier.

sequenceDiagram
    participant CI as CI Runner
    participant DB as Diff Store
    participant PB as Prompt Builder
    participant MX as Free Model Endpoint
    participant VQ as Verdict Queue
    participant G as Gate

    CI->>DB: put(diff_sha, patch)
    CI->>PB: build(diff_sha)
    PB-->>DB: get(diff_sha)
    PB->>MX: prompt + snapshot_sha
    MX-->>PB: verdict (maybe partial)
    PB->>VQ: enqueue(snapshot_sha, verdict)
    VQ->>G: evaluate(snapshot_sha)
    G-->>CI: reject | replay | pass

The failure I keep seeing in production is subtle: the arrow from PB to DB returns a cached patch whose SHA does not match the head commit CI is reviewing. When that happens, the verdict is computed on a different world than the one the branch actually contains.

The artifact: a replayable verdict runner

The fix is not a better model; it is a state machine around the review call. Here is the minimal runner I use, and please treat it as pseudocode that you adapt to your own stack.

# review_runner.py — replayable verdict gate (pseudocode)
def review(head_sha, rules_sha, model_version):
    snapshot = {
        "head_sha": head_sha,
        "rules_sha": rules_sha,
        "model_version": model_version,
        "cursor": get_cursor(head_sha),
    }
    patch = diff_store.get(snapshot)
    if patch is None or patch.base_sha != head_sha:
        return {"action": "reject", "reason": "stale_context"}
    prompt = build_prompt(patch.body, rules(snapshot))
    snapshot["prompt_sha"] = sha256(prompt)
    if verdict_exists(snapshot):
        return {"action": "pass", "replayed": True}
    verdict = call_model(snapshot, prompt)
    record(snapshot, verdict)
    if verdict.truncated:
        return {"action": "replay", "backoff": cursor_backoff()}
    return decide(patch, verdict)

Notice what this runner actually enforces. The stale-context check is not a model heuristic; it is a plain hash comparison between the diff SHA used by the prompt and the head SHA that CI is reviewing. The replay path uses the snapshot key for idempotency, so a duplicate message is never recorded twice.

Failure classes and gate actions

Here is the decision table I validate every time the endpoint degrades.

Failure class	Detection	Gate action
Stale diff SHA	snapshot.head_sha != patch.base_sha	reject
Truncated mid-stream verdict	verdict.truncated flag	replay with backoff
Duplicate delivery	snapshot key already exists	pass, no re-record
Response timeout	no verdict within deadline	replay with same key
Model version drift	version changed between attempts	reject, force re-eval
Rules changed mid-queue	rules_sha mismatch	reject, rebuild prompt

Each row maps to exactly one of three actions, and that is the whole design. The gate never guesses and never silently accepts.

Tradeoffs you should weigh honestly

There is no free lunch, and this gate is no exception.

You gain	You pay
Auditable and replayable verdicts	Prompt snapshots and hashes need storage
Protection from stale context	One extra hash comparison per review
Deterministic duplicate handling	A replay state machine to maintain
Clear blame attribution	Verdict latency can grow under retries

The storage cost is the real one. Prompt snapshots for large diffs add up, so I keep them in object storage and retain only the verdict by default, with the prompt retrievable by SHA on demand.

Validation path: run it against a real endpoint

Now let me get practical. I have been running this harness against MonkeyCode's free model endpoint, with the runner deployed on their free server option, so the whole experiment cost me nothing but time. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow is simple, and I recommend you run it before trusting any reviewer, free or paid:

1. Create a fixture repo with two branches where one clearly introduces a data race.
2. Run the runner against both branches and verify the verdict table reflects the actual code.
3. Inject staleness by pinning an older diff SHA and confirm the gate rejects instead of passing.
4. Kill the endpoint mid-stream and confirm the runner replays with the same snapshot key.
5. Assert that replays converge on the same verdict, and that duplicate messages do not double-count.

That last assertion is the one people skip, and it is exactly where the bug lives. A quick fixture run looks like this:

git clone https://example.invalid/race-fixture
python review_runner.py run --head main --rules rules.yaml
python review_runner.py inject-stale --head main --stale HEAD~1
python review_runner.py assert-convergence --head main

Limitations and who should not use this

Two caveats before you copy this. The gate protects the review data flow, but a replayable wrong verdict is still wrong, so it cannot fix a model that systematically misses bugs. The design also adds storage and state; if your PRs are tiny, your reviews are fast, and your team is three people, the hash checks may be more ceremony than value.

External endpoints change quotas and versioning without notice, so always read the current public docs before assuming any free tier behaves like your fixture. If you cannot keep a stable prompt snapshot or cannot tolerate retry latency, do not adopt this gate as a hard merge block.

The question you should answer before merging

I end every design review the same way. Which event order breaks this invariant — CI updates the head commit while the reviewer still answers the old diff, the old verdict lands in the queue after the new commit is pushed, and the gate sees a matching snapshot hash? Should the system reject that verdict, replay the prompt against the new diff, or compensate by scheduling a re-review of the final merged commit? Your answer is your review policy.