Postmortem: Model Failover Replayed a Foreign Eval Receipt

A failover can approve a patch the second model never scored. The first receipt remains on disk unless the gate binds it. This lab postmortem records the reproduction, the factors, and the fix.

Scope of this write-up

This write-up is a controlled lab reproduction, not a customer incident. No production metrics, customers, or timings are claimed here. Readers should run the self-test before trusting the proposed gate.

The conclusion up front

A receipt must name the diff, prompt, model, and workspace generation. A matching diff hash alone is not a completed review. The gate fails closed when any bound field is missing.

Lab timeline

The reproduction uses one recycled workspace and two opaque model ids. Those ids are lab labels, not product model names. Each step below is a file check or a command.

1. T0 copies a prior receipt into a recycled job directory.
2. T1 records model A and writes a pass receipt for the diff.
3. T2 drops model A and selects model B for the retry.
4. T3 skips scoring because receipt.json already exists on disk.
5. T4 runs a naive gate that checks only the diff hash.
6. T5 compares the job log model with the receipt model id.
7. T6 runs the bound gate and rejects the foreign receipt.

Why the naive gate passed

The naive check compared one stored hash, then stopped looking. The diff bytes matched, so the process returned success. A recycled server makes that single-field check easy to fool.

def naive_accept(receipt, diff_sha256):
    return receipt.get('diff_sha256') == diff_sha256 and receipt.get('verdict') == 'pass'

That helper ignores model id and workspace generation entirely. It also ignores whether any review reasons were stored. File presence then looks like a finished review attempt.

Contributing factors

• The receipt used a fixed name shared by every attempt.
• Workspace reuse kept the previous score file after failover.
• Failover treated file presence as completion of the new attempt.
• The gate trusted a diff hash without the scorer identity.
• The agent and the gate shared one writable server directory.
• No attempt id linked the job log to the approved receipt.
• Cleanup ran only on success, so crashes left the old file.

What the durable fix changes

The fix does not ask a model to try harder. It refuses to certify a receipt from another attempt. Bootstrap mints a workspace generation before any agent starts.

The expected model id is captured when the attempt begins. The gate recomputes hashes and compares every bound field. A missing field is a failed job, not a soft warning.

Receipt binder

Save the following program as bind_receipt.py in the job image. The program targets Python 3.9 or newer on a normal CI image. It is a proposed local gate, not a hosted review service.

#!/usr/bin/env python3
'''Bind an eval receipt to diff, prompt, model, and workspace generation.'''

import argparse
import hashlib
import json
import sys
import tempfile
from pathlib import Path


def sha256_file(path: Path) -> str:
    digest = hashlib.sha256()
    digest.update(path.read_bytes())
    return digest.hexdigest()


def build_receipt(diff_path, prompt_path, model_id, workspace_gen, verdict, reasons):
    if verdict not in {'pass', 'fail'}:
        raise ValueError('verdict must be pass or fail')
    if not isinstance(reasons, list) or not reasons:
        raise ValueError('reasons must be a non-empty list')
    return {
        'diff_sha256': sha256_file(diff_path),
        'prompt_sha256': sha256_file(prompt_path),
        'model_id': model_id,
        'workspace_generation': workspace_gen,
        'verdict': verdict,
        'reasons': reasons,
    }


def find_mismatches(receipt, diff_path, prompt_path, model_id, workspace_gen):
    expected = {
        'diff_sha256': sha256_file(diff_path),
        'prompt_sha256': sha256_file(prompt_path),
        'model_id': model_id,
        'workspace_generation': workspace_gen,
    }
    mismatches = []
    for key, value in expected.items():
        if receipt.get(key) != value:
            mismatches.append(key)
    if receipt.get('verdict') != 'pass':
        mismatches.append('verdict')
    reasons = receipt.get('reasons')
    if not isinstance(reasons, list) or not reasons:
        mismatches.append('reasons')
    return mismatches


def write_json(path: Path, payload: dict) -> None:
    text = json.dumps(payload, indent=2, sort_keys=True) + '\n'
    path.write_text(text, encoding='utf-8')


def cmd_write(args) -> int:
    reasons = json.loads(Path(args.reasons_file).read_text(encoding='utf-8'))
    receipt = build_receipt(
        Path(args.diff),
        Path(args.prompt),
        args.model,
        args.workspace_gen,
        args.verdict,
        reasons,
    )
    write_json(Path(args.out), receipt)
    return 0


def cmd_verify(args) -> int:
    receipt_path = Path(args.receipt)
    if not receipt_path.is_file():
        print('reject: missing_receipt')
        return 1
    try:
        receipt = json.loads(receipt_path.read_text(encoding='utf-8'))
    except json.JSONDecodeError:
        print('reject: receipt_json')
        return 1
    if not isinstance(receipt, dict):
        print('reject: receipt_type')
        return 1
    mismatches = find_mismatches(
        receipt,
        Path(args.diff),
        Path(args.prompt),
        args.model,
        args.workspace_gen,
    )
    if mismatches:
        print('reject: ' + ','.join(mismatches))
        return 1
    print('accept')
    return 0


def selftest() -> int:
    with tempfile.TemporaryDirectory() as tmp:
        root = Path(tmp)
        diff = root / 'patch.diff'
        prompt = root / 'prompt.txt'
        diff.write_text('--- a/app.py\n+++ b/app.py\n', encoding='utf-8')
        prompt.write_text('Review the diff. Fail closed on uncertainty.\n', encoding='utf-8')
        model_a = 'lab-model-a'
        model_b = 'lab-model-b'
        gen_1 = 'ws-gen-1'
        gen_2 = 'ws-gen-2'
        receipt = build_receipt(diff, prompt, model_a, gen_1, 'pass', ['tests listed'])
        stale = find_mismatches(receipt, diff, prompt, model_b, gen_2)
        if 'model_id' not in stale or 'workspace_generation' not in stale:
            print('selftest failed: stale receipt was accepted')
            return 1
        fresh = build_receipt(diff, prompt, model_b, gen_2, 'pass', ['tests listed'])
        if find_mismatches(fresh, diff, prompt, model_b, gen_2):
            print('selftest failed: fresh receipt was rejected')
            return 1
        diff.write_text('--- a/app.py\n+++ b/app.py\n+changed\n', encoding='utf-8')
        drifted = find_mismatches(fresh, diff, prompt, model_b, gen_2)
        if 'diff_sha256' not in drifted:
            print('selftest failed: drifted diff was accepted')
            return 1
    print('selftest ok')
    return 0


def main() -> int:
    parser = argparse.ArgumentParser(description='Bind or verify an eval receipt')
    sub = parser.add_subparsers(dest='cmd', required=True)

    write = sub.add_parser('write')
    write.add_argument('--diff', required=True)
    write.add_argument('--prompt', required=True)
    write.add_argument('--model', required=True)
    write.add_argument('--workspace-gen', required=True)
    write.add_argument('--verdict', required=True)
    write.add_argument('--reasons-file', required=True)
    write.add_argument('--out', required=True)
    write.set_defaults(func=cmd_write)

    verify = sub.add_parser('verify')
    verify.add_argument('--diff', required=True)
    verify.add_argument('--prompt', required=True)
    verify.add_argument('--model', required=True)
    verify.add_argument('--workspace-gen', required=True)
    verify.add_argument('--receipt', required=True)
    verify.set_defaults(func=cmd_verify)

    check = sub.add_parser('selftest')
    check.set_defaults(func=lambda args: selftest())

    args = parser.parse_args()
    return args.func(args)


if __name__ == '__main__':
    sys.exit(main())

Commands for the lab

Run the self-test before wiring this gate into CI. A failing self-test means the binder itself is unsafe to trust.

python3 bind_receipt.py selftest

The reasons file must be a JSON array of short strings. An object or an empty array is rejected by design.

Write a receipt only after an independent score file exists. Do not let the agent choose the pass bit by itself.

python3 bind_receipt.py write \
  --diff patch.diff \
  --prompt prompt.txt \
  --model $MODEL_ID \
  --workspace-gen $WORKSPACE_GEN \
  --verdict pass \
  --reasons-file reasons.json \
  --out $ATTEMPT_DIR/receipt.json

Verify against the attempt that is about to merge. A field mismatch must fail the job without a retry.

python3 bind_receipt.py verify \
  --diff patch.diff \
  --prompt prompt.txt \
  --model $MODEL_ID \
  --workspace-gen $WORKSPACE_GEN \
  --receipt $ATTEMPT_DIR/receipt.json

Bootstrap rules

Apply these rules in the job image, not in the prompt. They stop a free server from replaying another attempt.

• Mint WORKSPACE_GEN in bootstrap, before the agent process starts.
• Export MODEL_ID after selection, then freeze that value.
• Store each receipt under a directory named by attempt id.
• Ignore any receipt path left by a previous generation.
• Fail the job if verify exits nonzero, and do not retry.
• Compare the job log model id with the receipt before merge.
• Treat a missing receipt as failure, not as a skipped check.

Decision table

The table compares a naive diff check with the bound gate. Only the last row is a valid merge path. Every other observed state stays closed and blocks the merge.

Observed state	Naive gate	Bound gate
Same diff, new model id	accept	reject
Same diff, new workspace generation	accept	reject
Same identity, verdict fail	reject	reject
Same identity, empty reasons	accept	reject
Diff bytes changed after scoring	reject	reject
Receipt missing	depends on script	reject
Fresh matching receipt with reasons	accept	accept

How to read a reject

The verify command prints mismatched field names and exits 1. Operators should treat that line as the incident symptom. Do not delete the line from the job log after the run.

A model_id mismatch means the scorer changed after the receipt. A workspace_generation mismatch means the server reused state. A diff_sha256 mismatch means the patch moved after scoring.

Store the reject line beside the attempt id in the build record. That record is the audit trail for the failed certification. A later review can then separate failover from a bad patch.

Distinct failure class

This failure is not a stale cache key and not a lockfile retry. The receipt file is valid for a different attempt on purpose. The durable fix is identity binding, not a longer timeout.

Free server job contract

On a free server, bootstrap should wipe agent-writable score paths. Wiping is necessary even when the binder is present. A dirty directory hides other files the agent might trust.

Export the four bound values into the job environment. Pass those values as flags, not as prompt text. Prompt text can be edited by the agent under review.

Where free model access fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode enters this method as operator-supplied free model access. A free server option is the other operator-supplied claim used here.

A reused server can keep yesterday's receipt beside today's diff. Free model access can also switch scorers after a failed attempt. Those two conditions make receipt binding practical rather than optional.

This draft does not name models, quotas, hardware, or duration. It also does not claim that free access is permanent. Confirm current terms before planning capacity around that access.

The binder still matters if that access is unavailable later. Any host that recycles a workspace has the same hole. Put bind_receipt.py in the job image and block merge on mismatch.

Test plan

Use the self-test as the first local check. Then add the fixture cases below in CI.

1. Plant a model A receipt, then verify as model B, and expect exit 1.
2. Keep the model id, change the workspace generation, and expect exit 1.
3. Change one diff byte after write, and expect a hash mismatch.
4. Change one prompt byte after write, and expect a hash mismatch.
5. Write a fail verdict with reasons, and expect verify to reject it.
6. Pass an empty reasons list to write, and expect the command to raise.
7. Remove the receipt file, and expect verify to fail closed.
8. Write and verify with the same four fields, and expect exit 0.

Do not retry verify without new evidence from a new attempt. A reject is recorded data, not a transient network error.

Limitations

Hashes do not measure the quality of a review. A matching receipt can still describe a weak reading of the diff. The gate checks identity and the presence of reasons only.

A hostile runner can forge every bound field in the file. The binder is not an attestation service for model vendors. Pair it with a runner the team already trusts.

Model id is an opaque string supplied by the operator. The script does not list models or select one. The attempt must capture the id before scoring starts.

Workspace generation must come from bootstrap, not from the agent. If the agent mints that value, it can rewrite the attempt history. The gate will then certify a story the agent authored.

Free access can change without notice to the pipeline. Do not assume a specific model or server seat remains available. This article verifies none of those availability details.

The self-test does not call a remote model at all. It checks gate logic with temporary files only. A green self-test is not evidence of a reviewed patch.

Who should skip this approach

Skip this when the team needs vendor-signed proof of a model run. This script cannot provide that kind of proof. A signed attestation requires a different control plane.

Skip this for deploys that lack a separate release policy. The binder is a CI check, not a release approval board. A passing receipt should not be the only production gate.

Skip this when the workspace is always destroyed after the job. Also skip it when the model id cannot change between attempts. A simpler diff hash may be enough in that narrow case.

Skip this for quota planning, cost forecasts, or hardware sizing. This article verifies none of those operational numbers. Use vendor docs for capacity, not this lab gate.

Closing

Bind the receipt before failover can replay an older score. Mint the workspace generation outside the agent process. Fail closed when the model, prompt, or diff does not match.

Comments

[DEV SUPPORTS]

You need to verify your account.

tr.ee/dev-to