Cold-Cache Replay: The Only Retry That Should Reach Your Merge Queue

A retry counts as evidence only when it replays from a cold cache and leaves a record a later job can verify. A warm rerun proves that your cache works. It does not prove that your branch builds.

Most advice about flaky CI ends at "just retry the job". That advice quietly hands your merge queue a pass that was never produced under the same conditions as a fresh clone, and it leaves nothing behind for the merge job to inspect. This post fixes the boring half of the problem: make one replay emit a machine-checkable admission record, then make the merge path refuse anything else.

The workflow below is a proposal you can adapt, not a transcript of a run I am handing you as a benchmark. Label your own numbers when you measure them.

The failure mode, stated precisely

A pull request fails on a cold runner because a dependency resolves differently without the warm module cache. Someone clicks retry. The job runs on the same runner, reuses the toolchain, and goes green.

The green check is real, but its subject is different. It tested the tree plus the warm cache, and your merge queue only knows about the tree. When main is built from scratch later, the original failure returns, now attributed to a merge that "passed CI".

So the rule is short: a replay resets the environment and writes down what the environment was. Everything else is a cache hit wearing a check mark.

Write the admission record first

Define the record before you write the workflow. You want fields that the merge job can compare against current reality, not a prose summary. Keep it to one small JSON artifact.

{
  "schema": "admission/v1",
  "tree_sha": "9f1c4e2",
  "base_sha": "3ab07d1",
  "runner_image": "ghcr.io/acme/ci-runner@sha256:91a0...",
  "hook_version": 7,
  "cache": "cold",
  "seed": 4211,
  "toolchain": { "node": "20.11.1", "python": "3.12.3" },
  "result": "pass",
  "started_at": "2026-10-16T09:12:04Z"
}

Four fields do the heavy lifting. cache must equal cold, or the record is void by definition. base_sha catches the case where main moved after your replay. runner_image catches a silent runner rotation. hook_version lets you invalidate every old record in one commit when you change the test harness itself.

If you have already learned not to let a cancelled run count as a pass, the same logic applies here: a cancelled or timed-out replay writes no record, and a missing record is a failure, never a neutral state.

Step 1: purge, then replay

The workflow has one job that runs on a clean runner and deliberately destroys every cache it can reach. Note that the retry path and the first attempt use the same job; there is no separate "fast" path for reruns.

name: replay
on:
  pull_request:
    branches: [main]

permissions:
  contents: read
  checks: write

jobs:
  cold-replay:
    runs-on: ubuntu-24.04
    timeout-minutes: 45
    steps:
      - uses: actions/checkout@v4
        with:
          ref: ${{ github.event.pull_request.head.sha }}
          fetch-depth: 0

      - name: Record the base commit
        run: echo BASE_SHA=${{ github.event.pull_request.base.sha }} >> $GITHUB_ENV

      - name: Record the tree commit
        run: echo TREE_SHA=$(git rev-parse HEAD^{tree}) >> $GITHUB_ENV

      - name: Purge every warm artifact
        run: rm -rf node_modules .venv ~/.npm ~/.cache/pip ~/.cargo/registry

      - name: Read the runner image digest
        run: echo IMAGE_DIGEST=$(jq -r .image /etc/runner-manifest.json) >> $GITHUB_ENV

      - name: Install pinned toolchain
        run: |
          corepack enable
          npm ci --no-audit --no-fund

      - name: Run the suite with a fixed seed
        run: npm test -- --seed=${{ github.run_number }}4211 --reporter=json > test-report.json

      - name: Emit the admission record
        if: always()
        run: python3 scripts/emit_record.py

      - uses: actions/upload-artifact@v4
        with:
          name: admission-record
          path: admission.json
          retention-days: 14

Two details matter more than the rest. The seed is derived from the run number so a retry is a genuinely different ordering, not a repeat of the same lucky interleaving. And the record upload uses if: always(), so a failing replay still produces an artifact that says "result: fail" — invisible failures are how teams end up re-running until green.

Step 2: gate the merge on the record

The merge job never runs the test suite. It downloads the record and checks it against the repository as it exists right now. If any input moved, the record is stale and the merge is refused.

#!/usr/bin/env python3
# scripts/admit.py
import json, subprocess, sys

REQUIRED = {
    'schema', 'tree_sha', 'base_sha', 'runner_image',
    'hook_version', 'cache', 'seed', 'result',
}

def git(*args):
    return subprocess.check_output(['git', *args], text=True).strip()

def admit(record, head_ref, base_sha, image_digest, hook_version):
    missing = REQUIRED - record.keys()
    if missing:
        return False, f'missing fields: {sorted(missing)}'
    if record['schema'] != 'admission/v1':
        return False, 'unknown record schema'
    if record['cache'] != 'cold':
        return False, 'record came from a warm cache'
    if record['result'] != 'pass':
        return False, 'record is not a pass'
    if record['tree_sha'] != git('rev-parse', head_ref + '^{tree}'):
        return False, 'tree changed after the replay'
    if record['base_sha'] != base_sha:
        return False, 'base moved; rebase and replay'
    if record['runner_image'] != image_digest:
        return False, 'runner image rotated; replay on the new image'
    if str(record['hook_version']) != str(hook_version):
        return False, 'hook version changed'
    return True, 'admitted'

if __name__ == '__main__':
    record = json.load(open(sys.argv[1]))
    ok, reason = admit(
        record,
        head_ref=sys.argv[2],
        base_sha=sys.argv[3],
        image_digest=sys.argv[4],
        hook_version=sys.argv[5],
    )
    print(reason)
    sys.exit(0 if ok else 1)

Keep this script in the default branch and call it from a required check that the pull request author cannot edit. A gate that lives in the same diff it approves is decoration.

Step 3: classify the flake before you spend a retry

Not every red run deserves a replay. Split failures into two buckets before you rerun anything, because a product failure replayed three times is three times the noise.

Infrastructure failures look like exit code 143, OOM kills, DNS resolution errors, or a registry 5xx in the log. These are the only ones eligible for a replay, and each replay voids the previous record. Product failures — an assertion, a snapshot diff, a type error — block the branch and go to a human.

Budget the replays per pull request, not per job. Two replays is a common ceiling; if a third is needed, the branch has a real ordering dependency and you should fix it instead of paying for it.

Step 4: the green-to-merge decision table

Situation	Action
Record passes; tree, base, image and hook version all match	Auto-merge
Any infrastructure replay happened	Require a fresh cold replay before merge
Failure is a product assertion	Block; human triage
Base branch moved after the replay	Rebase, then cold replay
Runner image rotated since the replay	Cold replay on the new image
No record (cancelled, timeout, OOM)	Treat as failing, never neutral
Hook version changed in this diff	Reject the diff; harness changes land separately

That last row is the one people skip, and it is the one that keeps the gate honest. If a pull request can change the harness and the code it tests at the same time, your record describes a moving target.

Where a free model and a free server fit

If your patches come from an agent, the same record shape still applies — the agent is just another producer of a tree hash. In that setup the operator states that MonkeyCode offers free model access plus a free server option, which is enough to try this loop before committing paid CI minutes to it.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Use it for the part you can control: generate the patch with the free model access, run the cold replay on the free server option, and keep the admission record identical to the one your paid runners emit. Do not hard-code quota or hardware numbers into your workflow — free-tier terms change, so read the current page before you depend on a specific allowance, and never treat a free runner as a compliance or isolation boundary.

Limitations, and who should skip this

Cold replay costs time. A suite that takes twelve minutes warm can take twenty-five or more cold, and monorepos with large native builds will feel it first. If your build already requires a licensed artifact you cannot download on a clean machine, this approach will block you until you solve caching deliberately — which is a different project.

Skip it if your team has no required check that pull request authors cannot edit, because the record will not be enforceable. Skip it if every failure is already a product failure and your flake rate is near zero. The overhead only pays for itself when a warm cache is a plausible explanation for a green run.

A five-line checklist for the next pull request

1. Does the failing job emit an admission record even when it fails?
2. Does the record include the tree, the base, the runner image, and the hook version?
3. Is the merge gate reading that record from a path the author cannot modify?
4. Is the replay budget per pull request, not per job?
5. When the base moves, does the old record become invalid automatically?

If you can answer yes to all five, a green check means something you can defend after the outage.