Postmortem: A Full Disk Made the Patch Gate Fail Open, and Exit Code 0 Certified It

A patch gate is only as trustworthy as the evidence it reads. In this lab incident, the evidence writer hit a full disk and degraded to best-effort logging. The gate kept reading exit code 0, so it kept approving.

This is a capacity postmortem, not a replay postmortem. The bug is not that old data came back. The bug is that new data silently failed to arrive, and nothing downstream could tell the difference.

The incident was reproduced in a lab harness, not a production outage at a named company. Every command below is runnable.

What the system did

The setup is a common one for AI-generated patches. A runner pulls a candidate diff, asks a reviewer model for a verdict, writes a transcript, and exits.

The CI gate reads the exit code. Green means the transcript exists and the verdict is pass.

The runner was hosted on a free server option. Its disk was the smallest resource in the whole pipeline, and nothing measured it.

Timeline (lab clock, relative)

1. T+00:00 — Nightly batch starts. 140 candidate patches queue against the reviewer.
2. T+00:41 — Transcript directory crosses 400 MB. No rotation rule exists.
3. T+01:12 — First ENOSPC on a transcript write. The writer catches it and logs a warning.
4. T+01:12 — The same code path returns normally. Process exits 0.
5. T+01:13 — Gate sees exit 0, marks the patch approved, moves on.
6. T+02:05 — Batch ends. Exit status for the whole job is 0.
7. T+09:30 — A human notices the transcript directory has fewer files than approved patches.

The time between the first silent failure and detection was about eight hours. The gate reported success for all of it.

Contributing factors

• Best-effort evidence. Transcript writing sat in a try/except that logged and continued. Evidence was treated as a nice-to-have.
• Exit code as the contract. The gate never verified that a file existed, was non-empty, or matched a checksum.
• Shared filesystem. Earned evidence and scratch temp files lived on the same mount. Scratch growth consumed the evidence budget.
• No capacity signal. Health checks returned 200 from a static page. A full disk was invisible.
• Unbounded growth. Nothing capped transcript size per patch. One long reviewer response could double a file.

Reproduce it in about two minutes

A small loop is enough. Point the writer at a mount with little free space and watch the exit code.

mkdir -p /tmp/lab/evidence /tmp/lab/transcripts
# fill the mount until writes fail
fallocate -l 100% /tmp/lab/filler || true
bash run_batch.sh --patches fixtures/patches.jsonl; echo "exit=$?"

In the lab run, exit=0 came back while evidence/ held fewer files than patches submitted. The gate approved every one of them.

The durable fix, part 1: fail closed

Evidence writes must be mandatory and verified. No warning path, no continue.

import hashlib, json, os, pathlib

def write_evidence(run_dir: str, patch_id: str, transcript: bytes) -> pathlib.Path:
    ev = pathlib.Path(run_dir) / "evidence"
    ev.mkdir(parents=True, exist_ok=True)
    body = ev / f"{patch_id}.jsonl"
    with open(body, "wb") as f:
        f.write(transcript)
        f.flush()
        os.fsync(f.fileno())
    size = body.stat().st_size
    if size != len(transcript):
        raise RuntimeError(f"short write: {size} != {len(transcript)}")
    digest = hashlib.sha256(body.read_bytes()).hexdigest()
    manifest = ev / f"{patch_id}.manifest.json"
    manifest.write_text(json.dumps({"patch_id": patch_id, "bytes": size, "sha256": digest}))
    return manifest

fsync matters here. Without it, a crash can leave a manifest that describes bytes the disk never accepted.

The durable fix, part 2: gate on the manifest

Exit code stops being the contract. The manifest becomes the contract.

def gate(run_dir: str, patch_id: str) -> tuple[str, str]:
    ev = pathlib.Path(run_dir) / "evidence"
    manifest = ev / f"{patch_id}.manifest.json"
    if not manifest.exists():
        return "block", "missing manifest"
    meta = json.loads(manifest.read_text())
    body = ev / f"{patch_id}.jsonl"
    if meta["bytes"] == 0 or not body.exists():
        return "block", "empty or missing evidence"
    if hashlib.sha256(body.read_bytes()).hexdigest() != meta["sha256"]:
        return "block", "checksum mismatch"
    return "pass", "evidence verified"

This gate fails closed on three separate conditions. Missing evidence, empty evidence, and tampered evidence all block.

The durable fix, part 3: make capacity a probe

A 200 from a static page proves nothing. The readiness probe should write bytes.

#!/usr/bin/env bash
set -euo pipefail
RUN_DIR="${RUN_DIR:?run dir required}"
MIN_FREE_MB="${MIN_FREE_MB:-2048}"
LOG_BUDGET_MB="${LOG_BUDGET_MB:-512}"

free_mb=$(df -Pm "$RUN_DIR" | awk 'NR==2 {print $4}')
[ "$free_mb" -ge "$MIN_FREE_MB" ] || { echo "preflight: free=${free_mb}MB below floor"; exit 20; }

log_mb=$(du -sm "$RUN_DIR/transcripts" 2>/dev/null | awk '{print $1}')
log_mb=${log_mb:-0}
[ "$log_mb" -le "$LOG_BUDGET_MB" ] || { echo "preflight: transcripts=${log_mb}MB over budget"; exit 21; }

probe="$RUN_DIR/.probe.$$"
head -c 1048576 /dev/zero > "$probe" || { echo "preflight: probe write failed"; exit 22; }
rm -f "$probe"
echo "preflight: ok free=${free_mb}MB transcripts=${log_mb}MB"

Run it before the batch, and expose the same three numbers on the health endpoint. Capacity becomes a metric instead of a surprise.

Isolating scratch from evidence

Put transcripts on their own mount or volume. Give scratch a size ceiling. Delete scratch by run ID, never by wildcard.

A cleanup step that runs rm -rf $RUN_DIR/* on a shared mount will eventually remove something it did not create. Use a janitor with an allowlist of run IDs older than a fixed age.

Where the free tier actually fits

This incident happened on a small runner, but the constraint was not the free tier. The constraint was an unmeasured resource.

The lab pipeline calls MonkeyCode's free model access for the reviewer step and runs the batch loop on the free server option. Both are operator-stated availability claims. Confirm current terms and limits before depending on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The pattern that survived the postmortem is simple. Cheap, disposable compute runs the reviewer and the transcript writer. Durable evidence lives somewhere with verified capacity. The gate trusts checksums, not exit codes.

Decision table: what belongs on a small runner

Workload	Fits a free-tier runner	Guard required
Prompt regression evals, capped transcripts	Yes	Byte cap per patch, rotation, preflight
Nightly patch gating with manifest evidence	Yes	Manifest checksum gate, disk floor
Unbounded fuzz or long transcript dumps	No	Growth is not predictable
Building large vector indexes	No	Index build needs sustained disk
Long-lived stateful services	No	State survives restarts
Anything the gate reads as truth	Yes, read-only	Evidence is stored elsewhere

Limitations and who should not use this

• Teams with per-run latency SLOs should not run gating on shared, bursty hardware.
• Regulated pipelines needing write-once evidence retention need object storage, not a local disk.
• Pipelines writing more than a few hundred MB per run should measure first, then decide.
• The probe above tests capacity, not I/O latency. Add a timed write if latency drives your timeouts.

Checklist before the next batch

1. Delete the except: pass around transcript writes.
2. Add the sha256 manifest and gate on it.
3. Run the preflight script with a real disk floor.
4. Split scratch from evidence onto separate mounts.
5. Alert when free space drops below the floor, not when writes start failing.

The cheapest postmortem is the one that turns one silent failure into three loud ones. Start with the disk floor.