Postmortem: The Patch Gate Tested a Tree the Patch Never Reached

A green test run proves one thing. The tests passed on some tree. It does not prove the tree contained the patch.

This postmortem covers a gate that verified a workspace path. The patch had already left that path. The gate returned exit 0 on a clean checkout.

Scope note: this is a constructed walkthrough, not a record of one specific production outage. The shapes are common; the timeline is illustrative. The script and the test plan are meant to be run by the reader, and no results are claimed here.

The incident class in one paragraph

An automated worker received a patch, applied it, ran tests, and reported success. The patch never touched the tree the tests executed against. The worker had reused a warm workspace directory left over from an earlier run, and the gate resolved that directory by path at test time. Path is mutable state. The digest is not.

Blast radius

• One patch was certified and merged without an executed test against its own content.
• The receipt recorded a test command but no tree identity.
• The failure was invisible from the dashboard, because latency and exit codes looked normal.

Timeline (reconstructed)

• T+0 min. The worker starts on a reused host. A previous run's workspace directory still exists at /var/gate/work/run-0007.
• T+4 min. The model call returns a patch. The worker writes it to /tmp/patch.diff and applies it into run-0007.
• T+6 min. The supervisor restarts the worker after an idle disconnect. The job ID is preserved; the workspace path is preserved.
• T+7 min. The retried job re-clones the base commit into run-0007, overwriting the patched files. The patch file itself is untouched.
• T+12 min. The gate runs the test suite against run-0007. All tests pass, because the tree is pristine.
• T+13 min. The gate emits a receipt with exit 0 and the job ID. The merge queue accepts it.
• T+38 min. A reviewer asks which commit the tests ran against. The receipt has no tree digest, so nobody can answer.

Contributing factors

1. Directory identity stood in for content identity. The gate trusted a path, not a hash.
2. Retry logic was scoped to the job, not the workspace. A retried job inherited a directory it did not create.
3. The patch was stored outside the verified tree. It survived a re-clone that destroyed the applied state.
4. The receipt lacked the two facts needed for audit. Base commit and resulting tree digest were both missing.
5. Pass/fail was decided by test exit code alone. An empty or already-applied patch also exits 0.

The durable fix: digest-bound verification

One rule drives the fix. Never verify a path. Verify a tree digest, and make the gate refuse to continue when that digest does not change.

#!/usr/bin/env bash
# gate.sh BASE_SHA PATCH_FILE
# Bind every verification run to an immutable tree digest.
set -euo pipefail

BASE_SHA="${1:?usage: gate.sh BASE_SHA PATCH_FILE}"
PATCH_FILE="${2:?usage: gate.sh BASE_SHA PATCH_FILE}"
REPO_URL="${REPO_URL:?set REPO_URL}"
WORK_ROOT="${WORK_ROOT:-/var/gate/work}"

PATCH_SHA="$(sha256sum "$PATCH_FILE" | cut -c1-12)"
RUN_ID="$(date -u +%Y%m%dT%H%M%SZ)-${BASE_SHA:0:12}-$PATCH_SHA"
WORKDIR="$WORK_ROOT/$RUN_ID"

# 1. Fresh workspace per run. Fail if the directory already exists.
mkdir -p "$WORK_ROOT"
mkdir "$WORKDIR"
git clone --quiet --no-checkout "$REPO_URL" "$WORKDIR"
git -C "$WORKDIR" checkout --quiet --detach "$BASE_SHA"

# 2. Capture the tree before the patch.
BEFORE="$(git -C "$WORKDIR" rev-parse 'HEAD^{tree}')"

# 3. Apply. A patch that does not apply to the pinned base is a hard stop.
git -C "$WORKDIR" apply --index --whitespace=nowarn "$PATCH_FILE" \
  || { echo 'FAIL: patch does not apply to pinned base' >&2; exit 20; }

# 4. Refuse a no-op. Empty patch, replayed patch, or wrong base lands here.
AFTER="$(git -C "$WORKDIR" write-tree)"
[ "$BEFORE" != "$AFTER" ] \
  || { echo 'FAIL: patch produced no tree change' >&2; exit 21; }

The second half seals the tree during test execution.

# 5. Run the suite inside the sealed workspace.
( cd "$WORKDIR" && ./run-tests.sh )

# 6. Detect drift caused by the tests themselves.
git -C "$WORKDIR" diff --quiet -- . \
  || { echo 'FAIL: tracked files changed during tests' >&2; exit 22; }
[ -z "$(git -C "$WORKDIR" ls-files --others --exclude-standard)" ] \
  || { echo 'FAIL: untracked files left behind' >&2; exit 23; }

# 7. Emit a receipt that names the tree, not the path.
printf '{"base":"%s","patch_sha256_12":"%s","tree_before":"%s","tree_after":"%s","exit":0}\n' \
  "$BASE_SHA" "$PATCH_SHA" "$BEFORE" "$AFTER"

Why each line exists

• mkdir "$WORKDIR" fails on collision. Warm reuse becomes impossible by construction.
• HEAD^{tree} pins content, not a branch name or a directory.
• The no-op check catches double application, the most common retry artifact.
• The drift check catches tests that mutate the tree they are judging.
• The receipt makes the audit question answerable in one lookup.

Decision table

Check	Mandatory	Reason
Base commit pinned by SHA	Yes	Patch context is otherwise undefined
Fresh workspace per run	Yes	Removes retry inheritance
Tree digest before and after	Yes	Detects no-op and replay
Patch sha256 in the receipt	Yes	Ties approval to bytes
Working-tree drift after tests	Yes	Catches self-mutating suites
Wall-clock latency as a gate input	No	Speed is not correctness
Model identity in the receipt	Optional	Useful for correlation, not proof

Reproduction plan you can run

This is a plan, not reported output. Run it on any machine that has git and bash.

1. Create a repository with one source file and one test that asserts a specific string.
2. Write a patch that changes that string incorrectly. Expect the test command to exit non-zero.
3. Run gate.sh <base-sha> bad.diff. Expect a non-zero exit and no receipt line.
4. Write an empty patch. Run again. Expect exit 21, the no-op branch.
5. Write a patch that applies cleanly, then re-run with the same job ID and a pre-created workspace. Expect exit 20 or an early mkdir failure, never a green receipt.
6. Add a test that writes an untracked file. Expect exit 23. Decide whether that strictness fits the project.

The interesting result in step 5 is the absence of a receipt. Missing output is a better failure signal than a wrong exit code.

Where MonkeyCode fits, and where it does not

The gate above is deliberately boring. It makes no assumption about who produced the patch. That property is what makes a hosted model endpoint usable at all: the patch generator can be swapped without touching the verification logic.

MonkeyCode is relevant in two places in this workflow. The operator offers free model access, which covers patch generation during development and small backfills. The operator also offers a free server option, which can host the worker that runs gate.sh.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The operator describes the project as open source. Confirm the license and the current terms in the repository before you depend on them. Free allowances and host capacity are commercial terms, and they change. Do not size a pipeline against a number you read in a blog post, including this one. Read the project's own page on the day you decide.

Limitations that matter for this workflow

• A free host is shared and may restart without notice. The digest-bound gate is designed for that, but your retry budget must tolerate it.
• Ephemeral storage means workspaces vanish. Persist receipts to object storage or a database, not to the worker's disk.
• Model output is non-deterministic. Two runs may produce different patches for the same prompt. Deduplicate by patch_sha256, not by prompt.
• The gate adds a clone per run. On large repositories that cost dominates. Use a shared object cache if the host allows it.

Who should not use this setup

• Teams that need a contractual uptime or support commitment on the verification host.
• Codebases that cannot send source to a third-party model endpoint under any terms.
• Builds that require GPU, large memory, or multi-hour wall-clock time.
• Pipelines where a human reviews every patch anyway and the gate is decorative.

What changed permanently

Two lines moved out of the worker and into the gate: the tree digest and the patch digest. Everything else in the incident was ordinary. A restart, a retry, a reused directory. Ordinary conditions produce certification failures when identity is inferred from a path.

If you try the workflow, generate the patch through MonkeyCode's free model access and run gate.sh on the free server. Let the digest decide, and let the host be replaceable.