Your Agent Patch Gate Is Racing Itself: Isolate Before You Quarantine

If the same commit produces pass on Monday and fail on Tuesday, the patch is usually not the problem. The harness is. Agent patches are verified in parallel far more often than human patches were, and most gates still share ports, temp directories, database names, caches, and on-disk fixtures across every run.

Fix that interference first. Quarantine a test only after isolation is demonstrably clean, or the quarantine list becomes a graveyard for infrastructure bugs you stopped looking for.

The failure signature

Three patches, one gate, three runs of the identical commit set:

run	patch A	patch B	patch C
1	pass	fail	pass
2	fail	fail	pass
3	pass	pass	pass

Nothing in the diff changed between runs. Patch C is stable, A is not, B is consistently red. That pattern is the tell: a stable red plus an unstable neighbor is almost always a shared resource, not a broken assertion. The unstable one is writing to a location the red one also reads, or both are binding the same port and one loses the race.

Quarantining patch A's failing test would hide this. The next patch to break the same resource would inherit the same unexplained flake.

Step 1: Enumerate the interference surface

Before writing any isolation code, list what your runs share. This table is the whole design document.

Resource	Typical leak	Cheap containment
TCP ports	fixture server binds 3000	fixed port lease per digest
Temp dirs	/tmp/<fixed-name> reused	per-run TMPDIR
Database names	test_db truncate/seed races	per-digest database
Build/package cache	partially written cache read by a peer	per-digest cache dir or read-only shared cache
On-disk fixtures	one test rewrites JSON another reads	copy fixtures into the run root
Image tags	:latest resolved at different times	pin by digest
Clock, locale, TZ	env inherited from whoever started the job	set explicitly per run
Global artifact dir	tests write last-run.json	path from GATE_NS

Three or four rows will cover most of your instability. The remaining rows cost little to fix once you have a namespace.

Step 2: Derive the namespace from content, not from the build

A namespace that changes every run defeats caching. A namespace based on the CI build number is worse, because a rerun of the same patch looks like a new world and you cannot reproduce the failure.

Derive it from the patch content:

#!/usr/bin/env bash
set -euo pipefail

# Namespace identity = patch content. Reruns reproduce; different patches diverge.
digest=$(git diff --binary "$BASE_REV".."$HEAD_REV" | sha256sum | cut -c1-12)
ns="gate-${digest}"
root="${GATE_ROOT:-/srv/gate}/${ns}"

mkdir -p "$root"/{tmp,data,logs,fixtures}
export GATE_NS="$ns"
export TMPDIR="$root/tmp"
export GATE_DB="gate_${digest}"
export GATE_BASE_PORT=$(( 20000 + 16#${digest:0:4} % 20000 ))
export TZ=UTC LC_ALL=C

cp -r ./test/fixtures/. "$root/fixtures/"

The hash is derived, not random: it removes nondeterminism without removing reproducibility. It does not remove collisions. Two digests can still hash into the same port window, which is why the next step exists.

These scripts are illustrative and were not executed against a specific CI provider for this write-up. Treat the numbers as policy choices, not measured results.

Step 3: Lease what cannot be namespaced

Some resources are genuinely singular: a hardware device, a licensed service, a remote staging database. Do not pretend those are isolated. Serialize them with an atomic lease and fail closed.

# lease.py — atomic acquire; fail closed, never retry silently
import os, time, pathlib

def acquire(path: str, ttl_s: int):
    p = pathlib.Path(path)
    try:
        os.makedirs(p)                      # atomic: exactly one caller wins
    except FileExistsError:
        age = time.time() - p.stat().st_mtime
        if age > ttl_s:
            raise RuntimeError(f"stale lease {p} age={age:.0f}s — investigate, do not delete")
        raise RuntimeError(f"resource busy: {p}")
    (p / "owner").write_text(os.environ.get("GATE_NS", "unknown"))
    return p

The important line is the one that raises instead of retrying. A silent retry converts a scheduling error into a timing flake, and timing flakes are exactly what ends up on your quarantine list as an innocent-looking test name.

Step 4: Prove isolation with a concurrent soak and permanent canaries

Isolation you have not attacked is a hypothesis. Run the gate against the same patch set concurrently and require verdict stability.

for i in $(seq 1 "${SOAK_RUNS:-5}"); do
  ./run_gate --patch "$RED_CANARY"   --ns "soak-$i-red"   >> soak.jsonl &
  ./run_gate --patch "$GREEN_CANARY" --ns "soak-$i-green" >> soak.jsonl &
  ./run_gate --patch "$SUBJECT"      --ns "soak-$i-subj"  >> soak.jsonl &
  wait
done
python3 check_stability.py soak.jsonl

The canaries are the point. RED_CANARY is a patch with a known violated property and must fail on every run. GREEN_CANARY must pass on every run. check_stability.py groups results by patch id, asserts one unique verdict per patch, and exits non-zero if any canary deviates.

A red canary that ever passes is not a lucky run. It means a peer process satisfied the assertion, the fixture was already in the expected state, or the run never executed. Block merges until that is explained.

Step 5: Attribute before you quarantine

Use the soak output to decide what class of problem you have. The table is deliberately ordered so the infrastructure rows are checked first.

Observation	Class	Action
Verdicts vary across soak, canaries behave	harness bug	fix isolation, quarantine nothing
Red canary passed at least once	isolation broken	block merges, investigate
Verdicts stable in soak, unstable only at high concurrency	real race in the patch	fix the patch
Verdicts stable, flaky on serial fresh-namespace runs	test-level flake	eligible for quarantine with an expiry

Only the last row justifies a freeze, and it justifies one with an owner and an expiry date. Everything above it is a bug with a location.

Where a persistent host and extra model runs fit

This soak multiplies your suite: five runs, three patches, three concurrent processes. That is a scheduling problem, not a compute problem, and it wants a host that keeps state between runs. The lease directory, the shared read-only cache, and the soak log all need to survive, so an ephemeral runner with a fresh filesystem every time is a poor fit for the coordinator.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states that MonkeyCode offers free model access and a free server option. I use the server option as a candidate home for the gate coordinator and the lease directory, and the free model access to keep a second oracle affordable when patch volume rises, since a second opinion on the test diff is what makes the red-canary check meaningful.

Two honest constraints. The isolation work is model-agnostic — MonkeyCode does not fix your ports, your database names, or your fixture writes, and swapping hosts will not remove a single flake. And a filesystem namespace is not a security boundary: if you run untrusted agent code, keep OS-level isolation (containers or separate users) around it. Verify current availability, limits, and terms in MonkeyCode's own documentation rather than taking this paragraph as specification.

Who should not do this

If your gate verifies one patch at a time on a serialized runner, you have no cross-run interference and this entire article is overhead. Teams whose test suite is already fast and fully containerized per test can skip the soak and keep the canary check. And if your quarantine list has been growing for months without a single test being un-quarantined, adding a soak will not fix the culture that produced it — it will only document it.

Start with one run of the table in Step 1. You will likely find three shared resources in under an hour, and the next unattributable verdict will have an address instead of a quarantine entry. If you want the coordinator running on a persistent box to try the soak, MonkeyCode's free server option is one place to host it, and the scripts above do not depend on where it runs.

Comments

[Jo Do]

'Same commit passes Monday, fails Tuesday - the harness is the problem' matches every flaky-gate war story I have. Parallel agent runs multiply exactly the shared-state assumptions human CI got away with: ports, temp dirs, database names, caches. And the quarantine point is the sharp one - quarantining before isolation is proven clean doesn't remove the flake, it removes the evidence. Isolate first; the failures that survive isolation are signal worth a gate.