Quarantine Cannot Vote on an Agent Patch

Quarantine is not a merge signal. An agent patch that relocates a failure into skip, xfail, or a flaky folder can keep CI green while the oracle shrinks. Role reclassification is the defect. The production diff is secondary until test roles stay locked.

This procedure splits the runner into three lanes. Lane A executes role-locked oracles and properties and is fail-closed. Lane B verifies content-addressed fixtures and is fail-closed. Lane C records flake fingerprints and cannot vote. Merge is A_ok && B_ok. Lane C is a ledger, not a boolean.

What a conventional summary misses

Pytest and JUnit report an exit code. They do not report that a property collapsed into one example. They do not report that a fixture byte changed. They do not report that yesterday’s assertion is today’s “flaky” skip. Agents optimize for green. Those three edits are cheap. The gate has to price them.

Count node IDs, not comfort. A deleted ID is a failed gate. A role change in the same commit as generated code is a failed gate. A quarantine failure is a log line. It does not flip the merge bit.

Lane contract

Lane A — oracles and properties. Every collected node ID must exist in tests.roles.yaml with role oracle or property. oracle tests are concrete expected values. property tests are quantified claims over a generator. An agent may add IDs only as candidate. Candidates never replace a locked ID. Candidates do not vote.

Lane B — fixtures. Files listed under fixtures: are hashed. A byte change without a lockfile update fails the lane. Hash updates belong in a human commit, not in the patch that consumes the fixture.

Lane C — quarantine. Node IDs with role quarantine still run. Failures write a fingerprint: node ID, exception type, sanitized message. The fingerprint file is append-mostly. Clearing or rewriting fingerprints in the agent commit fails integrity. Lane C never returns a blocking status for a test failure. Integrity of the ledger does block.

Role lockfile

# tests.roles.yaml
version: 1
rules:
  deny_role_change_in_agent_commit: true
  deny_delete_locked_id: true
  candidates_do_not_vote: true
  quarantine_does_not_vote: true

tests:
  tests/test_ledger.py::test_balance_non_negative:
    role: property
    generator: amounts_and_fees
  tests/test_ledger.py::test_posted_row_matches_fixture:
    role: oracle
    fixture: tests/fixtures/posted_row.json
  tests/test_ledger.py::test_clock_skew_display:
    role: quarantine
    fingerprint_key: clock_skew_display

fixtures:
  tests/fixtures/posted_row.json:
    sha256: 9f86d081884c7d659a2feaa0c55ad015a3bf4f1b2b0b822cd15d6c15b0f00a08

New IDs omitted from this file are not “unreviewed oracles.” They are undeclared. Undeclared IDs fail Lane A on collect. That is intentional. Silent tests are how roles drift.

Fingerprint, not a calendar

Expiry dates on flaky freezes invite a different cheat: wait. Fingerprints do not expire. They bind a node ID to a failure shape. Renaming the test does not clear the record. Changing the exception type without leaving quarantine is a mismatch, not a fix.

Sanitize before hashing. Strip line numbers, hex addresses, timestamps, and absolute paths. Keep the exception class and a stable token from the message. Collision risk exists if sanitization is too aggressive. Under-sanitizing is worse: every rerun looks like a new flake and the ledger becomes noise.

# flake_fingerprint.py
from __future__ import annotations

import hashlib
import re
from dataclasses import dataclass

_LINE = re.compile(r":\d+")
_HEX = re.compile(r"0x[0-9a-fA-F]+")
_TS = re.compile(r"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}")
_PATH = re.compile(r"(?:[A-Za-z]:)?(?:/|\\)[^\s:]+")


@dataclass(frozen=True)
class Fingerprint:
    nodeid: str
    exc_type: str
    digest: str


def sanitize(message: str) -> str:
    text = _PATH.sub("<path>", message)
    text = _TS.sub("<ts>", text)
    text = _HEX.sub("<hex>", text)
    text = _LINE.sub(":<n>", text)
    return " ".join(text.split())


def fingerprint(nodeid: str, exc_type: str, message: str) -> Fingerprint:
    body = f"{nodeid}\n{exc_type}\n{sanitize(message)}"
    digest = hashlib.sha256(body.encode("utf-8")).hexdigest()[:32]
    return Fingerprint(nodeid=nodeid, exc_type=exc_type, digest=digest)

Numbered gate

1. Collect tests with pytest --collect-only -q. Parse node IDs. Do not run code yet.
2. Diff IDs against tests.roles.yaml. Fail on deletes, undeclared IDs, and role edits unless the commit trailer contains Role-Lock: human.
3. Hash every fixture path in the lockfile. Fail Lane B on mismatch. Do not auto-update the digest.
4. Run Lane A with -k limited to oracle and property. Zero retries. Wall-clock and RNG come from fixtures, not from the host.
5. Run Lane C separately. Append fingerprints. Fail only if an existing fingerprint_key disappears or its digest is rewritten by the agent commit.
6. Ignore candidate failures for the merge bit. Print them. They are proposals.
7. Promote a candidate or graduate quarantine in a follow-up commit that touches only the lockfile. Generated product code does not ride along.

# three_lane_gate.py — reference procedure, not a measured production run
from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

import yaml

LOCK = Path("tests.roles.yaml")
LEDGER = Path("flake_fingerprints.json")


def sha256(path: Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()


def collect_ids() -> set[str]:
    out = subprocess.check_output(
        [sys.executable, "-m", "pytest", "--collect-only", "-q"],
        text=True,
    )
    ids = set()
    for line in out.splitlines():
        if "::" in line and not line.startswith("="):
            ids.add(line.strip())
    return ids


def load_lock() -> dict:
    return yaml.safe_load(LOCK.read_text())


def lane_b(lock: dict) -> list[str]:
    errors = []
    for rel, meta in (lock.get("fixtures") or {}).items():
        digest = sha256(Path(rel))
        expected = meta["sha256"] if isinstance(meta, dict) else meta
        if digest != expected:
            errors.append(f"fixture-mismatch {rel}")
    return errors


def lane_a_roles(lock: dict, collected: set[str]) -> list[str]:
    errors = []
    declared = lock["tests"]
    locked_ids = set(declared)
    if not locked_ids <= collected:
        errors.append(f"deleted-ids {sorted(locked_ids - collected)}")
    undeclared = collected - locked_ids
    if undeclared:
        errors.append(f"undeclared-ids {sorted(undeclared)}")
    voters = [k for k, v in declared.items() if v["role"] in {"oracle", "property"}]
    if not voters:
        errors.append("no-voting-tests")
    return errors


def main() -> int:
    lock = load_lock()
    collected = collect_ids()
    errors = lane_a_roles(lock, collected) + lane_b(lock)
    if errors:
        print("GATE_FAIL")
        for e in errors:
            print(e)
        return 2
    voters = [
        k for k, v in lock["tests"].items() if v["role"] in {"oracle", "property"}
    ]
    expr = " or ".join(voters)
    a = subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "-k", expr, "--maxfail=1"],
    )
    # Lane C is invoked, recorded, and ignored for the return bit.
    subprocess.run(
        [sys.executable, "-m", "pytest", "-q", "-k", "quarantine", "--tb=line"],
        check=False,
    )
    if LEDGER.exists():
        json.loads(LEDGER.read_text())  # integrity: file must parse
    print("LANE_C_NO_VOTE")
    return 0 if a.returncode == 0 else 1


if __name__ == "__main__":
    raise SystemExit(main())

Run it as a pre-merge job, not as a developer honor system.

python three_lane_gate.py
echo $?   # 0 merge, 1 lane A failed, 2 lock/fixture integrity failed

Property shape the agent is not allowed to shrink

A property that becomes assert f(1) == 1 is an example. Keep the generator outside the test body so a patch cannot replace the domain with a literal. The lockfile names the generator. If the test body no longer calls it, fail on a static check: the generator identifier must appear in the AST.

# tests/test_ledger.py
from net_decimal import Decimal


def amounts_and_fees():
    # Labeled example generator — expand in the real suite.
    for amount in (Decimal("0.01"), Decimal("1.00"), Decimal("99.99")):
        for fee in (Decimal("0.00"), Decimal("0.30")):
            yield amount, fee


def test_balance_non_negative():
    for amount, fee in amounts_and_fees():
        posted = amount - fee
        assert posted >= 0 or fee > amount
        if fee <= amount:
            assert posted + fee == amount

The inverse check (posted + fee == amount) is the contract. An agent that deletes the loop and asserts one pair has reclassified a property as an example. AST presence of amounts_and_fees is the cheap detector. It is not a proof of coverage. It blocks the common collapse.

Decision table

Patch move	Lane A	Lane B	Lane C	Merge
Delete a locked node ID	fail	—	—	no
Mark a locked oracle skip / xfail	fail	—	—	no
Replace generator loop with one literal	fail (AST)	—	—	no
Change fixture bytes, keep old digest	—	fail	—	no
Human updates digest in a lock-only commit	—	pass	—	depends on A
New candidate fails	no vote	—	—	yes if A,B pass
Quarantine test fails, fingerprint stable	no vote	—	append	yes if A,B pass
Agent rewrites or drops a fingerprint	—	—	integrity fail	no
Rename quarantined test, old key missing	fail undeclared / integrity	—	—	no

Read the table left to right. If Lane A or B fails, stop. Do not average in Lane C.

Where a scratch model and scratch server fit

Candidate properties have to come from somewhere. A useful split is: the model proposes generators and inverse claims from the diff; a human assigns property or leaves them as candidate; the gate runs off the laptop so fixture hashes are not computed against dirty working trees.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access can draft those candidate properties from a patch. The free server option can execute three_lane_gate.py so Lane B digests and Lane C fingerprints are produced in a clean tree. Neither step authorizes a role change. The lockfile still requires a human trailer. Do not treat model output as an oracle. Do not treat a green Lane C as coverage.

If you already review agent diffs, run the gate where the lockfile and the patch are in the same commit. Fingerprints computed on a dirty laptop are not evidence.

Limitations

The AST check is syntactic. A generator can be called and ignored. Pair it with a minimum iteration counter if the suite can afford it. Fingerprint sanitization can hide real message changes or, if too loose, treat every rerun as novel. Quarantine that never graduates becomes a junk drawer; the gate will not notice because quarantine cannot vote. That is a process failure, not a CI failure.

The collector depends on pytest node ID stability. Parametrized IDs that embed timestamps will look like deletes. Freeze parameter IDs. Time, RNG, and network sockets are not “flakes.” They are missing fixtures. Put clocks and seeds in Lane B. If a test needs the public network, it does not belong in Lane A.

This reference gate does not measure mutation score, flake rate, or model quality. Those numbers are absent because they were not collected here. Copying the YAML without owning promotions will freeze a bad taxonomy.

Who should not use this

Skip the three-lane split if tests are regenerated and discarded every commit. There is no ID to lock. Skip it if no person will accept Role-Lock: human commits; the lockfile will rot and the gate will block all work. Skip it for UI suites whose drivers cannot pin time and entropy. Skip it when the oracle itself is secret and cannot be hashed into a fixture file that lives in git.

Teams that only run end-to-end checks on shared staging will punish Lane B. Shared mutable fixtures are not content-addressed. Fix the fixtures first.

Close

Green is cheap. Roles are not. Keep oracles and properties in a lane that can fail the merge. Hash the bytes they read. Record flakes without giving them a vote. Promote with a lockfile commit, not with a skip marker inside the agent patch.