Partition the Agent Diff Before You Quarantine a Failure

Quarantine is a record of instability, not a receipt for an agent patch. If the same diff edits production code and tests, no flake freeze token should exist until the test edit is split away, the fixture digest matches baseline, property checks are replayed on the production tree alone, and a planted mutant still dies under the assertions you intend to keep.

That sequence is the gate. A green rerun after a combined diff only shows that the new tree agrees with itself.

The failure mode in the combined diff

Agent patches frequently land as one commit. A function changes, and so does the test that names it. The next red run is easy to file as a flake because the command line did not change.

The command line is the wrong identifier. The oracle may have moved inside the test file, or the fixture bytes may have moved under a stable path.

Split the candidate into two artifacts before any rerun budget is spent. Production paths feed the behavioral checks. Test paths feed an oracle-strength check. A freeze token can be discussed only when the candidate under review contains no test paths, and only after the other checks below agree.

A two-run disagreement does not replace this split. Disagreement tells you the results differed. It does not tell you which side of the diff purchased the green result.

1. Partition paths before you rerun

Start from the file list. Do not launch the suite in this step.

git diff --name-only origin/main...HEAD > /tmp/patch-files.txt
grep -E '(^|/)(test_|tests/|_test\.py|spec/)' /tmp/patch-files.txt > /tmp/test-paths.txt || true
grep -v -E '(^|/)(test_|tests/|_test\.py|spec/)' /tmp/patch-files.txt > /tmp/prod-paths.txt || true
wc -l /tmp/prod-paths.txt /tmp/test-paths.txt

Non-zero counts on both files mean split_required. While that mark is set, the freeze ledger should reject an open request. The production-only tree is what property checks must see. How you build that worktree depends on the forge; the invariant does not.

Sample the path regex against recent commits before you enforce it. Tests that sit beside production modules, and helpers that embed production logic under tests/, will be mis-filed by a generic pattern.

2. Hash fixture bytes, not fixture names

A lock that stores a path misses a silent rewrite. Hash the bytes the loader will read.

find fixtures -type f -print0 | sort -z | xargs -0 sha256sum | sha256sum > /tmp/fixture-digest.txt
cat /tmp/fixture-digest.txt

Compare that digest with the last green build on the same branch, and with the digest from any rerun you are tempted to call intermittent.

Digest vs baseline	Test paths in candidate	Classification	Freeze token
Equal	None	Identity held; not yet a flake	Closed until later steps pass
Equal	One or more	Oracle may have moved	Closed until the mutant dies
Different	Either	Fixture epoch changed	Closed; drift, not a flake
Equal across disagreeing runs	None	Nondeterminism candidate	Eligible only if steps 3 and 4 pass

A digest change ends the flake hypothesis for that run. Review the fixture epoch, or revert it, before you spend reruns. Reruns on a moving fixture measure two experiments and store one label.

3. Replay properties on the production tree only

Property checks in this workflow are predicates over inputs you choose, with the seed written to disk. They are not a second copy of the example the agent just rewrote.

# Proposed runner. Not executed against a live suite in this draft.
import json
import random
from pathlib import Path

def replay_property(seed: int, trials: int, predicate) -> dict:
    rng = random.Random(seed)
    failures = []
    for i in range(trials):
        value = rng.randrange(0, 10_000)
        if not predicate(value):
            failures.append({"trial": i, "value": value})
            break
    return {
        "seed": seed,
        "trials": trials,
        "failed": bool(failures),
        "first_failure": failures[0] if failures else None,
    }

def load_seed(path: Path) -> int:
    return json.loads(path.read_text())["seed"]

Persist seed, trials, and failed next to the run id. A failure on the production-only tree is a behavioral reject. It does not enter the freeze ledger, even if a later full-tree command looks intermittent.

Keep the predicate in a module the current patch did not edit. If the only predicate lives in the edited test file, stop and add a reviewer-owned predicate first. An agent-written predicate can restate the defect and stay green.

Count assertion-line churn as a separate signal, not as a score. Removed or widened asserts are review input. They do not, by themselves, prove the oracle was weakened.

git diff -U0 origin/main...HEAD -- $(cat /tmp/test-paths.txt) \
  | grep -E '^[+-].*assert' \
  | sed -n '1,40p'

The mutant probe in the next step is what turns that suspicion into an exit code. Treat the diff listing as a queue for review, not as a numeric quality rating.

4. Plant one mutant and require a kill

One mutant is a narrow probe. It is not a suite-wide mutation score, and this draft does not report one. The probe asks a single question: do the tests you plan to keep still fail when production behavior is wrong in a way you planted?

# Proposed evidence gate. Not executed in this draft.
from enum import Enum
from dataclasses import dataclass

class Verdict(str, Enum):
    SPLIT_REQUIRED = "split_required"
    FIXTURE_DRIFT = "fixture_drift"
    PROPERTY_FAIL = "property_fail"
    ORACLE_WEAKENED = "oracle_weakened"
    FREEZE_ELIGIBLE = "freeze_eligible"
    HOLD = "hold"

@dataclass(frozen=True)
class Evidence:
    prod_files: int
    test_files: int
    digest_stable: bool
    property_failed_on_prod_only: bool
    mutant_killed: bool
    nondeterministic_same_digest: bool

def classify(ev: Evidence) -> Verdict:
    if ev.prod_files and ev.test_files:
        return Verdict.SPLIT_REQUIRED
    if not ev.digest_stable:
        return Verdict.FIXTURE_DRIFT
    if ev.property_failed_on_prod_only:
        return Verdict.PROPERTY_FAIL
    if ev.test_files and not ev.mutant_killed:
        return Verdict.ORACLE_WEAKENED
    if (
        ev.nondeterministic_same_digest
        and ev.test_files == 0
        and ev.mutant_killed
    ):
        return Verdict.FREEZE_ELIGIBLE
    return Verdict.HOLD

Plant the mutant on a disposable branch. Do not plant it on a shared default branch.

git switch -c audit/mutant-probe
# Apply one local, reversible behavior change in a production file.
python -m pytest tests/test_order_total.py -q --tb=line
echo "mutant_exit=$?"
git switch -
git branch -D audit/mutant-probe

mutant_exit=0 means the tests accepted a known-bad change. The verdict is oracle_weakened. Do not open a freeze, and do not merge that test edit on the strength of a green agent run.

A non-zero exit means this probe still bites. That result is necessary and not sufficient. A different fault could still slip past assertions this probe never touched.

A test-only candidate never becomes freeze_eligible in this function. If the edited tests kill the probe, the label is hold. If they do not, the label is oracle_weakened. Quarantine is not a side door for an assertion edit.

5. Issue a token only for the remaining shape

All of the following must hold before a freeze token is writable. Missing any one line keeps the ledger closed.

1. The candidate under review has test_files == 0.
2. The fixture digest matches baseline and matches across the disagreeing runs.
3. Property replay on the production-only tree did not fail.
4. The mutant probe against the tests you are keeping was killed.
5. The same command disagreed with itself on that digest.

Store the token as data. Bind voiding to digest change, not only to a calendar date your team already uses for quarantines. If the digest moves, the token is void even when the date is still open.

The object below is an example shape, not a recorded run. Leave model prose out of the file the merge job reads.

{
  "verdict": "freeze_eligible",
  "fixture_digest": "<sha256 from step 2>",
  "property_seed": 1729,
  "property_failed": false,
  "mutant_killed": true,
  "test_files_in_candidate": 0,
  "expires_on_digest_change": true
}

The merge job should read verdict and the digest, then stop. A comment in the pull request is not a substitute for those two fields.

What may be drafted elsewhere

Writing a reviewer-owned predicate is clerical work. Producing the digest and the mutant exit code needs a clean runner, so a laptop virtualenv does not become part of the evidence.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access can draft a candidate predicate for a human to edit, and the free server option can run the classification job away from an interactive workstation. Those two availability facts are not quotas, hardware specs, or a promise that the draft is correct. The gate remains classify(evidence).

If you already have that free model access, keep the prompt to a predicate, a fixed seed, and one failing example. Paste the draft into a file this patch does not modify. Use the free server option as the runner for the digest and the mutant exit, then copy the evidence JSON back and apply the table locally.

Limitations

The procedure does not estimate how many faults the suite would catch. One planted site can miss a weakened assertion on another branch.

Digest equality ignores nondeterminism that never lands in fixture bytes. Wall clocks, unordered rendered text, and ambient network calls can still flip a result while the hash stays put. Pin or forbid those sources before you treat nondeterministic_same_digest as meaningful. Otherwise the token launders an environment bug.

Property replay is only as honest as the predicate. Reviewer ownership reduces the chance that the same agent restates the bug. It does not prove the predicate is independent.

Path partitioning is a heuristic. Enforce it only after you have checked it against a sample of your own history. No pass rate, duration, or quota is claimed in this note; the artifact is a decision procedure, and it has not been executed against a live suite here.

Who should skip it

Skip the procedure when you cannot materialize a production-only tree. Without the split, the later checks run on the combined diff and answer a different question.

Skip it when a behavior edit cannot live on a disposable branch. Shared staging and incident-sensitive sandboxes are the wrong place to plant a mutant.

Skip it when fixtures are built only in memory and never written to bytes. Hashing an empty directory will look stable while inputs move.

Skip it when the goal is for an assistant to close the freeze or approve the merge. The classifier returns a label. A person still owns the token.

Four counts before merge

Read four values from the evidence file: production file count, test file count, digest match, mutant exit code. If the test count is non-zero, quarantine is not available for this candidate. Split the diff, keep the probe hostile, and only then decide whether the remaining red run is actually unstable.