Demote Tests in the Patch Write-Set Before You Merge

An agent patch is not mergeable because its suite is green. It is mergeable when a maintainer-owned oracle, outside the patch write-set, still encodes the faults you care about and still passes on a second machine. Agent-authored examples can inform review. Frozen flakes can document noise. Neither should be the gate.

Green is a count of assertions that executed. It is not a measure of independence. Agent patches routinely edit production code and tests in the same diff. When that happens, the suite and the change share a single author. Shared authorship is the failure mode. The rest of this article is a workflow that treats that overlap as data, not as a vibe.

The merge rule

Keep four classes. Apply one rule.

1. Property oracles live in a path the patch cannot write. They gate merge.
2. Digest-locked fixtures pin inputs the oracle consumes. They gate merge when the digest matches.
3. Write-set tests are any test file the agent diff touches. They are advisory only.
4. Variance freezes are tests that disagree across k remote runs. They are excluded from the gate until the ledger expires or the variance drops.

If a check is in class 3 or 4, a pass is not evidence. A fail is still a review signal. The merge job should ignore both for the exit code.

Why a second machine is part of the oracle

Local green is an environment result. Clock resolution, CPU count, file-system latency, and leftover caches all leak into agent patches that touch time, IO, or concurrency. A freeze recorded on a laptop is a freeze of that laptop. Replay later on CI and the class changes.

The cheapest correction is not a longer local loop. It is k identical runs on a machine the author did not provision. Section 4 treats that machine as a required input to the freeze ledger, not as an optional nicety.

Decision table

Class	How it is detected	Merge gate?	Review signal?
Property oracle	Path prefix oracle/ and not in git diff --name-only	Yes	Yes
Fixture lock	SHA-256 of files under fixtures/ matches fixtures.lock	Yes, if digest equal	Yes, if digest drift
Write-set test	Path under tests/ appears in the patch write-set	No	Yes
Variance freeze	Pass/fail or duration disagrees across k remote runs	No, until ledger expires	Yes

The table is the artifact. Tools below only emit it as JSON so CI can fail closed.

1. Inventory the write-set

Label this as a runnable helper, not as a measured production study. It reads the patch names and demotes every test the agent touched.

# classify_write_set.py
from __future__ import annotations

import json
import subprocess
from pathlib import Path

ORACLE_PREFIX = "oracle/"
TEST_PREFIXES = ("tests/", "test_")
FIXTURE_PREFIX = "fixtures/"


def names(ref: str = "HEAD") -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{ref}^", ref],
        text=True,
    )
    return [line.strip() for line in out.splitlines() if line.strip()]


def classify(paths: list[str]) -> dict:
    write_set = set(paths)
    oracle_touched = sorted(p for p in paths if p.startswith(ORACLE_PREFIX))
    fixture_touched = sorted(p for p in paths if p.startswith(FIXTURE_PREFIX))
    demoted = sorted(
        p for p in paths
        if p.startswith(TEST_PREFIXES) or Path(p).name.startswith("test_")
    )
    return {
        "oracle_write_set": oracle_touched,
        "fixture_write_set": fixture_touched,
        "demoted_tests": demoted,
        "merge_blocked_if_oracle_edited": bool(oracle_touched),
        "write_set": sorted(write_set),
    }


if __name__ == "__main__":
    report = classify(names())
    Path("inventory.write_set.json").write_text(json.dumps(report, indent=2))
    print(json.dumps(report, indent=2))
    if report["merge_blocked_if_oracle_edited"]:
        raise SystemExit(2)

Exit 2 is deliberate. If the patch edits oracle/, the gate has no remaining independent checks. Do not fall through to pytest and call that a merge.

CODEOWNERS belongs next to the script. One line is enough.

/oracle/    @maintainers
/fixtures.lock    @maintainers

A CODEOWNERS file is not a test. It is how class 1 stays class 1 when the agent can open a pull request.

2. Put properties in the path the patch cannot edit

Example-based asserts in tests/ are cheap for agents to satisfy. They encode a single vector. A property encodes a relation. Keep the relation in oracle/ so a write-set demotion cannot delete it.

The snippet below is a stdlib stand-in. It is not a substitute for a shrinking library. It is enough to show the boundary: the oracle imports production code and does not import anything from tests/.

# oracle/test_invariants.py
from __future__ import annotations

import unittest
from decimal import Decimal, ROUND_HALF_EVEN

from billing.totals import apply_discount, invert_discount


class DiscountInvariants(unittest.TestCase):
    CASES = [
        (Decimal("0.00"), Decimal("0.10")),
        (Decimal("19.99"), Decimal("0.00")),
        (Decimal("19.99"), Decimal("0.15")),
        (Decimal("100.00"), Decimal("1.00")),
    ]

    def test_discount_never_negative(self):
        for amount, rate in self.CASES:
            out = apply_discount(amount, rate)
            self.assertGreaterEqual(out, Decimal("0.00"), (amount, rate, out))

    def test_discount_never_exceeds_amount(self):
        for amount, rate in self.CASES:
            out = apply_discount(amount, rate)
            self.assertLessEqual(out, amount, (amount, rate, out))

    def test_round_trip_within_one_cent(self):
        quantum = Decimal("0.01")
        for amount, rate in self.CASES:
            if rate >= 1:
                continue
            discounted = apply_discount(amount, rate)
            restored = invert_discount(discounted, rate)
            delta = abs(restored - amount).quantize(quantum, rounding=ROUND_HALF_EVEN)
            self.assertLessEqual(delta, quantum, (amount, rate, restored))

Three properties, not thirty examples. The round-trip is an inverse check on the domain, not a coverage trophy. If the agent patch authors a new example in tests/test_discount.py that asserts apply_discount(19.99, 0.15) == 16.99, that file is in the write-set. Classify it as advisory. Do not let it rescue a failing inverse.

3. Lock fixtures by digest, not by filename

Filename stability is not content stability. Agents rename, reformat, or insert a trailing newline and keep the same path. Hash the bytes. Store the hashes in a file the write-set classifier already treats as protected.

# lock_fixtures.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ROOT = Path("fixtures")
LOCK = Path("fixtures.lock")


def digest(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def build() -> dict[str, str]:
    rows = {}
    for path in sorted(ROOT.rglob("*")):
        if path.is_file():
            rows[path.as_posix()] = digest(path)
    return rows


def verify() -> None:
    expected = json.loads(LOCK.read_text())
    actual = build()
    if expected != actual:
        missing = sorted(set(expected) - set(actual))
        extra = sorted(set(actual) - set(expected))
        changed = sorted(
            k for k in expected.keys() & actual.keys() if expected[k] != actual[k]
        )
        raise SystemExit(
            json.dumps({"missing": missing, "extra": extra, "changed": changed}, indent=2)
        )


if __name__ == "__main__":
    import sys
    if sys.argv[1:] == ["--write"]:
        LOCK.write_text(json.dumps(build(), indent=2) + "\n")
    else:
        verify()

CI runs python lock_fixtures.py with no flag. Humans run --write after an intentional fixture change. An agent patch that edits fixtures/ without a matching lock update fails before pytest starts. That is the point. Fixture drift is a spec change, not a test failure.

4. Freeze on cross-run variance, not on a single red

A single failure on a developer laptop is not a flake class. A disagreement across k runs on one remote image is. Record pass/fail and duration. Freeze only when the boolean result is not constant, or when duration spread exceeds a bound you picked in advance.

# variance_ledger.py
from __future__ import annotations

import json
import statistics
import subprocess
import sys
import time
from pathlib import Path

K = 5
DURATION_CV_MAX = 0.35


def run_once() -> dict:
    t0 = time.perf_counter()
    proc = subprocess.run(
        [sys.executable, "-m", "unittest", "discover", "-s", "oracle", "-q"],
        capture_output=True,
        text=True,
    )
    return {
        "ok": proc.returncode == 0,
        "seconds": time.perf_counter() - t0,
        "stderr_tail": proc.stderr[-500:],
    }


def classify(rows: list[dict]) -> dict:
    oks = [r["ok"] for r in rows]
    secs = [r["seconds"] for r in rows]
    mean = statistics.fmean(secs)
    cv = (statistics.pstdev(secs) / mean) if mean else 0.0
    boolean_stable = len(set(oks)) == 1
    duration_stable = cv <= DURATION_CV_MAX
    return {
        "k": len(rows),
        "pass_count": sum(oks),
        "boolean_stable": boolean_stable,
        "duration_cv": round(cv, 4),
        "duration_stable": duration_stable,
        "freeze": not (boolean_stable and duration_stable),
        "gate_ok": boolean_stable and oks[0] and duration_stable,
        "runs": rows,
    }


if __name__ == "__main__":
    report = classify([run_once() for _ in range(K)])
    Path("inventory.variance.json").write_text(json.dumps(report, indent=2))
    print(json.dumps({k: report[k] for k in report if k != "runs"}, indent=2))
    raise SystemExit(0 if report["gate_ok"] else 1)

k = 5 and DURATION_CV_MAX = 0.35 are starting constants, not measurements. Tune them per suite. The important part is the class change: a test that fails once locally does not enter a freeze file. A test that disagrees with itself on one remote image does.

Store freezes with an expiry, not a comment. A JSON ledger is enough.

{
  "oracle.test_invariants.DiscountInvariants.test_round_trip_within_one_cent": {
    "reason": "boolean_unstable",
    "k": 5,
    "pass_count": 3,
    "expires": "2026-09-25"
  }
}

Expired rows return to the gate. Infinite freezes are how suites lose oracles.

5. One job, four inputs, one exit

Numbered because the order matters. Skipping a step silently reclassifies evidence as proof.

1. Compute the git write-set. Fail if oracle/ is in it.
2. Verify fixtures.lock. Fail on digest drift.
3. Run unittest discover -s oracle as the only merge oracle.
4. Run the same oracle k times on a second machine. Write inventory.variance.json.
5. Publish tests/ results, including write-set tests, as advisory annotations.
6. Merge only when steps 1–4 agree. Do not AND the advisory suite into the exit code.

A minimal Compose-style job is a sketch, not a vendor config.

# ci/oracle-gate.yaml  (sketch)
steps:
  - run: python classify_write_set.py
  - run: python lock_fixtures.py
  - run: python variance_ledger.py
artifacts:
  - inventory.write_set.json
  - inventory.variance.json
  - fixtures.lock

The two JSON files are the review surface. A green pytest blob is not.

Where a free model and a free server actually sit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The workflow does not need a product name to work. It needs two cheap capacities that many teams skip because they look optional: a model that can propose inventory labels, and a server that is not the author's laptop.

MonkeyCode's free model access is a fit for the labeling step, not the gate. Feed it git diff --name-only and ask for a proposed class per path: oracle, fixture, write-set test, or unknown. A human still has to confirm. Models over-promote examples into properties. That error is why the write-set script is mechanical and the model is advisory.

MonkeyCode's free server option is a fit for step 4. The variance ledger is invalid if k runs share the laptop that authored the patch. Pin the image, pin the seed for any remaining entropy, and keep the freeze file off that laptop. If you already have a remote runner, use it. The method does not depend on a particular host. It depends on a host the patch author did not just use.

Limitations

Write-set demotion is syntactic. An agent can hide behavior changes behind a helper imported by oracle/ without listing a test file in the diff. CODEOWNERS on oracle/ reduces that. It does not eliminate it. Review still has to read production diffs.

Properties on a handful of cases are not a full hypothesis search. The inverse check above will miss rate values the four tuples never touch. If the domain is wide, add a shrinking runner later. Do not confuse a compact oracle with a complete one.

Duration coefficients of variation mix signal with machine noise. A noisy shared runner will freeze too much. A perfectly idle runner will freeze too little. The ledger is a classifier, not a physics constant. Recalibrate k when the runner class changes.

Fixture digests fight reformatting. They also fight legitimate golden updates. Teams that regenerate snapshots on every agent pass will live in --write and the lock becomes theater. If your fixtures are generated, stop generating them in the same job that merges production code.

Who should not use this

Do not install this gate on a repo where humans and agents share a single tests/ tree and there is no one to own oracle/. Demotion without an oracle path fails open: every test is advisory, every patch is mergeable.

Do not use variance freezes as a way to silence a race you have not classified. A freeze without an expiry is a skip. Skips accumulate. The inventory will look complete while the oracle shrinks.

Do not point a free remote runner at secrets, production data, or customer fixtures. Digest-locked fixtures should be synthetic. If the only reproducing input is production traffic, this workflow is the wrong one.

Closing constraint

Count classes, not dots. If the patch write-set includes the only checks that failed before the patch, you do not have a passing suite. You have a patch that learned the exam. Keep the exam in a directory the student cannot edit, hash the materials, and score it on a machine the student did not bring.