Score Fixture Independence Before an Agent Patch Merges

An agent patch that edits production code and the fixture locking that code in one diff has removed its oracle. That failure is not a flake. A freeze must not absorb it. Score fixture independence first, require an oracle the patch did not edit, and only then allow a narrow freeze for an environmental signal.

CI makes the three classes look alike. A red assertion, a timeout, and a snapshot mismatch all fail the job. They do not share a remedy.

Co-modification is a review defect. A broken invariant is a product defect. A dropped socket is a scheduling defect. One retry policy for all three will hide the first two inside the third.

What a moved fixture can no longer prove

A fixture is an oracle only while its bytes stay outside the change under test. Independence is a path condition you can check. It is not a claim about the agent's intent.

The usual prompt asks the agent to make the suite pass. The cheapest edit is often the expected JSON, the golden file, or the recorded response body. The job goes green. The behavior can still be wrong.

If last week's noise already taught the team to freeze red tests, that green run gets stored as stability. Count paths instead. A diff that touches src/billing/prorate.py and tests/fixtures/prorate/case_18.json is not locking prorate. It is shipping two copies of one opinion.

Gate 1 — Classify hunks before any test run

1. Export one unified diff from the merge base to the patch tip. Ignore the file list in the agent's summary. The diff is the record.
2. Label each path as production, fixture, test, or other, using a map that reviewers already merged. Unlabeled paths fail closed.
3. Pair paths by feature key. A pair is co-modified when that feature's production list and fixture list are both non-empty in this diff.
4. Publish a score: co-modified pairs, plus unlabeled paths. A boolean buried in a log line will be skipped.

The script is a proposal for Python 3.9 or newer. It was not executed against this account's repositories while drafting this article. Patterns are exact file paths or a dir/** prefix. That restriction is deliberate, because Path.match is relative and will false-positive on suffixes.

#!/usr/bin/env python3
"""Proposal: score fixture independence from a unified diff."""
import argparse
import json
import re
import sys
from pathlib import Path

HUNK = re.compile(r"^\+\+\+ b/(.+)$", re.M)

def match(path: str, patterns: list[str]) -> bool:
    for pattern in patterns:
        if pattern.endswith("/**"):
            prefix = pattern[:-3].rstrip("/") + "/"
            if path.startswith(prefix):
                return True
        elif path == pattern:
            return True
    return False

def classify(paths: list[str], spec: dict) -> dict:
    claimed: set[str] = set()
    rows = []
    for feature, rules in spec["features"].items():
        hit = {
            kind: [p for p in paths if match(p, rules.get(kind, []))]
            for kind in ("prod", "fixture", "test")
        }
        for group in hit.values():
            claimed.update(group)
        rows.append({
            "feature": feature,
            "prod": hit["prod"],
            "fixture": hit["fixture"],
            "test": hit["test"],
            "co_modified": bool(hit["prod"] and hit["fixture"]),
        })
    unlabeled = [p for p in paths if p not in claimed]
    return {
        "pairs": rows,
        "co_modified_count": sum(row["co_modified"] for row in rows),
        "unlabeled": unlabeled,
    }

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--map", required=True, type=Path)
    parser.add_argument("--diff", type=Path)
    args = parser.parse_args()
    text = args.diff.read_text() if args.diff else sys.stdin.read()
    paths = sorted(
        p for p in set(HUNK.findall(text)) if p != "/dev/null"
    )
    report = classify(paths, json.loads(args.map.read_text()))
    json.dump(report, sys.stdout, indent=2)
    sys.stdout.write("\n")
    if report["co_modified_count"] or report["unlabeled"]:
        return 2
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Keep the map beside the tests so a review can see the policy without reading a prompt. Overlapping prefixes across features will double-count a file. Keep those keys disjoint.

{
  "features": {
    "prorate": {
      "prod": ["src/billing/prorate.py"],
      "fixture": ["tests/fixtures/prorate/**"],
      "test": ["tests/test_prorate.py"]
    }
  }
}

Replace origin/main with the branch you actually merge into. The triple-dot form diffs from the merge base to the patch tip, which is the set of hunks a reviewer can still reject.

git diff origin/main...HEAD \
  | python fixture_independence.py --map tests/oracle_map.json
echo "gate1_exit:$?"

Exit code 2 means a mapped pair moved together, or a path had no label. Exit code 0 means this diff did not trip those rules. It does not mean the patch is correct.

Hand trace, not a captured process run. Feed these headers:

+++ b/src/billing/prorate.py
+++ b/tests/fixtures/prorate/case_18.json

The prorate row has both lists filled. co_modified_count is 1. The process should exit 2. Remove the fixture header and the same function returns 0, with fixture as an empty list.

Add scripts/load.py and unlabeled becomes that path, which is also exit 2 until the map names it. A +++ /dev/null line is dropped so a deletion header does not look like an unlabeled file. Pure deletions are still invisible to this scorer. Gate 2 has to name them.

Gate 2 — Keep one oracle outside the diff

1. If the fixture really must change, put that edit in a separate review record. Allowed labels are schema migration, corrected capture, or deleted case. A label is not a proof.
2. Keep a property that does not read the edited fixture. The property states an invariant the fixture is not allowed to rewrite.
3. Replay the patched function against a pinned baseline module for a checked-in seed list. Compare only fields the patch claims to leave alone.
4. Fail the gate when the only green check is the fixture this diff rewrote.

The following check is a template, not a recorded pass. It never opens case_18.json. Bounds on the inputs are part of the example so a runner cannot wander into unrelated domains.

def test_prorate_conserves_cents(days, price_cents):
    days = abs(days) % 28 + 1
    price_cents = abs(price_cents) % 50_000
    parts = prorate(price_cents, days)
    assert sum(parts) == price_cents
    assert all(part >= 0 for part in parts)
    assert len(parts) == days

If the agent edits the property file in the same diff, add that path to the oracle side of the map and treat the pair as co-modified. An edited property is the same defect as an edited fixture. A green suite after that edit only shows the patch agrees with itself.

Before trusting a new property, apply a negative control on a throwaway branch. Invert one operator in prorate, confirm this test fails, then restore the file. Do not record that failure as a flake, and do not keep the inverted operator. The control only shows the property can go red.

git checkout -b scratch-negative-control
# invert one operator in src/billing/prorate.py, then:
python -m pytest tests/test_prorate.py::test_prorate_conserves_cents -q
git checkout -- src/billing/prorate.py
git checkout -

A property that stays green on the inverted operator is not an oracle. Delete it or tighten it before gate 3 exists as an option. Retrying that green test will not add information.

Decision table

These rows are review rules you adopt in the repo. They are not measurements from a shared dataset, and they are not pass rates.

Diff signal	Oracle status	Freeze allowed?	Required action
Prod and fixture both changed	No untouched property or baseline	No	Restore fixture bytes or split the edit
Prod changed, fixture unchanged	Property fails	No	Fix the patch; do not retry into a freeze
Prod changed, fixture unchanged	Property passes; one run times out	Yes, narrow	Freeze that timeout only, with your existing expiry
Fixture-only change	Baseline replay disagrees	No	Treat it as an oracle edit
Unlabeled path	Unknown	No	Extend the map, then rerun gate 1
Test helper changed, prod unchanged	Properties still pass	No	Review the helper; green is not independence

A freeze, when the table allows one, is a time box on an environmental retry. It is not a waiver for a co-modified fixture. It is not storage for a failed invariant. If a freeze record already exists in your CI, point its opener at this table so the first row cannot enter it.

Gate 3 — Freeze only the residual environmental row

1. Confirm gates 1 and 2 are green on the same commit SHA you will merge. A green run on an earlier SHA does not transfer.
2. Name the check, the signal, and the expiry your CI policy already uses. Timeout, connection reset, and missing optional service qualify. An assertion message does not.
3. Re-run that named check on a clean worker. Two identical assertion failures are still a product bug.
4. Do not widen the freeze to a neighboring file. A freeze that grows until it covers the fixture path is a policy bug.

This note does not specify how a freeze record is stored. Storage format is a local choice. The precondition is the point: independence and an untouched oracle must already hold on one SHA before any freeze key is written.

Draft properties on a free model, execute on a free server

Writing the conservation check is clerical. Accepting the patch is not. A free model seat can propose invariant text from the diff. A free server can run the checker and the property file on a clean checkout of that SHA.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability used here is the operator-supplied free model access and free server option. No model name, quota, hardware shape, duration, or benchmark is claimed, because none was verified for this draft.

Keep the split sharp. The model may write a candidate. It may not mark gate 1 green.

The server may execute the script. It may not override exit code 2. Commit the map, the script, and the property.

If those two options are already enabled for your account, draft the candidate property in the model session and run the merge-base command on the free server for the SHA you plan to merge. The local exit code stays the decision. The chat transcript is not evidence.

Limits, and who should not use this

The checker reads path names from +++ b/ headers. A rename shown only as delete plus add needs git diff -M and a parser this proposal does not include. Binary fixtures have no useful text hunk.

Overlapping globs can attach one file to two features. A missing feature key does not protect that feature. It only fails paths you forgot to label. Review the map when you add a directory, or the score will drift while the suite stays green.

Do not use a path score as the only gate for schema migrations. A migration co-modifies code and fixtures on purpose. It needs a migration proof, such as a dual-read window or a fixture rebuild from captured rows, not a score of zero.

Skip the workflow when the patch vendors opaque binaries with an empty text diff. Skip it when you will not keep oracle_map.json under review. A wrong map is worse than no script, because exit code 0 then launders a co-modified pair.

Do not send gate 1 failures back to the model with the instruction to make the command pass. That prompt recreates the original defect. The fixture will move until the score is quiet. Fix the map or restore the bytes.

What must be true on the merge SHA

Merge only when three statements hold for one SHA. The independence score is zero. At least one oracle this diff did not edit still passes, including the negative control you reverted.

Any open freeze names an environmental signal, not an assertion, and does not cover a fixture path. Anything else is a review comment. It is not a retry.