A Frozen Test That Turns Green Is a Merge Blocker

Green CI is the wrong merge signal for an agent patch. A frozen flake that starts passing is a contract break, not a win. Score the diff with three AND-gated columns: human-owned properties, hashed fixtures, and a freeze file the agent is forbidden to graduate.

That rule is the whole strategy. Properties say what must stay true. Fixtures say which bytes are the world. The freeze file says which tests are not evidence. If any column is incomplete, the merge score is zero. A green check from a test the freeze still owns does not raise it.

Why a passing flake is the failure

Agent patches optimize for the tests they can see. A flaky test is a moving target. When the patch lands and that test turns green, three causes are more likely than a real fix: the assertion was weakened, the timing window was widened, or the input that used to fail was deleted.

None of those are proof. Treat the transition frozen -> passing as a blocker until a human removes the name from the freeze file after an agent-free rerun. Graduation is an edit to the freeze, not a side effect of the diff.

This is a proposal for a local gate, not a claim about any vendor CI. Label it as such. Run it on the checkout you already have.

The three-column ledger

Keep one human-owned file. Suggested path: qa/merge_ledger.toml. The agent may read it. The agent may not write it.

# qa/merge_ledger.toml
# Human-owned. Agent writes to this file are a score of 0.

[properties]
paths = ["tests/properties/"]
required = true

[fixtures]
lockfile = "qa/fixtures.lock.json"
allow_unhashed_new = false

[freeze]
file = "qa/flake_freeze.txt"
# A frozen name that passes under the patch fails the score.
passing_frozen_is_error = true

Column 1 is properties: invariants that do not name a single golden string. Column 2 is fixtures: content-addressed inputs. Column 3 is freeze: test node ids that cannot contribute a pass. The merge score is 1 only when all three hold. Otherwise it is 0. Do not average them.

Artifact: a scorer you can run

The script below is executable as written. It does not call a model. It does not need a network. Point it at a pytest run that emitted a JUnit file, then at the ledger.

#!/usr/bin/env python3
"""merge_score.py — AND-gate for agent patches. Proposal; run locally."""
from __future__ import annotations

import hashlib
import json
import sys
import tomllib
import xml.etree.ElementTree as ET
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def load_junit(path: Path) -> dict[str, str]:
    tree = ET.parse(path)
    out: dict[str, str] = {}
    for case in tree.iter("testcase"):
        node = f"{case.attrib.get('classname', '')}::{case.attrib.get('name', '')}"
        if case.find("failure") is not None or case.find("error") is not None:
            out[node] = "fail"
        elif case.find("skipped") is not None:
            out[node] = "skip"
        else:
            out[node] = "pass"
    return out


def check_fixtures(lock_path: Path) -> list[str]:
    lock = json.loads(lock_path.read_text())
    errors: list[str] = []
    seen = set()
    for rel, expected in lock["files"].items():
        seen.add(rel)
        path = ROOT / rel
        if not path.is_file():
            errors.append(f"missing fixture: {rel}")
            continue
        digest = sha256_file(path)
        if digest != expected:
            errors.append(f"hash drift {rel}: {digest} != {expected}")
    return errors


def check_freeze(freeze_path: Path, results: dict[str, str], strict: bool) -> list[str]:
    names = [ln.strip() for ln in freeze_path.read_text().splitlines() if ln.strip() and not ln.startswith("#")]
    errors: list[str] = []
    for name in names:
        status = results.get(name)
        if status == "pass" and strict:
            errors.append(f"frozen test passed (blocker): {name}")
        elif status == "fail":
            # Still frozen, still failing: allowed. It must not raise the score.
            continue
        elif status is None:
            errors.append(f"frozen test missing from JUnit: {name}")
    return errors


def check_properties(prop_dir: Path, results: dict[str, str]) -> list[str]:
    errors: list[str] = []
    if not prop_dir.is_dir():
        return [f"property dir missing: {prop_dir}"]
    ran = 0
    for node, status in results.items():
        if "tests/properties" not in node.replace("\\", "/"):
            continue
        ran += 1
        if status != "pass":
            errors.append(f"property failed: {node} ({status})")
    if ran == 0:
        errors.append("no property tests in JUnit output")
    return errors


def main(argv: list[str]) -> int:
    if len(argv) != 2:
        print("usage: merge_score.py junit.xml", file=sys.stderr)
        return 2
    ledger = tomllib.loads((ROOT / "qa/merge_ledger.toml").read_text())
    results = load_junit(Path(argv[1]))
    errors: list[str] = []
    errors += check_properties(ROOT / ledger["properties"]["paths"][0], results)
    errors += check_fixtures(ROOT / ledger["fixtures"]["lockfile"])
    errors += check_freeze(
        ROOT / ledger["freeze"]["file"],
        results,
        ledger["freeze"]["passing_frozen_is_error"],
    )
    score = 0 if errors else 1
    print(f"merge_score={score}")
    for err in errors:
        print(f"- {err}")
    return 0 if score == 1 else 1


if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

Pair it with a lockfile that is also human-owned:

{
  "files": {
    "tests/fixtures/invoice_v3.json": "replace-with-real-sha256",
    "tests/fixtures/empty_batch.ndjson": "replace-with-real-sha256"
  }
}

And a freeze file that lists node ids, one per line:

# qa/flake_freeze.txt — human-owned. Agent cannot delete lines.
tests.runtime.test_retry::test_backoff_under_load
tests.http.test_client::test_header_order_stable

Rebuild hashes without guessing:

python - <<'PY'
from pathlib import Path
import hashlib, json
root = Path(".")
files = [
    "tests/fixtures/invoice_v3.json",
    "tests/fixtures/empty_batch.ndjson",
]
lock = {"files": {}}
for rel in files:
    data = (root / rel).read_bytes()
    lock["files"][rel] = hashlib.sha256(data).hexdigest()
Path("qa/fixtures.lock.json").write_text(json.dumps(lock, indent=2) + "\n")
print("wrote qa/fixtures.lock.json")
PY
pytest tests/properties tests -q --junitxml=qa/junit.xml
python qa/merge_score.py qa/junit.xml

If merge_score=0, do not merge. Read the printed errors. They are the only review notes that matter for this gate.

Property tests that cannot be tautologies

A property test names an invariant, not a captured string from the patch. Write them in tests/properties/ so the scorer can see the path. Example for a serializer the agent is allowed to touch:

# tests/properties/test_codec_roundtrip.py
import json
from hypothesis import given, strategies as st

from billing.codec import dumps, loads

@given(st.dictionaries(st.text(min_size=1, max_size=16), st.integers()))
def test_loads_dumps_roundtrip(payload: dict) -> None:
    assert loads(dumps(payload)) == payload


def test_dumps_rejects_nan() -> None:
    try:
        dumps({"n": float("nan")})
    except ValueError:
        return
    raise AssertionError("NaN must not serialize")

Hypothesis here is optional. The invariant is not. If the agent rewrites test_loads_dumps_roundtrip into assert dumps(x) is not None, the path still matches and the scorer still runs it. Add a second human check: properties cannot shrink to a single trivial assert. A cheap lint:

# Fail if a property file has no assert / no raises and under 8 non-empty lines.
python - <<'PY'
from pathlib import Path
fail = False
for p in Path("tests/properties").glob("test_*.py"):
    lines = [ln for ln in p.read_text().splitlines() if ln.strip() and not ln.strip().startswith("#")]
    body = "\n".join(lines)
    if len(lines) < 8 or ("assert" not in body and "raise" not in body):
        print(f"thin property file: {p}")
        fail = True
raise SystemExit(1 if fail else 0)
PY

Thin files fail the gate before pytest runs. That is deliberate. Classification happens before color.

Numbered workflow

1. Freeze the world. Hash fixtures. List known flakes by node id. Commit qa/merge_ledger.toml, qa/fixtures.lock.json, and qa/flake_freeze.txt on a branch the agent cannot force-push.
2. Write or confirm properties under tests/properties/. Each file must encode an invariant you would keep if the implementation were replaced.
3. Produce the candidate patch from whatever author you use, including a checkout on a free server session if that is how you already generate diffs. Do not let that author edit the three ledger files.
4. Run pytest with JUnit output. Run merge_score.py. Require merge_score=1.
5. If a frozen test passed, stop. Do not celebrate. Diff the test body. If the test changed, revert the test. If the test did not change, rerun it twenty times on main without the patch. Only a human may then delete the freeze line.
6. If a fixture hash drifted, stop. Rehash only after a human reads the byte diff and updates the lockfile in a separate commit.

Step 3 is where a hosted coding environment is relevant. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those two facts are the only product claims used here. Do not treat a green run on that server as column evidence. Copy the ledger into the checkout and score the same way you would score a patch from a stranger.

Decision table

Observation after the patch	Score	Human action
All properties pass, fixture hashes match, frozen tests still fail or skip	1	Review the production diff; merge is eligible
Any property fails	0	Reject the patch; do not expand fixtures to hide it
Fixture hash drift, even if tests pass	0	Byte-diff the fixture; rehash in a human commit or revert
Frozen test now passes, test file unchanged	0	Agent-free rerun on main; freeze stays until that rerun is boring
Frozen test now passes, test file changed	0	Revert the test rewrite; freeze stays
New test added by the agent under tests/properties/	0 until reviewed	Move it out or rewrite it as a human-owned invariant
Freeze file or lockfile touched by the agent	0	Restore the three files from main

The table is the policy. Do not add a fourth column for coverage percentage. Coverage can rise while invariants die.

What this does not measure

The scorer does not prove the patch is correct. It proves the patch did not buy a green check with frozen tests, drifted fixtures, or missing properties. Those are different statements. Keep them separate.

It also does not replace load tests, typed API contracts, or a human reading the production diff. A roundtrip property will not catch a silent change in log redaction. A fixture hash will not catch a new network call. A freeze file will not catch a test the agent invented and immediately satisfied.

Limitations, and who should not use this

Do not use this gate on a repo with no human-owned tests. The score will either be stuck at 0 or you will be tempted to let the agent fill tests/properties/. Both outcomes are worse than no gate.

Do not use it as a performance benchmark. No timings are recorded. No model names, quotas, or hardware claims belong in the ledger. If you need latency numbers, collect them in a different job that cannot green a merge by itself.

Skip the freeze rule on suites that are already deterministic and fully hashed. The passing-frozen blocker is for suites that still flake. Applying it to a clean suite only adds a file nobody will maintain.

Teams that merge without JUnit artifacts cannot run the script as written. Produce XML first, or rewrite load_junit for your runner. Until then, the policy still holds: a flake that starts passing is not evidence.

If you already generate candidate diffs with MonkeyCode's free model access on the free server option, add the ledger to that checkout and require merge_score=1 before anyone looks at the check mark. The score is the review. The check mark is not.