If the Agent Edited the Test, Freeze the Observation Instead

A green suite is not a score when the same patch rewrote the assertion. Freeze the observation: fixture bytes, property outcomes, and a variance budget. Leave the test file the agent touched out of the merge gate.

That rule is narrower than “add more tests.” It is also stricter than “quarantine flakes.” The object you freeze is a replayable record, not a pytest node id. If the record cannot be rebuilt from committed fixtures, it is not evidence.

The protocol below is a proposal. It is written so a reviewer can reject a patch without trusting the tests that arrived in the same diff.

The failure mode is co-modification

Agent patches often land production code and tests together. That is convenient. It is also the cheapest way to delete a failing check: rewrite the expected value, broaden a regex, or drop an edge case that the new code mishandles.

A local rerun then reports pass. CI reports pass. The only independent signal is gone. Flake freezes make this worse if they key off the test name the agent just renamed.

Treat co-modification as a classification input, not as a style complaint. Measure it. Then score the patch against records the agent cannot author in the same change.

Three records the agent must not own

Keep three artifacts outside the writable test tree of the proposal:

1. Fixture bytes with a digest. Inputs are files, not “whatever the test constructed this morning.”
2. Property oracles that do not mention golden expected blobs from the patch. They check invariants: parse-roundtrip, conservation, bounds, idempotence.
3. Observation freezes keyed by digest plus property id, not by tests/test_foo.py::test_bar.

A freeze stores outcomes across repeated runs. It does not store “skip this node.” If variance exceeds a budget, the property stays unmerged. It does not become a silent skip.

Protocol

1. Measure the co-edit

Classify every path in the patch before any score. Production paths and test paths are separate buckets. A patch that only edits src/ is eligible for the normal suite. A patch that also edits tests/ loses the right to use those tests as the merge oracle.

git diff --name-only base...HEAD
git diff --stat base...HEAD -- src tests

# proposal: classify_coedit.py
from pathlib import Path

SRC_ROOTS = ("src/", "lib/")
TEST_ROOTS = ("tests/", "test/")

def classify(paths: list[str]) -> dict[str, list[str]]:
    src, tests, other = [], [], []
    for raw in paths:
        p = raw.replace("\\", "/")
        if p.startswith(SRC_ROOTS):
            src.append(p)
        elif p.startswith(TEST_ROOTS):
            tests.append(p)
        else:
            other.append(p)
    return {"src": src, "tests": tests, "other": other}

def tests_are_admissible(buckets: dict[str, list[str]]) -> bool:
    # Tests in the same patch cannot score that patch.
    return not buckets["tests"]

Short rule: if tests is non-empty, the suite that lives in those files is commentary. It is not a gate.

2. Bind fixtures by digest

Store inputs as immutable files. Hash them. Refuse a run that cannot show the digest in the freeze record. Do not let the agent patch “fix” a fixture in the same commit that changes the parser.

# proposal: fixture_bind.py
import hashlib
from pathlib import Path

def digest_file(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as f:
        for chunk in iter(lambda: f.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()

def bind_dir(root: Path) -> dict[str, str]:
    out = {}
    for p in sorted(root.rglob("*")):
        if p.is_file():
            out[p.relative_to(root).as_posix()] = digest_file(p)
    return out

If a later patch needs a new fixture, land the fixture first, on its own commit, with a human note. Then score the agent change against that digest. Mixing both in one diff collapses the independent input.

3. Run properties against those bytes

Property checks must not read expected JSON from the patch. They read fixture bytes and assert an invariant. The example below is a tiny invoice journal. It is illustrative, not a production ledger.

# proposal: properties.py
import json
from decimal import Decimal, InvalidOperation
from pathlib import Path

def load_journal(path: Path) -> list[dict]:
    rows = []
    for line in path.read_text(encoding="utf-8").splitlines():
        if not line.strip():
            continue
        rows.append(json.loads(line))
    return rows

def prop_roundtrip(path: Path) -> None:
    rows = load_journal(path)
    dumped = "\n".join(json.dumps(r, separators=(",", ":"), sort_keys=True) for r in rows)
    again = [json.loads(line) for line in dumped.splitlines() if line]
    assert again == rows

def prop_amounts_are_decimal(path: Path) -> None:
    for row in load_journal(path):
        try:
            amount = Decimal(str(row["amount"]))
        except (KeyError, InvalidOperation) as exc:
            raise AssertionError(f"bad amount in {row!r}") from exc
        assert amount == amount.quantize(Decimal("0.01"))

def prop_balanced_if_posted(path: Path) -> None:
    posted = [r for r in load_journal(path) if r.get("status") == "posted"]
    total = sum((Decimal(str(r["amount"])) for r in posted), Decimal("0"))
    # Invariant: posted lines in one journal net to zero.
    assert total == Decimal("0"), total

These oracles can fail even when the agent-updated unit test shows a new expected blob. That is the point.

4. Corroborate on a second host

Local flakes are not a freeze signal. Scheduler noise, dirty trees, and leftover env vars all look like intermittence. Re-run the same observation command on a host that does not mount the agent’s working tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free server option is one place to park that second run. Free model access is useful only after the observation exists: draft a candidate invariant from a failure log, then require a human to accept the property before it becomes a gate. Do not let a model approve its own patch.

Pin the command, the fixture digest, and the commit. Compare outcome vectors, not log text.

python -m observation_run --fixtures fixtures/invoices --commit "$BASE" --repeat 5
python -m observation_run --fixtures fixtures/invoices --commit "$HEAD" --repeat 5

If local HEAD fails and the second host fails with the same property id and digest, treat it as a regression. If only one host fails, you do not yet have a freeze. You have an environment bug.

5. Freeze the observation, not the node id

# proposal: observation_freeze.py
from dataclasses import dataclass
from collections import Counter

@dataclass(frozen=True)
class Observation:
    fixture_digest: str
    property_id: str
    commit: str
    outcomes: tuple[str, ...]  # "pass" | "fail" | "error"

@dataclass(frozen=True)
class FreezeDecision:
    action: str
    reason: str

def decide(obs: Observation, variance_budget: int, tests_in_patch: bool) -> FreezeDecision:
    counts = Counter(obs.outcomes)
    n = len(obs.outcomes)
    fails = counts["fail"] + counts["error"]
    if n == 0:
        return FreezeDecision("reject", "empty observation")
    if tests_in_patch:
        # Agent-authored tests cannot clear a fail, and cannot mint a freeze.
        if fails:
            return FreezeDecision("reject", "property failed; co-modified tests ignored")
        return FreezeDecision("hold", "pass is untrusted while tests changed")
    if fails == 0:
        return FreezeDecision("admit", "stable pass")
    if fails == n:
        return FreezeDecision("reject", "stable fail")
    if fails <= variance_budget:
        return FreezeDecision("hold", "under budget; do not freeze a name, rerun")
    return FreezeDecision("reject", "over variance budget; not a skip")

hold means rerun or split the fixture. It does not mean pytest.mark.skip. A skip keyed by test name is how co-modified flakes disappear.

Decision table

Co-modified tests	Local vector	Second-host vector	Action
no	all pass	all pass	admit
no	all fail	all fail	reject patch
no	mixed, fail ≤ budget	mixed, same property	hold; rerun; do not skip
no	mixed	disagree with local	environment bug; no freeze
yes	all pass	all pass	hold; tests in patch are not a score
yes	any fail	fail on same property + digest	reject; ignore rewritten unit tests
yes	fail	pass	reject freeze; split fixture or fix host
either	empty / missing digest	anything	reject observation

The table is the gate. Chat from the agent is not.

What this does not claim

It does not claim property checks replace unit tests. Unit tests remain useful when humans write them on a different commit from the behavior change. It does not claim a second host removes all flakes. It only removes the freeze decision from a single dirty tree.

It also does not assign a numeric reliability to any model. Free model drafts of properties are untrusted text until a reviewer accepts the invariant and the fixture digest.

Clock-dependent properties still need an injected clock. Network properties still need a fake transport. Those are fixture problems. They are not reasons to freeze a renamed test.

Who should not use this

Skip the protocol if agents are forbidden from editing tests/ and your review already blocks that path. Skip it if the system has no replayable fixtures: pure UI snapshots, live vendor APIs, or hardware clocks with no seam. Skip it if the team wants auto-merge on a green check from the same diff that changed assertions.

Do not use observation freezes as a backlog for known bugs. A hold that lasts more than one review cycle is an unfixed invariant, not a process win. Expire the hold by splitting the fixture or deleting the property. Do not convert it into a skip list.

Close

Score the patch against bytes and invariants the agent did not write in that diff. If you need a second machine for the corroboration step, a free server that never sees the proposal worktree is enough. The merge question stays the same: did the observation move, or did the test file?