Pin Fixtures, Run Properties, Freeze Flakes: A Test Contract for Agent Patches

An agent patch is not verified by a green job. It is verified by three artifacts that still make sense on an empty disk: a hashed fixture manifest, property checks that refuse ambient state, and a freeze list that turns flakes into merge blockers instead of retries. Retrying a non-deterministic test on an ephemeral runner does not produce evidence. It produces a lottery ticket.

This contract is a proposal, not a production study. The commands and files below are labeled examples. They are meant to be copied into a throwaway repo and executed, not cited as metrics.

The failure mode the green job hides

Agent-authored patches often arrive with agent-authored tests. Those tests tend to pass for three cheap reasons that have nothing to do with the change.

First, fixtures leak in from a previous job: caches, downloaded corpora, Docker layers, locale files, or a home-directory config the model never declared. Second, the assertions are tautological against the implementation the agent just wrote, so they cannot fail. Third, a flake is rerun until it passes, and the gate records only the last status.

Ephemeral runners make the first reason louder. A free or short-lived server is useful because it starts closer to empty. It is dangerous for the same reason. Anything the patch needs and does not reconstruct is an undeclared dependency.

A practical place to exercise the contract is a workspace that can call a model without a paid key and run the resulting patch on a free server. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode exposes free model access and a free server option; those two facts are the only product claims used here. No model names, quotas, or hardware details are assumed. The gate below does not depend on a vendor. It depends on treating the runner as hostile to leftover state.

The three-layer contract

Treat every agent patch as a triple.

1. Fixture manifest. A committed list of files, URLs, and hashes the tests are allowed to read. If reconstruction cannot match the hashes, the job fails closed.
2. Property checks. Assertions about invariants of the change, not about a single recorded output. They consume only reconstructed fixtures.
3. Flake freeze. A versioned list of tests that have shown pass/fail disagreement after reconstruction. Frozen tests are skipped and they block merge until they expire or are rewritten. They are never retried for credit.

If any layer is missing, the patch is untested, even if CI is green.

Layer 1: a fixture manifest the runner must rebuild

Keep the manifest in the repo. Do not generate it on the runner and then trust it. The agent may edit tests; it must not be the only writer of the hashes those tests depend on.

{
  "version": 1,
  "fixtures": [
    {
      "id": "invoice_utf8",
      "path": "testdata/invoice_utf8.json",
      "sha256": "e3b0c44298fc1c149afbf4c8996fb924",
      "role": "input"
    },
    {
      "id": "tax_table",
      "path": "testdata/tax_table.csv",
      "sha256": "2c624232cdd221771294dfbb310aca00",
      "role": "oracle-seed"
    }
  ]
}

The hashes above are placeholders. Replace them with real SHA-256 digests from sha256sum. Reconstruction is a gate, not a convenience script.

# reconstruct_fixtures.py — example, not a shipped tool
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

MANIFEST = Path("testdata/fixture_manifest.json")


def sha256(path: Path) -> str:
    h = hashlib.sha256()
    with path.open("rb") as fh:
        for chunk in iter(lambda: fh.read(65536), b""):
            h.update(chunk)
    return h.hexdigest()


def main() -> int:
    spec = json.loads(MANIFEST.read_text())
    errors = []
    for item in spec["fixtures"]:
        path = Path(item["path"])
        if not path.is_file():
            errors.append(f"missing {path}")
            continue
        digest = sha256(path)
        if digest != item["sha256"]:
            errors.append(f"hash mismatch {path}: got {digest}")
    if errors:
        print("FIXTURE RECONSTRUCTION FAILED", file=sys.stderr)
        print("\n".join(errors), file=sys.stderr)
        return 2
    print(f"reconstructed {len(spec['fixtures'])} fixtures")
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run order is fixed.

python reconstruct_fixtures.py || exit 2
pytest -q --strict-markers tests/properties tests/unit

--strict-markers matters. A freeze marker that is not registered should fail the job, not disappear into a skip.

If the agent “fixes” a test by rewriting testdata/invoice_utf8.json without updating the manifest hash, reconstruction fails. That is the intended outcome. Silent fixture edits are how golden-file games start. This layer does not compare two environments. It asks one empty disk whether the declared inputs still exist.

Layer 2: property checks that cannot see the room

Unit tests that assert fn(sample) == expected are allowed. They are not sufficient for an agent patch, because the agent can set expected. Property checks constrain the function across a space the agent does not get to enumerate in a commit.

Rules for this layer:

1. Properties import fixtures only through the reconstructed paths.
2. Properties do not read os.environ except for an allowlisted job id.
3. Properties do not call the network. If a corpus must be fetched, it belongs in the manifest and is pinned.
4. A property failure rejects the patch. No rerun budget.

Example using Hypothesis. Treat it as a template.

# tests/properties/test_tax_invariants.py — example
from decimal import Decimal, ROUND_HALF_EVEN
from hypothesis import given, settings, strategies as st

from app.tax import apply_tax
from tests.support.fixtures import load_csv  # reads only manifest paths


RATES = {
    row["region"]: Decimal(row["rate"])
    for row in load_csv("testdata/tax_table.csv")
}


@settings(max_examples=200, deadline=None)
@given(
    amount=st.decimals(min_value="0.01", max_value="1000000", places=2),
    region=st.sampled_from(sorted(RATES)),
)
def test_tax_is_monotone_and_rounded(amount: Decimal, region: str) -> None:
    rate = RATES[region]
    taxed = apply_tax(amount, region)
    expected = (amount * (1 + rate)).quantize(Decimal("0.01"), rounding=ROUND_HALF_EVEN)
    assert taxed == expected
    assert taxed >= amount

The property does not care which model proposed apply_tax. It cares that rounding is explicit and that the rate table is the reconstructed one. If the agent hard-codes a single region that its unit test covers, this job still moves.

A second property should lock a negative space the unit tests usually skip: invalid input remains invalid.

@given(raw=st.text(min_size=0, max_size=64))
def test_unknown_region_still_raises(raw: str) -> None:
    if raw in RATES:
        return
    try:
        apply_tax(Decimal("10.00"), raw)
    except ValueError:
        return
    raise AssertionError(f"unknown region accepted: {raw!r}")

That is not a must-reject corpus. It is a generator. The difference is operational: a corpus is a finite list an agent can overfit. A property is a rule the agent would have to rewrite in the test file, which the review diff should make obvious.

Layer 3: freeze flakes, do not retry them

Retries convert intermittent failures into a biased pass. On a free or busy runner, retries also hide queue noise as product noise. The contract forbids credit retries.

Detection is a paired execution after reconstruction, not a “rerun failed jobs” click.

# example detection, not a benchmark
python reconstruct_fixtures.py || exit 2
pytest -q --strict-markers tests --junitxml=/tmp/run1.xml
status1=$?
pytest -q --strict-markers tests --junitxml=/tmp/run2.xml
status2=$?
python tools/diff_junit.py /tmp/run1.xml /tmp/run2.xml --out flake_candidates.txt

diff_junit.py should report tests whose status changed, not tests that failed twice. Failed twice is a bug. Changed once is a flake candidate. Candidate is not yet a freeze. A freeze requires an owner field and an expiry. Example freeze file:

# tests/flake_freeze.yaml — example
version: 1
policy:
  credit_retries: 0
  expired_is_failure: true
flakes:
  - nodeid: tests/unit/test_parser.py::test_handles_emoji
    first_seen: "2026-09-16"
    expires: "2026-10-14"
    owner: unassigned
    evidence: "pass then fail after fixture reconstruct; locale-sensitive sort"

The pytest plugin is small. Unknown node ids fail the job. Expired rows fail the job. unassigned owner fails the job on protected branches. Skip is not silence.

# tools/flake_freeze_plugin.py — example
from __future__ import annotations

import datetime as dt
from pathlib import Path

import pytest
import yaml

FREEZE = Path("tests/flake_freeze.yaml")


def pytest_configure(config):
    config.addinivalue_line("markers", "frozen_flake: skipped by freeze file")


def _rows():
    data = yaml.safe_load(FREEZE.read_text())
    return data.get("flakes", [])


def pytest_collection_modifyitems(config, items):
    today = dt.date.today()
    frozen = {row["nodeid"]: row for row in _rows()}
    for item in items:
        row = frozen.get(item.nodeid)
        if not row:
            continue
        expires = dt.date.fromisoformat(row["expires"])
        if expires < today:
            item.add_marker(pytest.mark.xfail(
                strict=True,
                reason=f"freeze expired {row['expires']}",
            ))
            continue
        if row.get("owner") == "unassigned":
            item.add_marker(pytest.mark.xfail(
                strict=True,
                reason="freeze has no owner",
            ))
            continue
        item.add_marker(pytest.mark.skip(reason=f"frozen flake until {row['expires']}"))

xfail(strict=True) on expiry is deliberate. An expired freeze that starts passing is still a process failure: the file was not cleaned up. An expired freeze that still fails is a real failure. Either way the gate stays red until a human edits the YAML.

Credit retries stay at zero. Infrastructure retries for runner eviction are a different class and should be labeled as such in the job log. Mixing them is how flakes get a shadow budget.

Decision table for the gate

Observed signal	Retry for credit?	Gate result	Required artifact
Fixture path missing or hash mismatch	No	Fail closed	Manifest diff
Property check failed	No	Reject patch	Counterexample from the runner
Unit test failed twice after reconstruct	No	Reject patch	JUnit from both runs
Same node id pass/fail across the two runs	No	Fail, add freeze candidate	flake_candidates.txt
Freeze row with owner: unassigned	No	Fail on protected branch	YAML edit
Freeze row past expires	No	Fail	Delete or rewrite the test
Agent edited fixture bytes without hash update	No	Fail closed	Manifest vs sha256sum
Agent deleted a property file	No	Fail if coverage of properties drops to zero	File list check

The last row needs a cheap check, not a coverage vendor.

# example: fail if the properties directory is empty after the patch
count=$(find tests/properties -name 'test_*.py' | wc -l)
if [ "$count" -lt 1 ]; then
  echo "property layer missing" >&2
  exit 2
fi

Zero is the threshold that matters. Do not invent a percentage and then negotiate it.

A workflow that fits a free model and a free server

The sequence is the same whether the patch was typed by a person or proposed by a model.

1. Check out a clean worktree. Do not reuse a dirty workspace as a cache.
2. Ask the model for a patch against a stated invariant, not against “make tests pass.”
3. Apply the patch on the free server as if the disk were empty.
4. Reconstruct fixtures from the manifest. Stop on mismatch.
5. Run property checks once. Stop on failure.
6. Run the rest of the suite twice only to detect disagreement, not to hunt a pass.
7. Any disagreement becomes a freeze candidate with expiry and owner. It does not become a retry.
8. Record the manifest hash, the property counterexamples, and the freeze file digest in the job summary.

Step 2 is the one people skip. If the prompt is “get CI green,” the model will edit fixtures, delete properties, or rename a flaky test. The contract above makes those edits expensive because they show up as hash mismatches, empty directories, or YAML churn.

MonkeyCode is optional in that sequence. Free model access is a way to generate a candidate. A free server is a way to execute the candidate without relying on a developer laptop’s caches. The contract is what you keep if the product names change.

What this does not prove

The contract does not measure blast radius, does not compare local and remote interpreters, and does not classify tautological unit tests. Those are separate gates. Mixing them into one script recreates the problem: a single green rectangle that no longer names a failure mode.

Property checks are only as honest as the strategies. If the strategy never produces the region the bug lives in, the job is theater. Freeze files rot. A freeze without expiry is a skip list. A freeze with a two-year expiry is also a skip list.

Who should not use this approach:

• Teams whose tests must talk to live third-party APIs. Pinning becomes a legal and billing problem, not a hash problem.
• Suites that are already deterministic and fully hermetic. The freeze layer will add YAML noise for no signal.
• Patches that are documentation-only. Reconstructing binary fixtures is wasted work.
• Anyone hoping a free runner will replace review. The freeze owner field exists because a person still has to look.

Limitations of the example code: it does not sandbox filesystem writes, it does not prove the SHA-256 placeholders, and it does not handle parameterized node ids until you normalize them. Hypothesis settings are local choices, not evidence of thoroughness.

If a gate already retries failed agent tests, remove the retry budget before adding more fixtures. The cheapest honest signal is still a reconstructed disk, one property that can fail, and a flake that is not allowed to pass by trying again.