Pin the Fixture, Budget the Run

The patch was twelve lines. It touched one adapter and one test file. CI went green on the fourth attempt, and nobody asked why.

That green is not evidence. It is three reruns and a silent skip, stacked until the runner agreed. Agent patches make this worse, because volume hides the pattern.

I have not run the incident above. Treat it as an illustrative shape, not a log. The mechanism is common enough to design against.

Three leaks, one cause

A rerun is an unrecorded vote. Each retry quietly deletes one failure from the record. Two failures out of five looks identical to zero out of one.

Fixtures drift without anyone editing a test. A regenerated snapshot or a rewritten JSON file changes the input, not the code. The suite still passes, but it now measures a different world.

Skips are the third leak. A patch that adds @pytest.mark.skip to a failing test has improved the build without improving the software. Every skip in a patch is a claim someone forgot to defend.

These three leaks share a cause. The patch decides what counts as done. The budget has to come from outside the patch.

Fix the inputs before the oracle

Start with the fixtures. Hash them, commit the hash, and fail closed on any mismatch. This is cheap and it removes an entire class of "it passed locally" arguments.

# tools/pin_fixtures.py
import hashlib, json, pathlib

FIXTURES = pathlib.Path("tests/fixtures")
LOCK = pathlib.Path("tests/fixtures.lock.json")

def digest(path: pathlib.Path) -> str:
    return hashlib.sha256(path.read_bytes()).hexdigest()[:16]

lock = {p.name: digest(p) for p in sorted(FIXTURES.rglob("*")) if p.is_file()}
LOCK.write_text(json.dumps(lock, indent=2, sort_keys=True) + "\n")
print(f"pinned {len(lock)} fixtures")

Run it once, commit the lock file, and check it from a session fixture. A drifted fixture should stop collection, not fail a random assertion three modules later.

# tests/conftest.py
import json, pathlib, pytest
from tools.pin_fixtures import FIXTURES, digest

LOCK = json.loads(pathlib.Path("tests/fixtures.lock.json").read_text())

@pytest.fixture(scope="session", autouse=True)
def fixtures_pinned():
    seen = {p.name: digest(p) for p in FIXTURES.rglob("*") if p.is_file()}
    drift = [n for n, h in seen.items() if LOCK.get(n) != h]
    if drift:
        pytest.exit(f"fixture drift: {drift}", returncode=3)

The exit code matters. A distinct code lets CI say "inputs changed" instead of "code broke". Those two sentences route to different people.

A freeze list that expires

Flaky tests need to leave the gate, but they must not vanish. A frozen test stays in the repository, out of the run, and on a clock.

{
  "tests/test_payments_retry.py::test_backoff_jitter": {
    "frozen_on": "2026-10-02",
    "expires": "2026-11-01",
    "owner": "payments"
  }
}

Check the expiry during collection. An expired entry is a hard error, not a warning. The test was unfrozen by the calendar, not by a review.

That single rule changes the incentive. Freezing buys thirty days of quiet. It does not buy permanent silence, and it does not let a patch hide behind a stale entry.

Budget the spend

The patch does not get an unlimited suite. Give it a time limit, a test-count limit, and a skip limit of zero. If the suite exceeds any of them, the patch is unverified, not failed.

# tools/budget.py
import json, sys

def main(path: str, max_seconds: float, max_tests: int, max_frozen: int) -> int:
    report = json.load(open(path))
    summary = report["summary"]
    checks = {
        "seconds": report["duration"] <= max_seconds,
        "tests": report["total"] <= max_tests,
        "frozen": summary.get("deselected", 0) <= max_frozen,
        "skips": summary.get("skipped", 0) == 0,
    }
    breached = [name for name, ok in checks.items() if not ok]
    if breached:
        print("budget breach: " + ", ".join(breached))
        return 1
    print("budget ok")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv[1], 240, 180, 0))

The numbers are local and should stay local. A 240 second ceiling fits a small service. A monorepo needs a different figure, and inventing one here would be theater.

Wire the gate as three commands. The sequence is boring on purpose.

python tools/pin_fixtures.py --check
pytest -q --json-report --json-report-file=report.json
python tools/budget.py report.json 240 180 0

Property checks scoped to the diff

Full property suites are slow. Scoped ones are honest. Generate the inputs, keep the oracle fixed, and only run properties for modules the patch touched.

# tests/properties/test_retry_policy.py
from hypothesis import given, settings, strategies as st
from payments.retry import backoff

@given(st.integers(min_value=1, max_value=8),
       st.floats(min_value=0, max_value=30, allow_nan=False))
@settings(max_examples=200, deadline=None)
def test_backoff_is_monotonic(attempt, base):
    assert backoff(attempt + 1, base) >= backoff(attempt, base)

Monotonic backoff is a property a reviewer can state in one sentence. That is the test. If the property cannot be stated without reading the implementation, it is not a property, it is a copy.

Selection stays mechanical.

git diff --name-only origin/main -- 'src/**' > changed.txt
python tools/select.py changed.txt > selected.txt
pytest -q $(cat selected.txt) tests/properties

select.py maps module paths to test files and is deliberately dumb. A clever selector is a new source of untested behavior.

Where a free runner fits

This loop needs two things: a model to draft property candidates from a diff, and a machine to run a budgeted suite. MonkeyCode's free model access and free server option cover both for a first pass, and the token allowance on the free tier is operator-supplied. Check the current limits on the project page before you build a pipeline around them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

My own use is narrow. The model proposes candidate properties in the docstring style above, and I keep or discard each one by hand. The server exists so the budget gate runs somewhere that is not a laptop. That is the whole integration. If you want to try the loop once end to end, the free tier is enough for a single patch.

None of the harness above depends on it. The scripts are plain Python and pytest.

What this does not fix

A property suite with no oracle is a tautology generator. If backoff is the only definition of correct backoff, the test proves nothing.

Small suites should skip the budget entirely. Below roughly twenty tests, a fixed ceiling mostly records noise, and you will spend more time tuning numbers than reading failures.

Teams with genuinely non-deterministic dependencies need to fix those first. A freeze list over a broken clock is a filing cabinet, not a gate.

And the numbers in budget.py are mine. They are a starting point with no benchmark behind them, so measure your own suite before trusting the ceiling.

The twelve-line patch still needs a human. The budget just stops it from grading its own homework.