Give Your Agent Patch a Determinism Budget: Seed-Locked Properties, Pinned Fixtures, a Freeze Registry

Last week a property test failed on run 3, passed through run 7, and failed again after lunch. The patch under review was not mine. It had been generated by an agent, and the only evidence I had was the test suite. The suite said two contradictory things in the same hour. The patch did not necessarily become wrong because the test was flaky, but the gate could not prove it was right either.

The fix was not more retries. The fix was a determinism budget: one fixed seed for every property check, one pinned fixture profile for every environment variable, and a freeze list for tests that still refused to be stable. This post is that budget, implemented in Python, and runnable on any CI that supports pytest.

Why property tests become flaky

A property test is a random generator paired with an assertion. The generator is a fixture. The seed is also a fixture, but most teams treat it as invisible. When the seed changes, the example distribution changes. A real edge-case bug will then either appear inside your 200 examples or it will not.

That is not a failure of property testing. It is a missing fixture.

For agent patches the stakes are worse. The patch was not written by someone who knows which examples are dangerous. The gate must therefore be transparent about one thing: every test must run with the same inputs, in the same environment, or the verdict cannot be attributed to the patch.

Step 1 — Lock the seed into a fixture

If the seed lives in a central configuration object, every property test uses the same example stream.

# determinism.py
DETERMINISM = {
    "seed": 20260830,
    "max_examples": 200,
    "tz": "UTC",
    "cache_dir": "tmp",
}

The fixture applies the budget before any property test runs:

# conftest.py
import json
from determinism import DETERMINISM

with open("patch-gate.json") as f:
    PROFILE = json.load(f)

@pytest.fixture
def pinned_env(tmp_path, monkeypatch):
    monkeypatch.setenv("TZ", DETERMINISM["tz"])
    monkeypatch.setenv("PYTHONHASHSEED", str(DETERMINISM["seed"]))
    monkeypatch.setenv("APP_SEED", str(DETERMINISM["seed"]))
    monkeypatch.setenv("CACHE_DIR", str(tmp_path / "cache"))
    return tmp_path

Now the same seed is a first-class test fixture, not an accident of scheduling.

Step 2 — Pin everything the patch might touch

A seed alone does not protect you from a patch that reads TZ differently, changes cache paths, or depends on the current date. A fixture profile pins the full environment for the duration of the gate:

{
  "profile": "patch-gate",
  "seed": 20260830,
  "env": {
    "TZ": "UTC",
    "PYTHONHASHSEED": "20260830",
    "APP_SEED": "20260830",
    "CACHE_DIR": "tmp_path/cache"
  },
  "freeze": ["test_legacy_race", "test_sketchy_timing"]
}

The property test and the fixture meet in one function:

from hypothesis import given, seed, settings
from hypothesis import strategies as st

@seed(PROFILE["seed"])
@settings(max_examples=DETERMINISM["max_examples"], deadline=None)
@given(values=st.lists(st.integers()), value=st.integers())
def test_insert_preserves_sortedness(pinned_env, values, value):
    from bisect import insort
    out = values[:]
    insort(out, value)
    assert out == sorted(out)
    assert len(out) == len(values) + 1

If this fails, the failure is reproducible. If it passes, the patch saw a fixed example stream and a fixed environment.

Step 3 — Freeze the flaky before it gates

A test that still disagrees with itself under a fixed seed and a pinned profile is not evidence. Retrying converts it from noise into a decorative pass. Freezing converts it into an explicit skip:

# conftest.py
FROZEN = set(PROFILE["freeze"])

def pytest_collection_modifyitem(config, items):
    for item in items:
        if item.name in FROZEN:
            item.add_marker(pytest.mark.skip(reason="flake freeze until stable"))

The frozen test does not count toward the patch verdict. It is a debt line, not a signal.

The decision matrix

Observed result	Meaning	Gate action
Fixed-seed property check fails with counterexample	The patch changed behavior	Reject and attach the reproduction
Fixture-dependent test fails inside the patch diff	The patch relies on an unpinned variable	Re-run on the profile; if still red, reject
Fixture-dependent test fails outside the patch diff	Environment drift, not patch evidence	Fix the environment and re-run
Frozen flaky test would have failed	No new information	Do not count toward the verdict
All checks pass on fixed seed + pinned env	Stable under this budget	Human review only, not auto-merge

Where to run this budget

MonkeyCode's free server option is convenient for this because the gate runs in a throwaway environment, away from your local checkout. Its free model access can produce an initial fixture profile from your repo layout, but treat that profile as code and review it like code. The budget lives in your repository, not in the model's memory. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this budget does not prove

A fixed seed reduces exploration. The next 10,000 examples might still contain a regression that your 200 did not. A pinned environment also cannot replace real contract tests against external services. If you rely on a live database or a third-party API, disabling it in the fixture gives you a green suite that is more confident than the production system it represents.

Do not use this approach when the patch changes a randomness-critical component such as a sampler or a cryptographic primitive. Determinism there creates false safety.

The gate is only as honest as its fixtures

A patch review pipeline is a claim about attribution: this failure is the patch's fault, this failure is the environment's fault, and this failure is too noisy to use. The determinism budget makes that claim testable. Write the seed down. Pin the environment. Freeze the noise. Then let a human make the final call.