Your Fixture Passed. The Agent Patch Is Still Wrong.

Fixed fixtures are examples, not oracles. An agent patch can sail through a green suite and still break on an input your fixture never imagined. This article proposes a gate that treats fixtures as seeds for property checks, requires those checks to fail on the old code, and freezes flaky failures before they poison the next run. The whole thing is small enough for a free server.

A Passing Fixture Is a Single Data Point

Consider a normalizer that sorts event tags. The fixture stores ["alpha", "beta"]. The agent patch silently removes the sort. The test still passes, because the fixture is already sorted.

That is not a testing failure. It is the normal relationship between examples and programs. A fixture proves only one point in the input space. An agent model that trained on thousands of GitHub examples will not stop at a 30-line fixture file. To put up a barrier, generate adversarial variants of that fixture and check invariants across both old and new behavior.

The Gate in Three Layers

1. Property checks over mutated fixtures catch regressions in ordering, whitespace, extra fields, and boundary integers.
2. Fixture locks catch accidental output drift by storing a deterministic snapshot of what the old code did for the same input.
3. A flaky freeze keeps nondeterministic failures from blocking the pipeline. When a failure cannot reproduce within a fixed number of reruns on the same seed, it moves to a quarantine file with an expiry time.

Three layers is enough. More layers usually means more coordination, and a free-server budget does not need a distributed test harness.

The Artifact: Fixture Strategy + Old-Code Fail

The artifact is a Hypothesis test that mutates a base fixture and compares current behavior with behavior from HEAD~1. First, define a fixture strategy.

from copy import deepcopy
import hypothesis.strategies as st
from hypothesis import given, settings, HealthCheck

BASE = {
    "id": "evt_123",
    "ts": "2026-09-02T10:00:00Z",
    "tags": ["alpha", "beta", "gamma"],
    "meta": {"retries": 2, "source": "sensor"},
}

def fixture_strategy():
    return st.fixed_dictionaries({
        "id": st.text(min_size=1, max_size=24),
        "ts": st.sampled_from([
            "2026-09-02T10:00:00Z",
            "2026-09-02T10:00:00+00:00",
            "2026-09-02 10:00:00",
        ]),
        "tags": st.lists(
            st.sampled_from(["alpha", "beta", "gamma", "delta"]),
            max_size=6,
        ),
        "meta": st.fixed_dictionaries({
            "retries": st.integers(min_value=0, max_value=4),
            "source": st.sampled_from(["sensor", "webhook", "cron"]),
        }),
    })

Then the test. normalize_old is loaded from the previous commit, not from memory.

@settings(max_examples=300, suppress_health_check=[HealthCheck.too_slow])
@given(fixture_strategy())
def test_normalize_matches_old(event):
    current = normalize(deepcopy(event))
    original = normalize_old(deepcopy(event))
    assert current == original

The gate rule is asymmetric. The current implementation must pass this test, and the same test must fail against HEAD~1. If it fails on neither, the test cannot see the regression. If it fails on both, the regression is in the test or the fixture.

Run both sides in a shell:

git worktree add /tmp/agent-base HEAD~1
TARGET=/tmp/agent-base python -m pytest test_fixture_gate.py --max-examples 300
TARGET=current python -m pytest test_fixture_gate.py --max-examples 300

Wire TARGET through a conftest that inserts the requested path. The current run should be green. The old run should not. That asymmetry is the signal.

Why the Old-Code Fail Is the Real Signal

A test that passes on both versions is decoration. When an agent patch regresses a parser, a good property check will produce an input that the old code handled and the new code does not. If your property suite cannot find that input, it does not have enough discriminating power.

This is mutation testing, but inverted. Instead of injecting a fault into the code, you replay the actual fault the agent just introduced. The gate either catches it or it does not. A green checkmark on the new code means nothing without a red failure on the old code.

Running It on a Free Server

The heavy part is the @given(max_examples=300) loop, which is CPU-bound and ephemeral. The free server option from MonkeyCode is enough for a cron job that runs every few hours. If a mutated fixture fails, use the free model output to sketch an explanation; the pass/fail decision stays with the test suite.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already have CI, add this gate as a scheduled job instead of a blocking status check. Let it run silently for a week, then count how many times it flagged a patch that human review had called safe. That number is your test suite's actual discrimination rate.

Limitations

This approach assumes the old code is importable. For compiled languages or heavy migrations, loading HEAD~1 may be impractical. It also assumes behavior should stay the same; intentional behavior changes need an allowlist or the gate will reject them.

Use a small schema-based filter when your fixtures contain domain-specific fields. Mutating a UUID into an empty string is a fake bug. And if your project has a stable input DSL with few edge cases, a full property gate is probably overkill. Start with the base fixture, the old-code fail check, and a quarantine file of no more than ten entries.