Three Independent Checks for an Agent Patch: Fixture Contract, Reference Oracle, and a Flaky Quarantine

The first question to ask about an agent-generated patch is not did the tests pass? It is who wrote the tests? If the same agent produced both code and assertions, a green suite can mean the patch answered its own questions. This article uses three independent sources: a fixture contract checked in before the patch, a differential property test against a reference implementation, and a quarantine list for flaky tests. Each source stands alone, so if two of the three miss the bug, the third still has a chance.

Why fixtures are a contract, not data

A fixture set is a contract when it is committed before the agent starts. The agent can propose code changes, but it cannot touch tests/fixtures/behavior.json. That restriction matters. If tests are allowed to change during the patch, a bug can disappear by editing the expected output instead of fixing the code.

Example: a small function that applies a preference patch to a settings object.

domain/preferences.py:

def apply_patch(base, patch):
    merged = dict(base)
    merged.update(patch)
    return merged

A fixture declares the behavior that must survive any refactor:

tests/fixtures/behavior.json:

[
  {
    "name": "override disables notifications",
    "base": {"theme": "dark", "notifications": "on"},
    "patch": {"notifications": "off"},
    "expected": {"theme": "dark", "notifications": "off"}
  },
  {
    "name": "empty patch is a no-op",
    "base": {"theme": "dark"},
    "patch": {},
    "expected": {"theme": "dark"}
  }
]

The test reads the fixture and compares the result with the expected object:

import json
from pathlib import Path
import pytest
from domain.preferences import apply_patch

FIXTURES = json.loads(Path('tests/fixtures/behavior.json').read_text())

@pytest.mark.parametrize('case', FIXTURES, ids=lambda c: c['name'])
def test_fixture_contract(case):
    result = apply_patch(case['base'], case['patch'])
    assert result == case['expected']
    assert result is not case['base']

The last assertion is the one an agent often misses. It checks that applying a patch does not mutate the original object.

Differential properties: the reference oracle

Hand-written unit tests encode what one developer remembers. Reference properties encode a relationship. Before the agent touches real code, pick a previous version of apply_patch that the team trusts, or write a small reference implementation with the exact semantics you intend to keep.

Then use a property-based test to compare the new implementation with the reference over many inputs:

from hypothesis import given, strategies as st
from domain.preferences import apply_patch
from domain.preferences_reference import apply_patch as reference

@given(
    base=st.dictionaries(st.text(), st.text()),
    patch=st.dictionaries(st.text(), st.text()),
)
def test_agent_patch_matches_reference(base, patch):
    assert apply_patch(base, patch) == reference(base, patch)

@given(
    base=st.dictionaries(st.text(), st.text()),
    patch=st.dictionaries(st.text(), st.text()),
)
def test_patch_never_adds_keys_outside_union(base, patch):
    allowed = set(base) | set(patch)
    assert set(apply_patch(base, patch)) <= allowed

Differential testing is stronger than a single assertion because it generates thousands of inputs. When a case fails, Hypothesis shrinks it to the smallest counterexample. The agent can then fix a reproducible bug instead of investigating a vague some inputs are wrong.

If a reference implementation is impossible, use a weaker invariant: the patch must be deterministic, it must not add keys from outside the union, and it must not mutate its arguments. These three properties catch most accidental state bugs.

The quarantine file, not a retry loop

Flaky tests are normal in agent workflows because the patch changes timing, caching, or external calls. The standard answer is retry three times. That is the wrong default. A retry makes the flake invisible until it happens twice in a row. It also gives the agent false confidence.

Keep a quarantine instead. When a test fails sporadically, move it to a freeze list with metadata. The test stops running in the default suite, but the record stays in the repo.

tests/quarantine.json:

[
  {
    "name": "test_user_source_order",
    "first_seen": "2026-08-25",
    "last_seen": "2026-08-28",
    "failures": 3,
    "reason": "race between cache invalidation and polling"
  }
]

A small fixture in conftest.py skips tests that appear in the quarantine file:

import json
from pathlib import Path
import pytest

QUARANTINE = json.loads(Path('tests/quarantine.json').read_text())
QUARANTINED_NAMES = {item['name'] for item in QUARANTINE}

def pytest_collection_modifyitems(items):
    for item in items:
        if item.name in QUARANTINED_NAMES:
            item.add_marker(pytest.mark.skip(reason='quarantined flake'))

This is a freeze, not a deletion. The test must stay in the repository. A nightly job can re-enable quarantined tests and check whether they pass five times in a row. If they do, remove them from quarantine. If they do not, the metadata in the JSON shows whether the failure is one environment or a real regression.

Where MonkeyCode fits

MonkeyCode's free model access and free server option are relevant here in only one way: they make it cheaper to run this validation loop on each generated patch. The value of the loop is independent of the tool, and the same checks work with any agent. What matters is the discipline of running the fixture contract, the differential properties, and the quarantine review on every patch. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When this strategy will not help

This strategy fails in three situations. First, when there is no reference implementation and the invariants are too weak, differential tests are almost useless. Second, when the bug is visual or timing-dependent, property tests and JSON fixtures do not see the layout or the race. Third, when the quarantine is not reviewed. If everyone adds flaky tests and nobody cleans the list, the quarantine becomes an attic where regressions can hide.

Use this setup for data-processing functions, pure merges, serializers, and any patch that has a stable input/output shape. Do not use it for UI redesigns, network protocols with changing behavior, or systems where every run is expected to depend on the environment. The goal is to make the agent prove a claim against sources that the agent did not write.