Fixture Drift: Why Agent Patches Pass Green Tests and Break Production

Your agent patch is green. Every unit test passes. Two weeks later, production breaks and the blame lands on the patch.

Often the patch was fine. The fixture was a liar.

Fixture drift happens when test data no longer matches the contracts the code actually enforces. Agents change the code. Nobody changes the fixtures. The tests stay green because they assert against an old reality.

A better approach is to audit the fixtures themselves with property checks, mutate them to see if your tests care, and freeze the checks that flicker instead of retrying them.

Why fixtures decay

Fixtures look inert, but they act as a hidden specification. When an agent changes validation rules, date handling, or ID generation, the fixture may still encode the old rules.

Common drift patterns:

• Hardcoded timestamps that no longer match timezone logic
• IDs that violate new uniqueness constraints
• Nested objects missing fields the patch now requires
• Enum values that were renamed but only appear in fixtures
• Ordinal relationships broken by a new sorting behavior

The worst part: your test suite will not tell you. The agent changed code, the fixture didn't, and the two disagree silently.

Step 1: Write property predicates for your fixtures

Start by defining what must always be true about any fixture. These are not unit tests. They are invariants over all fixtures.

Here is a simple Python auditor that checks a few generic properties:

# fixture_auditor.py
import json, glob

FIXTURE_PATH = "fixtures/*.json"

def load_fixtures():
    data = []
    for path in glob.glob(FIXTURE_PATH):
        with open(path) as f:
            data.extend(json.load(f))
    return data

def unique_ids(fixtures):
    ids = [f["id"] for f in fixtures]
    return len(ids) == len(set(ids)), "duplicate id"

def positive_balances(fixtures):
    for f in fixtures:
        if f.get("balance", 0) < 0:
            return False, f"negative balance in {f}"
    return True, "ok"

def timestamps_in_order(fixtures):
    for f in fixtures:
        if f.get("created_at") and f.get("updated_at"):
            if f["created_at"] > f["updated_at"]:
                return False, f"created after updated: {f}"
    return True, "ok"

if __name__ == "__main__":
    fixtures = load_fixtures()
    checks = [unique_ids, positive_balances, timestamps_in_order]
    failed = False
    for check in checks:
        ok, msg = check(fixtures)
        if not ok:
            failed = True
            print(f"FAIL: {check.__name__}: {msg}")
    if not failed:
        print("All fixture properties hold.")
    else:
        print("Fixture drift detected.")

Run this on your existing fixture directory. If a property fails, you have found drift before the agent ever touches the code.

Step 2: Mutate fixtures and see if tests notice

Property checks catch known invariants. Mutation testing catches unknown ones.

The idea: mutate a field in a fixture and rerun your test suite. If the suite still passes, your tests did not observe that behavior. The fixture is not anchoring the contract it should be anchoring.

Here is a minimal mutation harness for a Python project:

# mutate_fixtures.py
import json, glob, shutil, subprocess

FIXTURE_PATHS = glob.glob("fixtures/*.json")
MUTATIONS_TO_TRY = ["id", "balance", "created_at", "status"]

for path in FIXTURE_PATHS:
    original = open(path).read()
    data = json.loads(original)
    # mutate first item's second field to a sentinel
    item = data[0]
    field = MUTATIONS_TO_TRY[len(path) % len(MUTATIONS_TO_TRY)]
    old = item.get(field)
    item[field] = "DRIFT_SENTINEL" if old != "DRIFT_SENTINEL" else "OTHER"
    open(path, "w").write(json.dumps(data, indent=2))
    result = subprocess.run(["pytest", "-q"], capture_output=True)
    survived = result.returncode == 0
    tag = "survived" if survived else "caught"
    print(f"{path}: mutating {field} -> {tag}")
    open(path, "w").write(original)

This is intentionally crude, but it is honest. It answers one question: if a field drifts, will any test notice? Run it on a small fixture set first. Expect a mix of caught and survived mutations. The survived ones are your blind spots.

Step 3: Freeze flaky checks, do not retry them

Once you add property checks, you will also find checks that are inherently timing-dependent or flaky. The old instinct is to retry. That hides the problem and slows CI.

The better instinct is to freeze the check. Pin it to a known failing state, record why, and exclude it from the default run.

# pytest.ini
[pytest]
addopts = -q
markers =
    frozen: known flaky, requires fixture refresh

Then in your test file:

import pytest

@pytest.mark.frozen
@pytest.mark.parametrize("fixture", load_fixtures())
def test_every_fixture_has_localized_timestamp(fixture):
    # Known failure: fixtures created before the timezone migration
    pass

Run with:

pytest -m "not frozen"

Freezing is not permission to forget. Keep a short backlog per frozen check with the responsible fixture file and a refresh date. If the frozen count grows beyond a small number, fix the fixtures instead.

Where MonkeyCode fits

MonkeyCode's free model access can shorten the first pass. Paste a diff from your agent patch into the model and ask for a list of invariants the fixture file should satisfy. That gives you a starting set of property predicates to add to fixture_auditor.py.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The draft is a suggestion, not a verdict. Every predicate still needs to run against your actual fixtures, and the mutation step remains the source of truth.

The free server option is useful too. Running an audit across a large fixture directory can spin your laptop fan into a jet engine. Offload that one command to MonkeyCode's free server, capture the report, and kill the process locally. The report is a plain text file, so there is no lock-in.

Limitations and who should not use this

This workflow will not catch every semantic change. If an agent refactors a calculation but every fixture still produces the same output, property checks and mutation both stay quiet. You still need code review for semantic intent.

It also requires ownership of the fixtures. If you are consuming third-party test data that changes without your control, freezing your checks means you are freezing against someone else's contract. That is a trap.

Teams with tiny fixture sets can skip this. If you have ten files and three properties, the script becomes ceremony. Use it when the fixture count makes manual auditing impossible.

The artifact you can leave with

The reusable artifact is the pair of scripts: fixture_auditor.py for invariants and mutate_fixtures.py for blind-spot discovery. Both are deliberately small. You can copy them, adapt the fields, and run them on any project that uses JSON fixtures.

The first time you see a mutation survive your entire suite, you will stop trusting green tests. That is the point.