An agent patch can pass every fixture in your repository and still be wrong. The fixtures are part of the oracle, and fixtures can be lazy. If the expected value in a fixture never changes, the suite can approve code that breaks exactly the case the fixture forgot.
This is a three-phase probe for evaluating agent-generated patches. Phase 1 runs property checks before anything else. Phase 2 mutates fixtures to prove they can fail. Phase 3 freezes flaky tests only after the first two phases pass. Order matters.
The probe fits on MonkeyCode's free server option, and MonkeyCode's free model access is useful for drafting the invariant skeleton from a patch's diff. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Why Golden Fixtures Are Weak Oracles
Golden fixtures assert one output for one input. They catch regressions on paths you already knew about. Agent patches are good at preserving known paths and bad at inventing new cases. If the fixture never expected an edge case, the patched code will happily return whatever it wants.
That is the trap: the suite turns green because the oracle is incomplete, not because the behavior is correct.
Phase 1: Run Property Checks Before You Trust the Diff
Write an invariant that must hold for all inputs, then feed the function random inputs. A simple C++ interval merger demonstrates the idea.
#include <algorithm>
#include <cassert>
#include <random>
#include <vector>
struct Interval { int start; int end; };
// The patch under review. Replace this with the agent's implementation.
std::vector<Interval> merge_intervals(std::vector<Interval> input);
long covered(const std::vector<Interval>& intervals) {
long total = 0;
for (const auto& x : intervals) total += x.end - x.start;
return total;
}
bool invariant(const std::vector<Interval>& result) {
for (size_t i = 1; i < result.size(); ++i) {
if (result[i].start < result[i - 1].end) return false;
}
return true;
}
void property_probe(int iterations) {
std::mt19937 rng(7);
std::uniform_int_distribution<int> length(0, 20);
std::uniform_int_distribution<int> value(0, 1000);
for (int i = 0; i < iterations; ++i) {
std::vector<Interval> input;
for (int j = 0; j < length(rng); ++j) {
int a = value(rng), b = value(rng);
if (a > b) std::swap(a, b);
input.push_back({a, b});
}
auto result = merge_intervals(input);
long input_cover = covered(input);
long result_cover = covered(result);
assert(result_cover == input_cover);
assert(invariant(result));
}
}
The two assertions are cheap and decisive. The first says the merge did not invent or lose range. The second says no overlapping intervals remain. If either one fails, reject the patch before looking at fixtures.
Phase 2: Mutation-Test Your Fixtures
A fixture that cannot fail is worthless. Mutation testing tells you which fixtures are actually watching the behavior that changed.
Take each fixture, apply one small change that should change the test outcome, and run the suite. If the suite still succeeds, the fixture is blind.
#!/usr/bin/env python3
import copy
import json
import os
import subprocess
import sys
import tempfile
def delete_key(data, key):
data.pop(key, None)
def increment(data, key='expected'):
data[key] = data[key] + 1
def make_mutations(data):
for key in data:
yield (f'delete_{key}', lambda d, k=key: delete_key(d, k))
if 'expected' in data:
yield ('increment_expected', lambda d: increment(d))
for fixture_path in sys.argv[1:]:
with open(fixture_path) as f:
original = json.load(f)
for name, apply in make_mutations(original):
mutated = copy.deepcopy(original)
apply(mutated)
with tempfile.NamedTemporaryFile('w', suffix='.json', delete=False) as tmp:
json.dump(mutated, tmp)
tmp_path = tmp.name
result = subprocess.run(['./test_runner', tmp_path],
capture_output=True)
os.unlink(tmp_path)
if result.returncode == 0:
print(f'WEAK {fixture_path}: survived mutation {name!r}')
Run it on the fixtures the patch touched. A WEAK line is an action item: add an expectation, narrow the fixture, or reject the patch because the oracle would not have caught the regression. The script is a template; apply mutations only to fields the patch's behavior affects, otherwise you will get false alarms.
Phase 3: Freeze Flaky Tests, But Only After 1 and 2
A flaky test is a signal with a high error rate. Retrying it hides the noise and teaches the loop to ignore red runs. Deleting it hides the question. Freezing it is the middle path.
One useful rule: freeze only after Phase 1 and Phase 2 pass. If property checks fail, a freeze masks the regression. If fixture mutations are weak, a freeze makes the blindness permanent.
| Phase | Result | Action |
|---|---|---|
| 1 property | fail | reject patch, do not freeze |
| 2 fixture mutation | weak | add fixture expectation, rerun |
| 1 and 2 pass | test is flaky | add to frozen list with a ticket |
| 1 and 2 pass | test is stable | keep it in the gate |
Here is a small pytest hook that reads a frozen_flakes.txt file and skips only those tests unless UNFREEZE is set.
import os
import pytest
FROZEN = set()
if os.path.exists('frozen_flakes.txt'):
FROZEN = set(line.strip() for line in open('frozen_flakes.txt'))
@pytest.hookimpl(tryfirst=True)
def pytest_collection_modifyitems(config, items):
unfreeze = os.getenv('UNFREEZE', '') == '1'
for item in items:
if item.name in FROZEN and not unfreeze:
item.add_marker(pytest.mark.skip(reason='frozen flaky test'))
This does not delete the test. You can still run it with UNFREEZE=1 pytest when you want to investigate.
Limitations
Property checks do not prove correctness. They prove that the invariants you wrote still hold. If the invariant is missing, the probe cannot see the bug.
Fixture mutation is noisy when fixtures contain timestamps, random seeds, or machine-specific paths. Normalize those fields before you run the script.
Free model access and a free server option are useful for this workflow, but they are not a production-grade CI guarantee. If this becomes a release gate, run it on a supervised runner and add a human to review the WEAK lines.
Who should not use this? Teams with zero tolerance for false alarms should replace blanket fixture mutations with a curated list. Teams that already have a strong property-based suite may only need Phase 2. Teams that just want a green badge should freeze everything; that is the opposite of this workflow.
Build the Probe In Order
Run the phases in order. Properties first. Fixture mutation second. Freeze third. If your agent patch survives all three, the evidence is real, not just green.
A suite is an oracle. An oracle that cannot fail is not a test. Make it fail before you let the patch pass.
Top comments (0)