Freeze the Flaky, Check the Properties: A Three-Layer Test Strategy for Agent Patches

A flaky test is not a test. It is a slot machine that decides whether an agent patch gets accepted.

I keep seeing the same loop: the agent sees one failing test, rewrites a function until that test passes, and the suite goes green. Ten minutes later the same test fails because the order of a map changed. The agent learned nothing about behavior; it learned how to satisfy an unstable assertion.

The fix is not a better model. It is a test strategy with three layers:

1. Freeze the flaky tests.
2. Fix the fixture inputs.
3. Check properties, not exact outputs.

MonkeyCode's free model access and free server option lower the cost of running this loop long enough to collect evidence. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. Freeze the flaky tests

A flaky test is noise in the agent's reward function. Every rerun gives it a contradictory label. Before you let an agent touch production code, find those labels and freeze them.

Run the suspicious suite ten times and aggregate failures:

for run in $(seq 1 10); do
  ./build/unit_tests --reporter compact 2>&1 \
    | sed -n 's/.*FAILED: //p'
done | sort | uniq -c | sort -rn | head -20

Any test that fails between 1 and 9 times is a freeze candidate. Add it to frozen.yaml:

frozen:
  - test: "EventParserTest.InvalidTimestamps"
    failures_in_10: 4
    frozen_on: "2026-08-30"
    reason: "depends on wall-clock time from a mocked timer"

Your runner skips frozen tests, and your review checklist treats them as open debt. The agent no longer sees the flaky failure as an instruction.

2. Fix the fixture inputs

If two patch attempts run on different inputs, you cannot compare their behavior. So pin the inputs before starting the agent.

Create an immutable fixture directory with timestamps, random seeds, and file order fixed:

FIXTURE_DIR="fixtures/frozen-2026-08-30"
HYPOTHESIS_PROFILE=max
RANDOM_SEED=31337
TZ=UTC
LANG=C

./baseline_binary "$FIXTURE_DIR" > baseline.out
./patched_binary  "$FIXTURE_DIR" > patched.out

Use the exact same fixture set in every experiment. If the database, clock, or locale differ between runs, you are not testing the patch; you are testing the environment.

3. Check properties instead of exact outputs

Exact-output tests are easy for an agent to overfit. You tell it "assert x equals 3", and the agent eventually fabricates something that makes that one assertion true. Property checks describe an invariant over a space of inputs, so the agent has to preserve behavior, not a literal value.

A minimal Python example that exercises the same idea on a small event sorter:

# sort_events.py
from hypothesis import given, strategies as st

def sort_events(pairs):
    # agent patch replaced this with an unstable sort
    return sorted(pairs, key=lambda p: p[0])

@given(st.lists(st.tuples(st.integers(), st.integers())))
def test_timestamps_are_non_decreasing(events):
    out = sort_events(events)
    for (ta, _), (tb, _) in zip(out, out[1:]):
        assert ta <= tb

@given(st.lists(st.tuples(st.integers(), st.integers())))
def test_no_event_is_lost(events):
    out = sort_events(events)
    assert sorted(out) == sorted(events)

if __name__ == "__main__":
    test_timestamps_are_non_decreasing()
    test_no_event_is_lost()
    print("properties ok")

The first property catches "I changed the order of events"; the second catches "I dropped events." Neither property tells the agent what to assert. They tell it what must remain true.

How the three layers interact

The freeze is a debt tag, not a permanent excuse. When you freeze a test, add one property check that preserves what that test was guarding. That way the agent can pass the old flaky test, but it still has to respect the invariant.

Fixtures tie the property checks to a reproducible world: same inputs, same seeds, same locale. Without them, two runs of the same property check can diverge for environmental reasons and become flaky again.

Limitations and who should skip this

This strategy assumes the system under test is deterministic. If your code depends on wall-clock threads, network calls, or random hardware behavior, freeze the test, mock the boundary, and only then apply property checks.

Do not use this strategy as a license to delete failing tests. A frozen test must have a reason, an owner, and a property-check replacement. If you cannot name the invariant, the flaky test is telling you about a design problem, not a test problem.

Also skip this if you need pixel-level validation or human conversation flows. Property checks describe state, not perceived quality; those domains need a different oracle.

Start with the freeze. Count the flaky failures, pin the fixtures, write one property. The agent will stop optimizing for the test runner and start optimizing for behavior.