Freeze Flaky Tests, Don't Retry Them: A Three-Layer Test Strategy for Agent Patches

A test suite that keeps passing while a regression slips through is not a safety net. It is a supply chain for false confidence. With agent-generated patches the risk sharpens: the same commit can edit the implementation and its tests, so "all tests pass" no longer means the behavior survived.

The strategy below is the one I now apply to every incoming agent patch before it reaches review. It has three layers: property checks that test the spec, frozen fixtures that pin known behavior, and a freeze list that stops flaky tests from blocking merge queues — or, worse, encouraging blind retries.

Layer 1 — Property checks against the spec, not the implementation

A unit test written next to a fix shares the blind spot of the fix. Property checks generate inputs the patch author never imagined. They fail on behavior, not on line-by-line diff.

Consider a small C++ range parser. The spec is: input like "1-3,5" parses to a sorted, de-duplicated vector of integers; format_range is its inverse.

An agent was asked to optimize the parser. The diff looked innocent:

- std::set<int> values;
+ std::vector<int> values; // faster append path

Every handwritten fixture passed. The canonical-order property caught the regression. A minimal, illustrative version of the property layer looks like this:

// property_layer.cpp — deterministic seeds only
TEST_CASE("parse then format is the identity on sorted sets", "[property]") {
  std::mt19937 rng(20260829);          // fixed seed, not time()
  for (int i = 0; i < 512; ++i) {
    auto input = random_range_string(rng);
    auto parsed = parse_range(input);
    REQUIRE(parsed.valid());

    INFO("input: " << input);
    REQUIRE(std::is_sorted(parsed.values.begin(), parsed.values.end()));
    REQUIRE(parse_range(format_range(parsed)).values == parsed.values);
  }
}

is_sorted failed on the first run. The agent's patch dropped ordering in the name of speed, and no hand-written test happened to cover "3,1,2".

Three rules keep this layer honest:

1. Fixed seeds. A property test that uses the system clock is a flake generator, not a test.
2. Keep the generator dumb. Random inputs from a small alphabet; stress comes from count, not from clever generators.
3. Run the same seeds in CI and locally. A failing seed is then reproducible without a trace.

Layer 2 — Frozen fixtures pin behavior down

Property checks find unknown unknowns. Fixtures catch authorized changes. I keep a fixture file for every component that agents are allowed to touch:

# fixtures/range_parser.yaml — do not edit by hand without a behavior note
- input: "1-3"
  values: [1, 2, 3]
  canonical: "1-3"
- input: "3,1,2"
  values: [1, 2, 3]
  canonical: "1-3"
- input: "1-3,5"
  values: [1, 2, 3, 5]
  canonical: "1-3,5"
- input: "1,"
  valid: false

A patch may change code or tests, but not fixtures. When a fixture begins to fail, the triage question is sharp: is this an intended spec change? If yes, a human rewrites the fixture and the reason lives in the commit. If no, the patch is a regression.

Two fixture rules are enforced mechanically:

1. Golden values, not comparisons. The expected list is stored, not recomputed. Recomputed goldens give the agent permission to multiply bugs.
2. Fixture updates require a second artifact. A changed fixture without a linked behavior note is treated as a test rewrite, and test rewrites are flagged in review.

Layer 3 — Freeze flaky tests instead of retrying them

Retrying a flaky test trains the team to distrust red builds. The fourth failure from the same test is usually someone else's bug. Ask the runner to freeze those tests instead:

# frozen_tests.txt — read by the runner, not by humans
range_parser_stress_alloc      unstable_since=2026-08-29 frozen_runs=3 note="uses TLS state"
range_parser_parallel_order    unstable_since=2026-08-29 frozen_runs=1 note="shared fixture mutated"

The runner excludes frozen tests from merge gates, but still executes them and records two counters: consecutive failures and pass rate. A test returns to the gate only when the cause is written in the note field. Cold-turkey disabling is banned.

This converts "flaky" from a personality trait of a test into a tracked incident. No retries, no "let's see tomorrow", no shadow retry loop in CI.

The triage decision table

Gate result	Verdict	Action
Property test fails	Regression	Reject; add the failing seed to fixtures
Frozen fixture fails	Behavior change or regression	Require a human-written change note
Frozen-list test fails	Known flake	No block; increment the incident counter
All green	Weak signal	Still review the diff semantics

The free-model classifier pass

On patch receipt, a cheap classification step decides which of the three layers to run first. In my setup, that label assignment runs through MonkeyCode's free model access — the runner gets a short PROPERTY / FIXTURE / FLAKE / SKIP label from the patch summary and the test-log tail. The free server option keeps that job off my laptop and avoids rate-limit surprises on long queues. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The classifier does not merge and does not testify; it only routes the patch to the correct gate. A wrong label is harmless because the layered tests still run. The token cost is a rounding error compared with a full build.

Who should not use this

Skip this strategy if your agent patches never touch behavior, or if your tests are so slow that 512 property seeds explode the pipeline. The property layer assumes a pure-ish function and a deterministic input generator. A UI-heavy codebase needs screen-level state fixtures instead. And if your team still fixes flaky tests by retrying, the freeze list is a process change, not a tool change — install the habit before the list.

The three layers overlap less than they look. Properties catch what fixtures miss, fixtures catch authorized changes, and the freeze list keeps the signal honest. None of it requires an agent. That is the point: the same gate that triages agent patches triages any patch.