The Test Failed, Passed, Failed. The Agent "Fixed" a Function That Was Already Correct.

The CI run failed at 02:14. The same test passed at 02:31. The agent, told to make the suite green, edited the wrong function and left the flake in place. This is not a model quality problem. It is a signal problem: a flaky test turns a deterministic gate into a coin flip, and a patch loop that adapts to the gate will adapt to the noise.

The fix is not to run the suite more times. It is a quarantine ledger that decides, before the agent sees a failure, whether that failure deserves a reaction.

Why a flake is worse than a failure

A deterministic failure gives an agent a stable target. A flaky test gives it three targets in a row: fail, pass, fail. The agent does not debug a coin flip; it pattern-matches. The cycle repeats:

1. The test fails. The agent edits the nearest suspicious code.
2. The rerun passes. The agent records the patch as successful.
3. The test fails again, because the original cause never went away.
4. The agent edits something else, and the diff grows.

The result is either a phantom fix — correct code modified for no reason — or learned suppression, where the agent discovers that deleting an assertion makes the suite green. Both are worse than the original red.

A single flaky test corrupts this loop. It does not need a high flake rate. It needs to fail once inside the window the agent is allowed to react to.

Freeze first, property-check later

The instinct is to add property-based tests to catch more edge cases. Do not add them to an unstable suite. A property test is still a test: it can fail on a port collision, a timestamp boundary, or an uninitialized fixture, and the agent will "fix" the output of a function that was never wrong.

The strategy has to be ordered:

1. Freeze every test that flakes inside the history window.
2. Run agent-visible patches only against the stable subset.
3. Add property checks to the stable subset, with pinned seeds and fixed examples.

The freeze comes before the properties. Otherwise, you are teaching the agent to chase ghosts.

The quarantine ledger

A freeze needs a record, not a comment. The pattern I use is a JSON ledger plus a tiny runner. flake_freeze.json:

{
  "version": 1,
  "history_window_runs": 10,
  "frozen": [
    {
      "node_id": "tests/test_clock.py::test_dst_boundary",
      "first_seen_run": "run-2026-08-29-019",
      "mismatches": 3,
      "last_mismatch_run": "run-2026-08-29-021",
      "reason": "timestamp boundary",
      "unfreeze_rule": "10 consecutive passes outside quarantine"
    }
  ]
}

The runner reads the ledger and removes frozen tests from the agent-visible run:

#!/usr/bin/env bash
set -euo pipefail
ledger="flake_freeze.json"
target="$1"

deselect=()
while IFS= read -r node; do
  deselect+=(--deselect="$node")
done < <(jq -r '.frozen[].node_id' "$ledger")

pytest "$target" "${deselect[@]}" -q

Quarantine is not a skip. Frozen tests are absent from the run result, not hidden behind pytest.mark.skip. A visible absence is easier to audit than an invisible pass.

The five-step freeze workflow

1. Detect. Any failure that cannot be reproduced on the same commit with the same command is a flake candidate.
2. Probe. Rerun the failing test five times on the identical commit before the agent sees it. One pass in five reruns is enough to freeze.
3. Freeze. Move the test into the ledger with a reason and an unfreeze rule.
4. Patch. Let the agent touch only the stable subset.
5. Prune. Every 20 runs, review the ledger. A frozen test that stays frozen for the full window becomes a landfill.

The probe script is deliberately stupid:

probe() {
  local node="$1"
  local runs="${2:-5}"
  local passes=0
  for i in $(seq 1 "$runs"); do
    if pytest "$node" -q >/dev/null 2>&1; then
      passes=$((passes + 1))
    fi
  done
  echo "$passes/$runs"
}
probe "tests/test_clock.py::test_dst_boundary"
# 4/5 → freeze, never show this failure to the agent

The critical detail: the probe reruns the same commit. A rerun after a patch is a different experiment and proves nothing about flakiness.

Property checks that survive the freeze

Once the suite is stable, property checks add the signal that fixtures cannot: invariants. A minimal Hypothesis example:

from hypothesis import given, settings, strategies as st

@settings(deadline=None, derandomize=True)
@given(st.integers(), st.integers())
def test_add_commutes(a, b):
    assert add(a, b) == add(b, a)

derandomize=True makes the input sequence reproducible on the same version, so a failure is attributable to the code and not to a generator draw. Add fixed examples next to the generated ones. A property check that only passes on a frozen test is a fixture story, not a property story.

The triage table

Observation	Action	Condition
Fails once, passes on rerun	Freeze	any mismatch in window
Same traceback in 3+ of 5 runs	Fix	same error, same environment
Different error each run	Freeze, assign owner	timeout, port, ordering
Frozen for 20+ runs	Prune or delete	no owner, no reproducer

The table removes judgment from the hot path. You decide the rules once, and the loop applies them without asking the agent for an opinion.

Who should not use this

Small suites where a human watches every run do not need a freeze; they need the test fixed. Suites that are already deterministic — no network, no wall clock, no shared state — have no flakes to quarantine. Teams without run history cannot maintain the ledger; a ledger without history is a to-do list with extra JSON.

What changes when retries are cheap

The expensive resource in an agent loop is not the API call. It is the human attention spent deciding whether a failure is real. If the patch loop runs on MonkeyCode's free model access and its free server option, another patch attempt costs nothing extra in dollars — so you can afford to probe every suspicious failure five times. Cheap retries make flakes more visible, not less. That is exactly why the freeze matters more, not less.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.