The Flaky Freeze Needs an Expiry Date: A Gate Runner for Agent Patches

A test freeze is the only gate that converts failure into silence. Done naively, it is a permanent skip list that blinds your suite to the exact regression the agent patch introduced. Done correctly, it is a quarantine with an expiry date, an owner, and a re-enrollment check. This article is the operational layer under the three-gate strategy: the runner, the ledger, and the ordering rules that keep the freeze honest.

Why a freeze rots

A frozen test does not disappear. It waits. Every week without re-enrollment makes the entry harder to revisit, because the context evaporates: the patch merged, the branch is gone, the flake is someone else's memory.

A freeze without an expiry is a memory leak in your test suite. The allocation never gets collected, and the signal it held is gone forever. The fix is to treat the ledger as debt. Every entry needs four fields: an owner, an expiry, a flake count, and a re-enrollment procedure. If any field is missing, the runner refuses to freeze.

The quarantine ledger

The ledger is a single JSON file. It is the source of truth for every test that is not allowed to fail.

{
  "test_id": "ring_buffer/property_ops",
  "frozen_at": "2026-08-27T09:00:00Z",
  "frozen_until": "2026-09-03T09:00:00Z",
  "flake_count": 4,
  "owner": "finley",
  "reason": "timing-sensitive destructor under TSan",
  "re_enrollments": 0
}

Three rules keep this honest:

1. frozen_until is mandatory. No expiry, no freeze.
2. flake_count grows on every failed re-enrollment.
3. After three re-enrollments, the entry escalates. The test is fixed or deleted; it cannot stay frozen forever.

The gate runner

The runner executes three gates in a fixed order: properties, fixtures, suite. The first two never offer a freeze. Only the suite gate does, and only after retries with backoff.

#!/usr/bin/env python3
"""Gate runner: properties -> fixtures -> suite -> quarantine with expiry."""
import json, os, subprocess, sys, time
from datetime import datetime, timedelta, timezone

LEDGER = "quarantine.json"
RETRIES = 3
FREEZE_DAYS = 7
MAX_RE_ENROLLMENTS = 3

TESTS = [
    ("ring_buffer/property_ops", "./build/ring_buffer_test"),
    ("parser/fixture_unicode", "./build/parser_test --fixture unicode"),
]

def now():
    return datetime.now(timezone.utc).isoformat()

def run(cmd, timeout=120):
    try:
        subprocess.run(cmd, shell=True, timeout=timeout, check=True)
        return True
    except (subprocess.CalledProcessError, subprocess.TimeoutExpired):
        return False

def load_ledger():
    if not os.path.exists(LEDGER):
        return {"entries": {}}
    with open(LEDGER) as f:
        return json.load(f)

def save_ledger(ledger):
    with open(LEDGER, "w") as f:
        json.dump(ledger, f, indent=2)

def freeze(ledger, test_id, cmd, reason):
    entry = ledger["entries"].get(test_id, {})
    entry["flake_count"] = entry.get("flake_count", 0) + 1
    entry["frozen_at"] = now()
    entry["frozen_until"] = (
        datetime.now(timezone.utc) + timedelta(days=FREEZE_DAYS)
    ).isoformat()
    entry["owner"] = os.environ.get("GATE_OWNER", "unassigned")
    entry["reason"] = reason
    entry["cmd"] = cmd
    entry["re_enrollments"] = entry.get("re_enrollments", 0)
    ledger["entries"][test_id] = entry
    save_ledger(ledger)
    print(f"FROZEN {test_id} until {entry['frozen_until']}")

def re_enroll_due(ledger):
    return [tid for tid, e in ledger["entries"].items()
            if e["frozen_until"] <= now()]

def main():
    ledger = load_ledger()

    # Gate 1: properties. A failure is a real bug. Never frozen.
    if not run("./property_check $RANDOM"):
        sys.exit("property gate failed: patch blocked")

    # Gate 2: fixtures. A failure is a regression or a spec change.
    if not run("ctest --test-dir build -R fixtures"):
        sys.exit("fixture gate failed: patch blocked")

    # Gate 3: suite. Retries with backoff, then quarantine.
    for test_id, cmd in TESTS:
        if run(cmd):
            continue
        for attempt in range(2, RETRIES + 1):
            time.sleep(2 * attempt)
            if run(cmd):
                break
        else:
            freeze(ledger, test_id, cmd, "failed all retries after agent patch")

    # Re-enroll expired freezes.
    for test_id in re_enroll_due(ledger):
        entry = ledger["entries"][test_id]
        if run(entry["cmd"]):
            del ledger["entries"][test_id]
            print(f"RELEASED {test_id}")
        else:
            entry["re_enrollments"] += 1
            entry["frozen_until"] = (
                datetime.now(timezone.utc) + timedelta(days=FREEZE_DAYS)
            ).isoformat()
            if entry["re_enrollments"] >= MAX_RE_ENROLLMENTS:
                print(f"ESCALATE {test_id}: fix or delete")
        save_ledger(ledger)

if __name__ == "__main__":
    main()

The property gate deserves its own artifact. A property harness generates inputs the fixtures never saw. Here is a minimal one for a ring buffer:

// property_check.cpp — invariant harness for a ring buffer.
// Build: g++ -O2 -o property_check property_check.cpp
// Run:   ./property_check 42
#include <cstdint>
#include <cstdlib>
#include <deque>
#include <iostream>
#include <random>
#include <vector>

template <typename T>
struct RingBuffer {
  std::vector<T> data;
  size_t head = 0, count = 0;
  explicit RingBuffer(size_t cap) : data(cap) {}
  bool push(T v) {
    if (count == data.size()) return false;
    data[(head + count) % data.size()] = v;
    ++count;
    return true;
  }
  bool pop(T& out) {
    if (count == 0) return false;
    out = data[head];
    head = (head + 1) % data.size();
    --count;
    return true;
  }
};

int main(int argc, char** argv) {
  uint64_t seed = argc > 1 ? std::strtoull(argv[1], nullptr, 10) : 1;
  std::mt19937_64 rng(seed);
  RingBuffer<int> rb(8);
  std::deque<int> pushed;
  for (int i = 0; i < 100000; ++i) {
    if (rng() % 2 == 0) {
      int v = static_cast<int>(rng());
      if (rb.push(v)) pushed.push_back(v);
    } else {
      int out = 0;
      if (rb.pop(out)) {
        if (pushed.empty() || out != pushed.front()) {
          std::cerr << "INVARIANT VIOLATION at step " << i << "\n";
          return 1;
        }
        pushed.pop_front();
      }
    }
    if (rb.count > rb.data.size()) {
      std::cerr << "INVARIANT VIOLATION: count > capacity\n";
      return 1;
    }
  }
  std::cout << "properties passed (seed " << seed << ")\n";
  return 0;
}

Five steps, in order

1. Run the property harness with a random seed. A failure blocks the patch. No freeze is offered, because a violated invariant is a bug, not a flake.
2. Run the fixture suite. A failure means a regression or a deliberate spec change; the patch author must explain it in the PR body.
3. Run the full suite. A failing test gets up to three attempts with backoff.
4. A test that fails all attempts becomes a freeze candidate. The runner writes a ledger entry with an expiry and an owner. It does not skip the test; it defers it.
5. When the expiry passes, the runner re-enrolls the test. Pass means released. Fail means the flake count grows, and after three re-enrollments the test is escalated to fix-or-delete.

The decision table

Symptom	Verdict	Freeze allowed?
Property check fails	Real invariant violation	No
Fixture fails	Regression or intentional spec change	No
Suite test fails once, passes on retry	Flake candidate	Only with expiry + owner
Suite test fails all retries after an agent patch	Freeze candidate	Yes, 7-day expiry
Test flakes again after 3 re-enrollments	Broken under current harness	No — fix or delete

The ordering is not cosmetic. Properties are the cheapest and broadest gate, so they run first and cost seconds. Fixtures are precise because each one encodes a past failure. The full suite is the most expensive and the most likely to produce flakes, so it runs last and is the only gate that can be bypassed — and only with a timestamp attached.

Where the free model and free server fit

In this workflow, the patch and the property harness come from a free model session in MonkeyCode, and the gate runner executes on MonkeyCode's free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The separation is the point: the model writes the code, the ledger decides its fate. The model never gets to argue that its own patch is flaky; the runner counts the retries, and the ledger records the debt.

Who should not use this

If your suite is deterministic and fast, the ledger is bureaucracy. A quarantine mechanism for a suite that never flakes is overhead you will maintain for no signal.

If nobody owns the expiry dates, the quarantine rots back into a skip list. The ledger is only as honest as the owner field.

If you do not know the invariant, the property gate is theater. A property harness that asserts nothing will pass forever and prove nothing.

And never freeze a property check or a fixture. Those failures are real signals. Freezing them is how a bug becomes a feature.

The free server option has its own availability and limits. Check the current documentation before building a pipeline on it.

The freeze is a promise

A test freeze is not a decision. It is a promise with a deadline. The deadline forces the re-enrollment, and the re-enrollment forces someone to look at the test again. Steal the ledger schema: a dozen lines of JSON will save you from a permanently blind suite.