A Frozen Test Still Needs a Failure Path: Canaries and Exit Criteria for Quarantine Lanes

The short version

Freezing a flaky test does not remove it from your suite. It copies it into a second suite — unblocked, scheduled nightly, and visited by nobody. That second suite has its own failure modes, and the worst one is silent: it stops reporting failures while still reporting green.

So treat the quarantine lane like production code. Give it a canary that must always fail, give every freeze a machine-checkable exit condition, and make a broken lane louder than a flaky test.

The artifact below is three files: a canary test, a collection guard, and a stdlib-only verdict script. Everything runs on plain pytest + Python 3.11 with no extra services.

Why a freeze can hide a real failure

Four concrete ways a freeze eats signal, all of which look like a healthy build:

Failure mode	What you see	What actually happened
Skip marker leaks into the blocking job	Blocking CI green	A frozen test is skipped everywhere, including the lane meant to replay it
Quarantine job failures downgraded to warnings	Nightly green	continue-on-error: true swallowed real assertion failures
Test renamed or moved	Ledger entry just stops counting	The freeze is now orphaned; nobody is replaying anything
Collection error in the quarantine file	Zero tests collected, exit 5 ignored	The lane ran for weeks with nothing in it

None of these are flakiness problems. They are observability problems on the freeze mechanism itself.

Step 1 — Add a canary that must fail

A canary is a test whose only job is to fail. If the lane ever reports it as passing, skipped, or missing, the lane is broken and the job should go red immediately.

# tests/quarantine/test_canary.py
# Intentionally always fails. Never fix this test, never delete it.
def test_quarantine_lane_canary():
    raise AssertionError(
        "CANARY: the quarantine lane reported a pass. "
        "The lane is no longer a trustworthy replay channel."
    )

A permanently red test feels wrong, which is exactly why it works. It converts "the lane is quiet" into "the lane is provably executing and provably reporting failures."

Guard the other direction too: frozen tests must never be collected by the blocking job. A collect-only assertion catches skip-marker leakage without running anything.

# tests/quarantine/conftest.py
import json
import pathlib
import pytest

CANARY = "test_quarantine_lane_canary"
LEDGER = pathlib.Path(__file__).parents[2] / "freezes.json"


def pytest_collection_modifyitems(session, config, items):
    names = {item.name for item in items}
    if CANARY not in names:
        raise pytest.UsageError(f"{CANARY} not collected; refusing to run the lane")


def pytest_collection_finish(session):
    frozen = {e["nodeid"] for e in json.loads(LEDGER.read_text())["entries"]}
    leaked = sorted(
        i.nodeid for i in session.items
        if any(i.nodeid.endswith(f.split("::")[-1]) for f in frozen)
        and i.nodeid != "tests/quarantine/test_canary.py::" + CANARY
    )
    if leaked:
        raise pytest.UsageError(f"frozen tests leaked into this lane: {leaked[:3]}")

Run the blocking job with pytest --ignore=tests/quarantine, and add one extra CI step that asserts the frozen node IDs are absent from --collect-only output. Two lines of grep; it catches the leak that costs you the most.

Step 2 — Give every freeze an exit condition

A freeze without a numeric exit condition is a permanent deletion with extra steps. The ledger is a small JSON file (stdlib-only parsing, reviewable in a PR diff):

{
  "ledger_version": 1,
  "entries": [
    {
      "id": "FZ-014",
      "nodeid": "tests/api/test_upload.py::test_retry_after_409",
      "frozen_at": "2026-10-30",
      "owner": "finley",
      "reason": "time-window assumption in the retry helper; 7/40 failures on the same commit",
      "exit": { "kind": "zero_failure_runs", "n_required": 60 },
      "runs": 0,
      "failures": 0,
      "last_run": null
    }
  ]
}

n_required: 60 is a policy number, not a proof. With zero failures in 60 scheduled runs, the rule of three puts the 95% upper bound on the true flake rate near 5% (3/60) — see Hanley & Lippman-Hand, JAMA 1983, doi:10.1001/jama.1983.03330370054030. Pick the number you can defend in review; 20 runs (~15%) is too weak for anything that blocks merges.

One rule makes the counter honest: any failure resets runs to 0. A test that fails on run 58 has not earned 58 runs of trust.

Step 3 — Compute the verdict from JUnit XML

No dashboards. The lane emits quarantine.xml, the script reads it, and the exit code drives CI.

#!/usr/bin/env python3
# scripts/freeze_verdict.py
"""Replay-lane verdict. Exit: 0 ok, 2 canary passed, 3 canary missing, 4 bad ledger."""
import argparse, json, sys, xml.etree.ElementTree as ET

CANARY = "test_quarantine_lane_canary"
RULE_OF_THREE = 3.0


def load_status(path):
    root = ET.parse(path).getroot()
    status, nodeids = {}, {}
    for case in root.iter("testcase"):
        name = case.get("name", "")
        nodeid = f"{case.get('classname', '')}::{name}"
        if case.find("failure") is not None or case.find("error") is not None:
            state = "fail"
        elif case.find("skipped") is not None:
            state = "skip"
        else:
            state = "pass"
        status[name] = state
        nodeids[nodeid] = state
    return status, nodeids


def verdict(entry, nodeids):
    short = entry["nodeid"].split("::")[-1]
    state = nodeids.get(entry["nodeid"]) or nodeids.get(f"::{short}") \
        or next((v for k, v in nodeids.items() if k.endswith("::" + short)), None)
    if state is None:
        return "stale", "not collected; freeze may be orphaned by a rename"
    if state == "fail":
        entry["failures"] += 1
        entry["runs"] = 0  # reset the trust counter
        return "keep", "failed again; counter reset"
    if state == "skip":
        return "keep", "skipped, not executed; counter unchanged"
    entry["runs"] += 1
    n = entry["runs"]
    if n >= entry["exit"]["n_required"] and entry["failures"] == 0:
        return "unfreeze", f"{n} clean replays, upper bound ~{RULE_OF_THREE / n:.1%}"
    return "keep", f"{n}/{entry['exit']['n_required']} clean replays"


def main():
    ap = argparse.ArgumentParser()
    ap.add_argument("--junit", required=True)
    ap.add_argument("--ledger", required=True)
    ap.add_argument("--write", action="store_true")
    args = ap.parse_args()

    status, nodeids = load_status(args.junit)
    if CANARY not in status:
        print("CANARY MISSING: lane collected nothing or excluded the canary")
        return 3
    if status[CANARY] != "fail":
        print(f"CANARY {status[CANARY]}: lane is not reporting failures")
        return 2

    ledger = json.load(open(args.ledger))
    for entry in ledger["entries"]:
        action, note = verdict(entry, nodeids)
        print(f"{entry['id']:8} {action:9} {note}")
    if args.write:
        json.dump(ledger, open(args.ledger, "w"), indent=2)
        open(args.ledger, "a").write("\n")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Wire it into a nightly job where the replay step is allowed to fail but the verdict step is not:

name: quarantine-nightly
on:
  schedule: [{ cron: "17 3 * * *" }]
  workflow_dispatch:
jobs:
  replay:
    runs-on: [self-hosted, replay]
    steps:
      - uses: actions/checkout@v4
      - run: pip install -r requirements.txt
      - run: pytest tests/quarantine --junitxml=quarantine.xml
        continue-on-error: true          # canary is supposed to fail here
      - run: python scripts/freeze_verdict.py --junit quarantine.xml --ledger freezes.json --write
      - uses: actions/upload-artifact@v4
        if: always()
        with: { name: quarantine-xml, path: quarantine.xml }

Exit 2 or 3 fails the job. That is the whole alerting model: a broken lane pages; a still-flaky frozen test does not.

Where MonkeyCode fits, and where it does not

The nightly replay is the cheapest job in the pipeline and also the one nobody wants to spend metered minutes on. This workflow is designed to run the replay step on MonkeyCode's free server option, with the verdict script executed there too, so a frozen test costs you a queue slot instead of blocking-job budget.

The second piece is the ledger entry, which is the part humans skip. Free model access is a reasonable fit for drafting reason and a suggested n_required from the CI log slice plus the patch diff — as a proposal in a pull request, reviewed by the owner named in the entry, never auto-merged.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What I am not claiming: I have not published numbers for quotas, model list, machine class, or queue latency, and availability of the free model access and free server option is operator-supplied rather than something I independently benchmarked. Treat both as conveniences, not guarantees, and keep the verdict script runnable on plain CI as a fallback.

Limitations, stated plainly

• The rule of three assumes exchangeable runs. Nightly CI is not i.i.d.: same machine, same cache, same clock skew. 60 clean nights on one runner is weaker evidence than 60 clean nights across rotated seeds and at least two runners. Rotate the seed on every replay; if your seed matrix already exists, feed it here instead of inventing a second one.
• The canary detects reporting breakage, not hiding. A canary proves the lane reports a failure. It cannot prove the frozen test itself still exercises the same code path the patch touched.
• Renames look like health. The script flags orphans as stale, but only a human confirms whether the test moved or was quietly deleted.
• The counter is conservative by design. One failure at run 59 resets you to zero. That is intentional friction: unfreezing is a claim about a distribution, not about a lucky week.

Who should not use this

Skip it if you have fewer than about ten frozen tests — a spreadsheet and a weekly look is cheaper than a ledger, a script, and a nightly job. Skip it if nobody is willing to be named as owner; an unowned freeze is a deletion and should be argued as one. And skip it if your replay runs less than daily, because a 60-run exit condition becomes a two-month wait that everyone stops reading.

If your freeze count is zero, you do not need any of this yet. Bookmark it for the week someone says "just skip it and move on" — which is the moment this becomes three files instead of a habit.

Rollout checklist

1. Add tests/quarantine/test_canary.py and the collection guard; run the lane once and confirm exit code 1 with the canary's message.
2. Create freezes.json with one entry per frozen test; require owner and exit.n_required to be non-null in review.
3. Add the nightly job; confirm a deliberately broken canary (rename it) produces exit 3.
4. Break the reporting on purpose — mark the canary @pytest.mark.skip — and confirm exit 2.
5. Only then migrate existing freezes into the ledger, one PR per entry, with the flake evidence pasted in reason.

The order matters. Instrument the lane before you put anything valuable in it.

Comments

[Alex Shev]

Quarantining a flaky test without an exit path turns temporary uncertainty into permanent blind spot. The canary idea gives the lane a signal, but the ownership and expiry are just as important: name the owner, define the expected failure pattern, and set a date when the test must be repaired, rewritten, or removed. That keeps the quarantine honest.