Bucket the Red: Invariant Miss, Fixture Drift, or Noise

Agent patches fail in three ways that look identical on a red CI job. One is a broken invariant. One is a drifted fixture. One is unreproducible noise. Spending a property, a lock, or a freeze on the wrong class does not make the suite stricter. It teaches the next patch how to look green.

Classify the failure first. Spend the test budget second. Coverage is not a budget. A classified miss is.

Coverage counts the wrong thing

Line coverage records that the patch executed a path. It does not record whether an oracle would have failed if that path were wrong. Agent-authored asserts inflate the first number. They often leave the second number at zero.

A workable weekly budget is how many classified failures you can convert into oracles. Three classes. Three spend types. Mix them and the gate starts rotting on the next merge.

This is adjacent to a louder industry argument: models can satisfy the tests used to measure them. The engineering response is not a longer prompt. It is a failure taxonomy the agent is not allowed to edit.

Three buckets, three spends

Invariant miss. Same input, same seed, same fixture hash. Fail, fail, fail on a remote rerun. Behavior changed. Spend a property over a domain, not one equality on one example.

Fixture drift. The invariant still holds. A recorded payload, header, or timestamp moved. Spend a lock or a deliberate refresh with a human diff. Do not freeze.

Noise. Outcomes flip across reruns, or the message cites wall time, a local path, or an unpinned entropy source. Spend replay. A freeze is last, dated, and owned. Never the first control.

If you cannot tell which bucket you are in, stop writing tests. Fix the runner.

Decision table

After 3 remote reruns	Fixture hash	Seed	Bucket	Spend
fail, fail, fail	unchanged	pinned	invariant miss	Strengthen a property. Reject a single agent-written equality.
fail, fail, fail	changed	pinned	fixture drift	Diff the fixture. Lock or refresh. No freeze.
mixed pass/fail	unchanged	pinned	noise	Capture the input. Fix isolation. Freeze only with an expiry.
mixed pass/fail	unchanged	missing	unclassified	Pin the seed. Rerun. Write nothing yet.
green only on tests the patch added	n/a	n/a	untrusted green	Ignore those tests for merge.

The last row is a constraint, not a slogan. Tests the patch authored describe the patch's story. They do not describe the system.

Toy subject under test

The classifier below is subject-agnostic. The example domain is a small bucketizer so the properties are readable. Label it as a worked example, not a production metric.

# bucketizer.py
from typing import List, Sequence


def clamp_and_bucket(values: Sequence[float], edges: Sequence[float]) -> List[int]:
    """Map each value to the index of the first edge that is >= value.

    Values above the last edge clamp to len(edges) - 1.
    Empty edges are rejected. Edges must be strictly increasing.
    """
    if not edges:
        raise ValueError("edges must be non-empty")
    for i in range(1, len(edges)):
        if edges[i] <= edges[i - 1]:
            raise ValueError("edges must be strictly increasing")
    out: List[int] = []
    last = len(edges) - 1
    for v in values:
        placed = last
        for i, e in enumerate(edges):
            if v <= e:
                placed = i
                break
        out.append(placed)
    return out

A property for this function talks about length, range, and monotonicity. A fixture talks about a recorded edges.json. Noise talks about neither.

Step 1 — Emit a comparable run record

Do not parse terminal colors. Write one JSON object per rerun. Pin the seed in the environment, not in a comment.

# emit_run.py
import hashlib, json, os, traceback, unittest
from pathlib import Path
from bucketizer import clamp_and_bucket

FIXTURE = Path(__file__).with_name("edges.json")
SEED = os.environ.get("TEST_SEED", "")


def fixture_hash() -> str:
    return hashlib.sha256(FIXTURE.read_bytes()).hexdigest()[:16]


class BucketTests(unittest.TestCase):
    def test_recorded_edges_length(self):
        edges = json.loads(FIXTURE.read_text())
        got = clamp_and_bucket([0.0, 1.5, 9.9], edges)
        self.assertEqual(len(got), 3)

    def test_monotonic_on_sorted_probe(self):
        edges = json.loads(FIXTURE.read_text())
        probe = [-1.0, 0.0, 0.5, 2.0, 50.0]
        got = clamp_and_bucket(probe, edges)
        self.assertEqual(got, sorted(got))


def run_and_emit(path: str, run_id: int) -> None:
    suite = unittest.defaultTestLoader.loadTestsFromTestCase(BucketTests)
    results = []

    def consume(test):
        nodeid = test.id()
        try:
            test.debug()
            results.append(
                {
                    "nodeid": nodeid,
                    "outcome": "passed",
                    "seed": SEED,
                    "fixture_hash": fixture_hash(),
                    "message": "",
                }
            )
        except Exception as exc:
            results.append(
                {
                    "nodeid": nodeid,
                    "outcome": "failed",
                    "seed": SEED,
                    "fixture_hash": fixture_hash(),
                    "message": f"{exc}\n{traceback.format_exc()}",
                }
            )

    for t in suite:
        consume(t)
    Path(path).write_text(json.dumps({"run_id": run_id, "tests": results}, indent=2))


if __name__ == "__main__":
    import sys
    run_and_emit(sys.argv[1], int(sys.argv[2]))

Sample fixture:

[0.0, 1.0, 2.0, 5.0]

Step 2 — Rerun off the laptop, three times

Laptop load, laptop clocks, and laptop DNS turn invariant misses into noise. The classification is only as good as the runner. Three processes. Same seed. Same fixture tree. Different working directories if the suite writes temp files.

# rerun.sh — proposal: run this on a remote job, not on a developer laptop
set -euo pipefail
export TEST_SEED="${TEST_SEED:-agent-patch-17}"
mkdir -p runs
python emit_run.py runs/run-1.json 1
python emit_run.py runs/run-2.json 2
python emit_run.py runs/run-3.json 3
python classify_failures.py runs/run-1.json runs/run-2.json runs/run-3.json

Remote execution is the point. A green laptop and a red CI box are two different oracles pretending to be one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. If the team does not already have a remote runner, MonkeyCode's free server option is enough to execute rerun.sh away from the laptop. Free model access is relevant later, when an invariant miss needs a drafted property. Neither replaces the classifier.

Step 3 — Classify before anyone writes a test

# classify_failures.py
import json, sys
from collections import defaultdict
from typing import Dict, List, Tuple

Bucket = str


def load(path: str) -> dict:
    return json.loads(open(path).read())


def group(runs: List[dict]) -> Dict[str, List[dict]]:
    by_node = defaultdict(list)
    for run in runs:
        for t in run["tests"]:
            by_node[t["nodeid"]].append(t)
    return by_node


def classify(records: List[dict]) -> Tuple[Bucket, str]:
    outcomes = [r["outcome"] for r in records]
    seeds = {r["seed"] for r in records}
    hashes = {r["fixture_hash"] for r in records}
    messages = " ".join(r["message"].lower() for r in records)

    if "" in seeds or not seeds:
        return "unclassified", "pin TEST_SEED and rerun"
    if any(tok in messages for tok in ("timed out", "errno", "connection", "/tmp/", "\\temp\\")):
        if len(set(outcomes)) > 1:
            return "noise", "isolation: path, network, or time in the message"
    if hashes != {next(iter(hashes))}:
        return "fixture_drift", f"fixture hash moved: {sorted(hashes)}"
    if outcomes == ["failed", "failed", "failed"]:
        return "invariant_miss", "stable fail under pinned seed and fixture"
    if outcomes == ["passed", "passed", "passed"]:
        return "stable_pass", "no spend"
    return "noise", "outcome flipped across reruns"


def main(paths: List[str]) -> int:
    runs = [load(p) for p in paths]
    rows = []
    spend = {
        "invariant_miss": "add a property; reject one-example equality",
        "fixture_drift": "diff fixture; lock or refresh",
        "noise": "capture input; freeze only with expiry",
        "unclassified": "pin seed; write no tests",
        "stable_pass": "no spend",
    }
    for nodeid, recs in sorted(group(runs).items()):
        bucket, why = classify(recs)
        rows.append(
            {
                "nodeid": nodeid,
                "bucket": bucket,
                "why": why,
                "spend": spend[bucket],
            }
        )
    print(json.dumps(rows, indent=2))
    # Fail the job if anything is unclassified. Silence here is how flakes get frozen.
    if any(r["bucket"] == "unclassified" for r in rows):
        return 2
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1:]))

A stable pass is not a merge signal by itself. It only means this nodeid does not consume budget this round.

Step 4 — Spend only on the classified bucket

Invariant miss. Add a property the patch cannot satisfy with a literal. Keep the generator in-repo. The following is a proposal, not a measured campaign.

# test_properties.py
import random
from bucketizer import clamp_and_bucket

EDGES = [0.0, 1.0, 2.0, 5.0]


def test_length_range_monotonic():
    rng = random.Random(0xC0FFEE)
    for _ in range(200):
        values = [rng.uniform(-2.0, 12.0) for _ in range(rng.randint(0, 8))]
        got = clamp_and_bucket(values, EDGES)
        assert len(got) == len(values)
        assert all(0 <= x < len(EDGES) for x in got)
        # Monotone in the value, not in insertion order.
        paired = sorted(zip(values, got))
        buckets = [b for _, b in paired]
        assert buckets == sorted(buckets)

Reject an agent patch that replaces the loop with assert clamp_and_bucket([1.0], EDGES) == [1]. That is a story about one point.

Fixture drift. Hash the file in CI. Print a unified diff. A human chooses lock or refresh. Automatic refresh is how a broken serializer becomes the new gold.

sha256sum edges.json > edges.json.sha256
# in CI:
sha256sum -c edges.json.sha256

Noise. Record the input that flipped. If you cannot replay it, you may not freeze it. If you freeze it anyway, store an owner, a ticket, and an expiry in the same commit. A freeze without a kill date is a skipped invariant.

# flakes.toml — proposal schema
# [test_bucketizer.BucketTests.test_recorded_edges_length]
# owner = "platform-testers"
# expires = "2026-10-02"
# replay = "runs/noise-capture.json"

CI should fail when expires is in the past and the test is still skipped. That check is the whole control.

Step 5 — Draft properties from traces, then throw most of them away

Invariant-miss traces are useful prompts. They are not oracles. A model can propose generators and bounds from the failing message field. A human keeps one proposal in ten.

Suggested review questions, applied in order:

1. Does the proposal quantify over a domain, or does it replay the failing literal?
2. Can the agent patch edit the proposal in the same PR? If yes, move it to a path the patch cannot touch.
3. Does it still fail on the current patch? If it passes, it is not the miss you classified.
4. Does it depend on wall time, directory order, or network? If yes, it is noise wearing a property costume.

Free model access is a drafting aid for step 5, not a merge voter. Paste the classified JSON, not the whole repository. Keep the accepted property in review like any other production code.

What this does not claim

The classifier is a heuristic over three reruns. It is not a proof of determinism. Three is a default, not a measured optimum. It will mislabel a rare race as an invariant miss if the race lost all three times. It will mislabel a broken fixture generator as noise if the generator is itself flaky.

It also does not score tests the patch authored. That is a different control. Do not fold it into this script.

No latency, token, or hardware figures are attached to the remote runner here. Use whatever remote you already trust. The requirement is repeatability, not a vendor.

Who should not use this

Do not use the taxonomy as the only gate on security, payments, or privacy patches. A property about list length will not catch an authorization hole.

Do not use it if the suite cannot pin a seed or hash a fixture. The buckets collapse into one, and you are back to starring the red job.

Do not let the agent own classify_failures.py, flakes.toml, or the property file. If the patch can rewrite the classifier, every bucket becomes stable_pass.

Skip the model-drafting step when the miss is already a one-line invariant. Writing len(got) == len(values) does not need a generator of generators.

Close

Green is cheap. Classification is the scarce resource. Convert an invariant miss into a property, a drifted recording into a lock, and noise into a replayable input. Freeze last, and only with a date. If you need a remote box so the three reruns are not a laptop artifact, the free server option is a sufficient place to run rerun.sh once and read the JSON.