A Quarantine Ledger for Agent Patches

A CI suite runs 300 tests. Two of them fail on almost every diff. The failures land in files nobody touched.

Reviewers learn to scroll past red. An agent opens forty diffs a week. Each one looks green except the usual two. The suite stops being a gate and becomes wallpaper.

This post describes a workflow for that situation. It has three parts: a quarantine ledger with expiry dates, a property harness for agent patches, and a fixture budget that makes silent fixture edits visible. Every snippet below is runnable pytest against a small synthetic importer. The counts are illustrative, not measured.

The ledger

A ledger is one JSON file in the repo. Each entry names a test, an owner, a reason, and an expiry date.

{
  "tests": [
    {
      "id": "tests/test_billing.py::test_retry_after_webhook",
      "owner": "payments",
      "reason": "clock-dependent, fails ~3 of 20 runs on clean main",
      "opened_at": "2026-10-16",
      "expires_at": "2026-11-13"
    }
  ]
}

A conftest hook reads the ledger at collection time. Unexpired entries get skipped. Expired entries run again, which is the enforcement you want. The skip markers themselves are plain pytest, documented at https://docs.pytest.org/en/stable/how-to/skipping.html.

# conftest.py
import json
import time
from pathlib import Path
import pytest

LEDGER = Path("quarantine.json")
RUN_LOG = Path("artifacts/quarantine-run.json")


def pytest_collection_modifyitems(config, items):
    entries = json.loads(LEDGER.read_text())["tests"] if LEDGER.exists() else []
    today = time.strftime("%Y-%m-%d")
    by_id = {e["id"]: e for e in entries}
    report = {"date": today, "skipped": [], "expired": [], "runnable": 0}

    for item in items:
        entry = by_id.get(item.nodeid)
        if entry and entry["expires_at"] > today:
            item.add_marker(pytest.mark.skip(reason=f"quarantined: {entry['reason']}"))
            report["skipped"].append(item.nodeid)
        else:
            if entry:
                report["expired"].append(item.nodeid)
            report["runnable"] += 1

    RUN_LOG.parent.mkdir(parents=True, exist_ok=True)
    RUN_LOG.write_text(json.dumps(report, indent=2))

Expiry is the whole point. A skip without a date is a deletion in disguise. The run log also gives you one number to watch. The skipped count should fall over a quarter. If it rises, the suite is decaying faster than you repair it.

Properties instead of examples

Agent patches tend to preserve shapes and break rules. An example test asserts that row 42 credits ten dollars. A property test asserts that accepted plus rejected equals input, for any input.

Here is the code under test, small enough to reason about.

# app/importer.py
from dataclasses import dataclass


@dataclass
class ImportResult:
    accepted: int
    rejected: int
    credited_sum: int
    checksum: int


class Importer:
    def __init__(self, conn):
        self.conn = conn

    def run(self, rows):
        accepted = rejected = credited = 0
        for row in rows:
            if row["amount"] == 0:
                rejected += 1
                continue
            cur = self.conn.execute(
                "insert or ignore into ledger_row (id, amount, currency) values (?, ?, ?)",
                (row["id"], row["amount"], row["currency"]),
            )
            if cur.rowcount == 1:
                accepted += 1
                credited += row["amount"]
        self.conn.commit()
        checksum = self.conn.execute(
            "select coalesce(sum(id), 0) from ledger_row"
        ).fetchone()[0]
        return ImportResult(accepted, rejected, credited, checksum)

And the fixtures, with an in-memory database and a seeded generator.

# tests/conftest.py
import random
import sqlite3
import pytest
from app.importer import Importer


class Snapshot:
    def __init__(self, conn):
        self.conn = conn

    def count(self, table):
        return self.conn.execute(f"select count(*) from {table}").fetchone()[0]

    def sum(self, table, column):
        return self.conn.execute(
            f"select coalesce(sum({column}), 0) from {table}"
        ).fetchone()[0]


@pytest.fixture
def snapshot():
    conn = sqlite3.connect(":memory:")
    conn.execute(
        "create table ledger_row (id integer primary key, amount integer, currency text)"
    )
    yield Snapshot(conn)
    conn.close()


@pytest.fixture
def importer(snapshot):
    return Importer(snapshot.conn)


@pytest.fixture
def rows():
    def build(seed, n=40):
        rng = random.Random(seed)
        return [
            {
                "id": rng.randrange(10_000),
                "amount": rng.randrange(-500, 500),
                "currency": rng.choice(["USD", "EUR"]),
            }
            for _ in range(n)
        ]

    return build

Two properties carry most of the weight here: idempotency and conservation.

# tests/properties/test_import_properties.py
import pytest

pytestmark = pytest.mark.property

SEEDS = range(1, 26)


@pytest.mark.parametrize("seed", SEEDS)
def test_import_is_idempotent(seed, importer, snapshot, rows):
    batch = rows(seed)
    first = importer.run(batch)
    assert snapshot.count("ledger_row") == first.accepted

    second = importer.run(batch)
    assert second.accepted == 0
    assert snapshot.count("ledger_row") == first.accepted


@pytest.mark.parametrize("seed", SEEDS)
def test_rejection_never_breaks_totals(seed, importer, snapshot, rows):
    batch = rows(seed)
    result = importer.run(batch)
    assert result.accepted + result.rejected == len(batch)
    assert snapshot.sum("ledger_row", "amount") == result.credited_sum

Twenty-five seeds across two properties is fifty cheap runs. That is a friendlier signal than one hand-written example. It also survives refactors better, because the invariant does not move when the row format does. When a property fails on seed 17, you keep seed 17 forever as a regression.

A fixture budget

Fixtures are shared state that agents can quietly rewrite. Pin a fingerprint and fail the build when the fixture set drifts.

# tests/test_fixture_budget.py
import hashlib
from pathlib import Path

PINNED = "9f2c41ab77de0013"  # update only in the same diff as a ledger entry


def fixture_fingerprint(root="tests/fixtures"):
    digest = hashlib.sha256()
    for path in sorted(Path(root).rglob("*")):
        if path.is_file():
            digest.update(path.name.encode())
            digest.update(path.read_bytes())
    return digest.hexdigest()[:16]


def test_fixtures_did_not_drift():
    assert fixture_fingerprint() == PINNED, "fixture set changed; justify it in the diff"

This test is intentionally annoying. It forces the patch author to state why the fixture moved. Most agent diffs leave fixtures alone, so the cost is near zero. The benefit arrives the day a diff rewrites a fixture to match a bug.

Re-entry

Quarantine ends with evidence, not optimism. Run the test against clean main, twenty times, before you delete the entry.

#!/usr/bin/env bash
# scripts/reentry.sh <nodeid>
set -euo pipefail
TEST_ID="$1"
git switch main && git pull --ff-only
for i in $(seq 1 20); do
  pytest "$TEST_ID" -q || { echo "flake on run $i"; exit 1; }
done
python - "$TEST_ID" <<'PY'
import json, pathlib, sys
path = pathlib.Path("quarantine.json")
data = json.loads(path.read_text())
data["tests"] = [t for t in data["tests"] if t["id"] != sys.argv[1]]
path.write_text(json.dumps(data, indent=2))
print("removed", sys.argv[1])
PY

Twenty passing runs on clean main is a weak guarantee, and you should say so out loud. It still beats deleting a skip because the log annoyed someone.

Deciding what to do

What you observe	Ledger action	Effect on the merge gate
Fails 2 of 10 runs on clean main	Open entry with a 30-day expiry	Excluded from the verdict, still recorded
Fails only on the agent diff	No entry	Merge blocked, treat as a real regression
Passes locally, fails in CI	Freeze clock, timezone, locale	New entry for the environment diff
Diff edits fixtures	Fingerprint test fails	Author must justify the change
Property fails on one seed	Keep that seed as a regression	Merge blocked until fixed

The rule behind the table is simple. Unknown failures lose their vote. Known failures keep it.

Where MonkeyCode fits

MonkeyCode is the operator's open-source project, and the fit here is narrow. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The operator states that free model access and a free server option are available, with a free allowance of 10 million tokens.

I have not verified quotas, hardware, or how long the offer lasts. Treat every limit as changeable and check the current terms before designing a nightly job around it. If the numbers hold for you, the free model access is where patch generation happens, and the free server option is where the repeated sweeps run. Fifty property runs per diff adds up quickly, and that repeated compute is exactly what a metered runner pushes you to skip.

Nothing above depends on MonkeyCode. Swap in any model and any runner, and the ledger, the properties, and the fingerprint test behave the same way.

Limits and who should skip this

A ledger without expiry becomes a graveyard. Review the skipped count monthly and treat growth as a bug. Property tests only work when you can state the invariant in one sentence. If you cannot, example tests are honest enough.

This approach fits three groups poorly. Teams with a deterministic suite should fix the flake and move on. Teams that need per-test evidence for compliance should not hide tests behind a skip marker. Teams without a named ledger owner should not start, because the file will rot within a quarter.

Start with one entry and one property. The ledger is not a place to hide failures. It is a record of what you do not know yet, with a date attached.