Keep a Blind Fixture Set for Agent Patches

A green suite the agent can read is a contaminated oracle. Split fixtures into a visible set and a holdout set, then gate the patch on properties that never entered the prompt. Public tests still catch typos. They do not prove the change is safe.

Agent patches fail in a specific way. The model sees the test names, the assertions, and often the fixture files. It then writes code that satisfies those files. The suite goes green. The bug sits in a path no fixture covered, or in a path the model was never shown.

This article is a testing plan, not a model bake-off. The artifact is a split layout, a small property harness, and a flake ledger that records evidence instead of vibes.

What the split is for

Three jobs, three directories. Mixing them is how contamination happens.

1. Public fixtures live in tests/public/. Humans and agents may read them. They encode the examples you are willing to leak.
2. Holdout fixtures live in tests/holdout/. CI mounts them after the patch is written. The agent session must not index this tree.
3. Properties live in tests/properties/. They check relations over generated inputs, not one golden output per file.

Public tests are a linter with extra steps. Holdout tests are the oracle you still trust. Properties are how you spend the holdout budget on more than a handful of rows.

A concrete module under test

The example is a reservation helper. Capacity is finite. Overbook is forbidden. Idempotent retries with the same request_id must not double-count. Label this as a worked example, not a production measurement.

# reserve.py
from dataclasses import dataclass

@dataclass(frozen=True)
class Stock:
    sku: str
    on_hand: int
    reserved: int

class ReserveError(ValueError):
    pass

def available(stock: Stock) -> int:
    return stock.on_hand - stock.reserved

def apply_reserve(stock: Stock, qty: int, seen_ids: set[str], request_id: str) -> Stock:
    if qty <= 0:
        raise ReserveError("qty must be positive")
    if request_id in seen_ids:
        return stock
    if qty > available(stock):
        raise ReserveError("insufficient stock")
    seen_ids.add(request_id)
    return Stock(stock.sku, stock.on_hand, stock.reserved + qty)

A public test can teach the agent the happy path. That is useful. It is also the leak.

# tests/public/test_reserve_examples.py
from reserve import Stock, apply_reserve

def test_simple_reserve():
    seen = set()
    s = apply_reserve(Stock("sku-1", 10, 0), 3, seen, "r1")
    assert s.reserved == 3
    assert "r1" in seen

Holdout fixtures the session cannot see

Keep holdout data as JSON, not as Python that an indexer will concatenate into the prompt. One object per line is enough.

{"sku":"sku-9","on_hand":4,"reserved":2,"qty":2,"request_id":"h-1","expect":"ok"}
{"sku":"sku-9","on_hand":4,"reserved":2,"qty":3,"request_id":"h-2","expect":"insufficient"}
{"sku":"sku-9","on_hand":4,"reserved":2,"qty":2,"request_id":"h-1","expect":"idempotent"}

Load them only in CI.

# tests/holdout/test_blind_rows.py
import json
from pathlib import Path
import pytest
from reserve import Stock, ReserveError, apply_reserve

HOLDOUT = Path(__file__).parent / "rows.jsonl"

def rows():
    with HOLDOUT.open() as f:
        for line in f:
            yield json.loads(line)

@pytest.mark.holdout
@pytest.mark.parametrize("row", list(rows()), ids=lambda r: r["request_id"])
def test_holdout_row(row):
    stock = Stock(row["sku"], row["on_hand"], row["reserved"])
    seen = set()
    if row["expect"] == "ok":
        out = apply_reserve(stock, row["qty"], seen, row["request_id"])
        assert out.reserved == stock.reserved + row["qty"]
        return
    if row["expect"] == "insufficient":
        with pytest.raises(ReserveError):
            apply_reserve(stock, row["qty"], seen, row["request_id"])
        return
    # idempotent: apply twice, reserved moves once
    out1 = apply_reserve(stock, row["qty"], seen, row["request_id"])
    out2 = apply_reserve(out1, row["qty"], seen, row["request_id"])
    assert out1 == out2

CI must exclude this path from the files sent to the agent. A simple rule: the generate job never checks out tests/holdout/. The verify job does.

# .github/workflows/agent-gate.yml (proposal)
jobs:
  generate:
    steps:
      - uses: actions/checkout@v4
        with:
          sparse-checkout: |
            *
            !tests/holdout
  verify:
    needs: generate
    steps:
      - uses: actions/checkout@v4
      - run: pytest -m "holdout or property" --strict-markers

Sparse checkout is not cryptography. Anyone with repo read access can open the files. The point is to keep them out of the prompt, not out of the company.

Properties over the holdout budget

Row files are finite. Properties spend the same code paths on many inputs. Seed them. Log the seed. Do not pretend a random run is a proof.

# tests/properties/test_reserve_props.py
import random
import pytest
from reserve import Stock, ReserveError, apply_reserve, available

@pytest.mark.property
def test_reserve_never_exceeds_on_hand(seed=20260907):
    rng = random.Random(seed)
    for i in range(200):
        on_hand = rng.randint(0, 20)
        reserved = rng.randint(0, on_hand)
        qty = rng.randint(1, 25)
        stock = Stock("sku", on_hand, reserved)
        seen = set()
        rid = f"p-{i}"
        try:
            out = apply_reserve(stock, qty, seen, rid)
        except ReserveError:
            assert qty > available(stock) or qty <= 0
            continue
        assert 0 <= out.reserved <= out.on_hand
        assert out.reserved == reserved + qty

A second property is the one agents often skip: replay.

@pytest.mark.property
def test_same_request_id_is_idempotent(seed=20260907):
    rng = random.Random(seed)
    for i in range(200):
        on_hand = rng.randint(1, 20)
        qty = rng.randint(1, on_hand)
        stock = Stock("sku", on_hand, 0)
        seen = set()
        rid = "same"
        a = apply_reserve(stock, qty, seen, rid)
        b = apply_reserve(a, qty, seen, rid)
        assert a == b

Mark both. Run public tests in the generate loop if you want fast feedback. Run holdout and properties only after the patch is frozen as a diff.

pytest -m "not holdout and not property"   # cheap, visible
pytest -m "holdout or property"            # blind gate

Classify failures before you freeze anything

A holdout miss is not automatically a product bug. Classify it. Four buckets are enough.

Bucket	Signal	Action
Contamination	Public test was rewritten to match the patch, holdout still fails	Reject the patch; restore the public test
Real regression	Holdout and properties fail on the same relation	Reject; file the row that shrunk the failure
Spec drift	Holdout expects old policy, patch implements a documented change	Update holdout in a human review, never in the agent loop
Flake	Same seed, same diff, pass/fail across reruns	Do not skip. Record it.

The ledger is a JSON file keyed by test node id. It is not a calendar. It does not expire because Friday arrived.

{
  "tests/properties/test_reserve_props.py::test_reserve_never_exceeds_on_hand": {
    "seed": 20260907,
    "runs": 30,
    "fails": 2,
    "last_diff": "sha256:…",
    "state": "watch"
  }
}

Proposal for state transitions:

1. watch — fail rate in (0, 0.10] on a fixed seed. Keep running. Do not skip.
2. frozen — fail rate above 0.10 or non-deterministic across identical diffs. The test does not gate merges. It still runs and records.
3. active — zero fails in the last N identical-seed reruns after a human unfreeze.

A freeze that silently drops the test is how you get a gate that rejects nothing. Keep the row in CI output. Fail the job only for active tests. Print frozen counts as a separate metric.

# scripts/ledger_gate.py  (proposal)
import json
import sys
from pathlib import Path

ledger = json.loads(Path("flake_ledger.json").read_text())
junit_fails = set(sys.argv[1:])  # node ids from the pytest run

blocking = []
for node, row in ledger.items():
    if row.get("state") == "frozen":
        continue
    if node in junit_fails:
        blocking.append(node)

if blocking:
    print("blocking failures:")
    for n in blocking:
        print(" ", n)
    sys.exit(1)
print("no active holdout/property failures")

Wire it after pytest. Do not put the skip logic inside the test body. That hides the flake from the log.

A workflow you can run this week

Numbered on purpose. Skip a step and the split collapses.

1. Move any fixture the agent has already seen into tests/public/.
2. Write at least ten holdout rows the public tests do not duplicate. Prefer boundaries: zero capacity, exact capacity, replayed request_id, qty of one past the remainder.
3. Add two properties with a fixed seed. Log the seed in CI.
4. Change the generate job so tests/holdout/ is not present in the workspace the agent reads.
5. Run public tests during generation. Run holdout and properties on the resulting diff only.
6. On failure, shrink to one row or one seed. Classify with the table above.
7. Record flakes in the ledger. Never pytest.mark.skip from the agent.

That is the whole gate. The model can still memorize public tests. It cannot memorize files it never received.

Generating the candidate patch can happen on a workstation or on a shared box. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are one way to produce that candidate without standing up a private GPU fleet. They do not replace the holdout job. Treat the server as the place the diff is written. Treat this split as the place the diff is judged.

What this does not buy you

Holdout size is statistical power. Ten rows will not find a bug in a branch the rows never enter. Properties with a weak generator have the same hole. If qty is never zero, you will not test the qty <= 0 branch.

CI logs can leak holdout contents into the next prompt if you paste failures back to the agent. Paste the relation, not the row. "Idempotent request_id double-counted" is a prompt. The JSON line is a gift.

Repo-wide RAG, IDE context, and git grep from the agent session all punch through sparse checkout. If the generate environment can read the tree, the split is theater. Use a second checkout, a second repo, or a secrets store the job mounts only at verify time.

A frozen test is a debt instrument. If half the property file is frozen, you no longer have a gate. Schedule a human to shrink and repair those tests. Do not let the agent unfreeze them.

Who should not use this

Do not use a blind corpus as the only review on auth, payments, or anything that can move money. Holdout tests are still tests. They miss classes of harm that are not encoded as rows.

Do not use it on a repo where the full tree is always in the prompt. The split needs an enforcement point.

Do not use it if you cannot pin seeds and diffs. A flake ledger without identity is a spreadsheet of anecdotes.

Solo scripts with one golden file gain little. Write the holdout first, or skip the ceremony and read the diff.

Limits, restated

Public tests leak. Holdout tests work only if they stay out of the prompt. Properties need seeds. Flakes need a ledger that still runs the test. None of this is a substitute for a human reading the diff on the first merge to a given module.

If you already generate patches with free model access on a free server, add the verify job before you add more examples to tests/public/. Extra public fixtures make the agent look better. They do not make the oracle stronger.