Merge Agent Patches on Harvested Failures, Not on Tests They Wrote

Agent-authored tests are a weak merge signal. The same process that produced the patch also produced the proof. A stronger gate is a harvest loop: human-owned properties, searched inputs, shrunk failures pinned as fixtures, and a dual-run that demands a red witness on the base tree.

This article is a testing workflow, not a model review. The code below is a worked example you can run locally. It does not claim production metrics.

The evidence problem

Agent patches fail in a narrow pattern. The production change is plausible. The tests that land with it are also plausible. Both were generated against the same prompt, so they agree with each other more often than they agree with the domain.

Happy-path assertions pass. Boundary behavior is missing. Remainder cases, empty collections, and mixed signs never appear. When they do appear, the patch often “fixes” them by changing the assertion, not the code.

Coverage numbers do not catch this. A line can execute and still encode the wrong invariant. You need inputs that try to break the invariant, then you need those inputs to outlive the patch.

What the harvest loop actually does

The loop has one job: turn a property into a frozen set of counterexamples, then refuse a patch that neither keeps them green nor shows that at least one of them was red before the change.

1. Write properties in a file the patch authoring path cannot treat as disposable.
2. Sample candidate inputs from cheap sources. Random values, hand-written edges, and optional model-proposed JSON are enough.
3. Execute each candidate against the property. On failure, shrink to a smaller input that still fails.
4. Pin the shrunk input as a fixture. The fixture is now part of the merge corpus.
5. Dual-run the corpus on the base tree and the patched tree. Merge only when the corpus stays green after, and at least one fixture was red before.

Agent-written unit tests can still land. Treat them as comments with a test runner. They are not the gate.

Step 1: Own one property the patch cannot rewrite

Start with an invariant that is true for every legal input. Do not start with an example.

The example domain here is integer allocation: split amount cents across n parties. The function must not create or destroy cents. Shares must be integers. For n > 0, every share is at least amount // n after a remainder pass, and the sum is exact.

# alloc.py
from typing import List

def allocate_cents(amount: int, n: int) -> List[int]:
    if n <= 0:
        raise ValueError("n must be positive")
    if amount < 0:
        raise ValueError("amount must be non-negative")
    q, r = divmod(amount, n)
    shares = [q] * n
    for i in range(r):
        shares[i] += 1
    return shares


def property_conservation(amount: int, n: int) -> None:
    shares = allocate_cents(amount, n)
    if len(shares) != n:
        raise AssertionError(f"len={len(shares)} n={n}")
    if any(s < 0 for s in shares):
        raise AssertionError(f"negative share {shares}")
    if sum(shares) != amount:
        raise AssertionError(f"sum={sum(shares)} amount={amount}")
    if n > 0 and max(shares) - min(shares) > 1:
        raise AssertionError(f"unbalanced {shares}")

A weak agent patch will “fix” an off-by-one by dropping the remainder loop and adding a test for allocate_cents(100, 4) == [25, 25, 25, 25]. That test is true and useless. Conservation on amount=1, n=3 is the actual check.

Step 2: Sample inputs from three cheap sources

Do not ask the patch model to write the tests for its own change. Build a candidate mill that is independent of the patch prompt.

# harvest.py
import json
import random
from pathlib import Path
from typing import Dict, Iterable, List, Tuple

Candidate = Dict[str, int]

EDGES: List[Candidate] = [
    {"amount": 0, "n": 1},
    {"amount": 1, "n": 1},
    {"amount": 1, "n": 3},
    {"amount": 2, "n": 3},
    {"amount": 10, "n": 3},
    {"amount": 10**6, "n": 7},
]

def random_candidates(k: int, rng: random.Random) -> List[Candidate]:
    out = []
    for _ in range(k):
        out.append({"amount": rng.randint(0, 10_000), "n": rng.randint(1, 64)})
    return out

def load_model_candidates(path: Path) -> List[Candidate]:
    if not path.exists():
        return []
    rows = json.loads(path.read_text())
    cleaned = []
    for row in rows:
        amount, n = int(row["amount"]), int(row["n"])
        if n > 0 and amount >= 0:
            cleaned.append({"amount": amount, "n": n})
    return cleaned

The third source is optional. A model may propose JSON objects. It does not propose assertions. If a proposed row fails schema or sign checks, drop it. The mill stays cheap because invalid JSON is not a test failure. It is just an unused candidate.

Step 3: Execute, shrink, pin

Run every candidate against property_conservation. On failure, reduce amount and n while the property still fails. Pin the smallest failing pair.

from alloc import property_conservation

def run_one(c: Candidate) -> str | None:
    try:
        property_conservation(c["amount"], c["n"])
        return None
    except (AssertionError, ValueError) as exc:
        return type(exc).__name__ + ": " + str(exc)

def shrink(c: Candidate) -> Candidate:
    current = dict(c)
    changed = True
    while changed:
        changed = False
        for key in ("amount", "n"):
            if current[key] <= (0 if key == "amount" else 1):
                continue
            trial = dict(current)
            trial[key] = max(0 if key == "amount" else 1, current[key] // 2)
            if run_one(trial):
                current = trial
                changed = True
    # unit decrements after binary shrink
    for key in ("amount", "n"):
        while current[key] > (0 if key == "amount" else 1):
            trial = dict(current)
            trial[key] -= 1
            if not run_one(trial):
                break
            current = trial
    return current

def pin(corpus: Path, c: Candidate, err: str) -> None:
    corpus.mkdir(parents=True, exist_ok=True)
    name = f"amt{c['amount']}_n{c['n']}.json"
    payload = {"amount": c["amount"], "n": c["n"], "error": err}
    (corpus / name).write_text(json.dumps(payload, indent=2) + "\n")

Shrinking matters. A failure at amount=8741, n=41 is hard to review. The same bug at amount=1, n=3 is a one-line argument in review. Pin the small one. Keep the large one only if shrinking loses the failure, which usually means the property is not deterministic.

A minimal driver:

def harvest(corpus: Path, model_json: Path, seed: int = 20260917) -> int:
    rng = random.Random(seed)
    candidates = EDGES + random_candidates(200, rng) + load_model_candidates(model_json)
    pinned = 0
    seen = set()
    for c in candidates:
        err = run_one(c)
        if not err:
            continue
        small = shrink(c)
        key = (small["amount"], small["n"])
        if key in seen:
            continue
        seen.add(key)
        pin(corpus, small, err)
        pinned += 1
    return pinned

Seed the random mill. Unseeded harvests produce corpus churn that looks like flakiness. It is not flakiness. It is an unstable search. Freeze the seed in CI the same way you freeze fixture bytes.

Step 4: Dual-run as the merge signal

A green corpus on the patched tree is not enough. A patch can delete the bug and also delete the only input that could have shown it. Dual-run both trees.

# gate.py
import json
import subprocess
import sys
from pathlib import Path

def eval_tree(tree: Path, fixture: Path) -> bool:
    payload = json.loads(fixture.read_text())
    script = (
        "import sys; sys.path.insert(0, sys.argv[1]); "
        "from alloc import property_conservation; "
        "property_conservation(int(sys.argv[2]), int(sys.argv[3]))"
    )
    proc = subprocess.run(
        [sys.executable, "-c", script, str(tree), str(payload["amount"]), str(payload["n"])],
        capture_output=True,
        text=True,
    )
    return proc.returncode == 0

def gate(base: Path, patch: Path, corpus: Path) -> int:
    fixtures = sorted(corpus.glob("*.json"))
    if not fixtures:
        print("empty corpus: harvest before merge")
        return 2
    base_pass = []
    patch_fail = []
    red_witness = False
    for fx in fixtures:
        b_ok = eval_tree(base, fx)
        p_ok = eval_tree(patch, fx)
        base_pass.append(b_ok)
        if not b_ok and p_ok:
            red_witness = True
        if not p_ok:
            patch_fail.append(fx.name)
    if patch_fail:
        print("patched tree still fails:", ", ".join(patch_fail))
        return 1
    if not red_witness:
        print("no red witness on base; patch did not fix a harvested failure")
        return 1
    print(f"ok fixtures={len(fixtures)} base_red={base_pass.count(False)}")
    return 0

The red-witness rule is the part most CI setups skip. If the base tree already passed every pinned fixture, the patch did not earn the corpus. It may still be a refactor. Refactors need a different lane: no production behavior change, corpus unchanged, dual-run all green on both trees. Do not reuse the “bugfix” gate for that lane.

Commands for a local check:

python - <<'PY'
from pathlib import Path
from harvest import harvest
print("pinned", harvest(Path("corpus"), Path("model_candidates.json")))
PY

python gate.py  # or: python -c "from pathlib import Path; from gate import gate; raise SystemExit(gate(Path('base'), Path('patch'), Path('corpus')))"

Keep base/ and patch/ as two checkouts or two copies of alloc.py. The gate only imports the module from the tree path it is given. That isolation is the point. A patch that edits harvest.py to skip shrinking is a different change and should fail a path filter in CI.

Step 5: Treat agent tests as comments

If the agent adds test_allocate_even_split, leave it. Do not delete it in a holy war. Also do not count it.

CI can run agent tests for smoke. The merge job should depend on gate.py plus a path rule: a production patch that also rewrites corpus/ needs a human label. Corpus growth is allowed when the harvest finds a new shrink. Corpus deletion is not allowed in the same commit as the fix that made the fixture green. Green means the fixture stays, and now passes.

Decision table

Observation	Merge action
Patch tree fails any pinned fixture	Block. The invariant still dies.
Patch tree green, base tree red on at least one fixture	Allow as a behavior fix.
Both trees green on the whole corpus	Allow only as refactor, and only if alloc.py behavior hash is explained by review.
Corpus empty	Block. Harvest has not run.
Patch edits alloc.py and deletes fixtures	Block. That is oracle thinning.
Model JSON contains rows that fail schema	Ignore rows. Do not fail the gate.
Same seed yields different pins	Stop. The property or the function is not deterministic enough to harvest.
Failure shrinks away to a legal no-op	Drop the candidate. It was noise, not a bug.

The table is the policy. Encode it in the gate, not in a chat message to the agent.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The harvest mill needs two cheap resources: a proposer for extra candidates, and a machine that can run the loop without blocking a laptop. MonkeyCode’s free model access and free server option fit that split if you already use them. The model emits JSON candidates. The server runs harvest() and gate() against two trees. Neither resource writes the property, and neither resource is the merge authority.

Do not send the production patch to the proposer and ask it for tests of that patch. Send the property docstring and the candidate schema only. Keep the proposer off the assertion vocabulary. If the free server is busy or unavailable, the seeded random mill plus the edge list still produce a corpus. The gate does not require a model in the critical path.

Limitations

Harvest assumes a pure function and a deterministic property. Time, clocks, network, and shared temp directories poison shrinking. If the failure depends on wall time, you will pin noise.

Integer domains shrink well. Strings and ASTs shrink only if you write a reducer. Without a reducer, you will pin kilobyte blobs that reviewers cannot read. That is worse than no harvest.

The red-witness rule rejects some good patches. A performance-only change has no red fixture. Put it on the refactor lane. Do not weaken the bugfix lane to let it through.

Dual-run with python -c is a teaching artifact. A real repo should import from installed packages or PYTHONPATH per worktree. It should also hash fixture files so silent JSON edits show up in git diff.

This workflow does not measure model quality. It measures whether a patch removed a known failing input without rewriting the evidence.

Who should skip this

Skip harvest if you do not have a property yet. A list of examples is not a property. Writing assert allocate_cents(100, 4) == [25, 25, 25, 25] and then searching around it will freeze the implementation, not the invariant.

Skip it for UI snapshots, flake-prone HTTP suites, and anything that needs live credentials. Those need hermetic fixtures and human review, not a candidate mill.

Skip it if the agent is allowed to edit the harvest scripts in the same change as production code. The loop is only as strong as the path filter that keeps property_*, harvest.py, and gate.py out of the patch’s write set.

If you already run properties in CI, add shrinking and dual-run before you add more tests from the agent. The missing piece is usually the red witness, not another generated assert.