Agent Patches Need a Test Contract: Properties, Fixtures, and a Quarantine

An agent patch arrives with tests that pass. That is a claim, not evidence. The tests were written by the same model that wrote the code, so they share its blind spots. In my last few agent-patch reviews, the unit tests caught nothing that mattered; the failures came from three layers the agent never saw: property checks, pinned fixtures, and a flaky quarantine. This article is the gate I run before any agent-written change merges.

Why the agent's own tests are not enough

The agent's test suite is a self-portrait. It passes because the implementation and the tests share an author and an assumption. When the assumption is wrong — eviction order, capacity edge, key reuse — both sides agree to be wrong.

A concrete failure I keep seeing: the agent changes an eviction policy from LRU to something faster, rewrites the unit tests to match, and every test is green. The cache is now a different data structure with the same name. No unit test notices, because the unit tests were rewritten to describe the new behavior.

Layer 1: Property checks

A property is a statement that must hold for any correct implementation. It does not say how. It says what.

For an LRU cache, three properties cover most of the contract:

1. Capacity invariant: after any sequence of operations, len(cache) <= capacity.
2. Round-trip: put(k, v) followed by get(k) returns v, as long as k was not evicted in between.
3. Eviction order: inserting a new key evicts the least-recently-used key; touching a key protects it from the next eviction.

Here is the check I run. Plain Python, no framework, seeded and deterministic:

def check_eviction_order(make_cache, capacity=8):
    c = make_cache(capacity)

    for i in range(capacity + 1):
        c.put(f"k{i}", i)

    assert c.get("k0") is None, "oldest key must be evicted first"

    c.get("k1")                 # touch k1: it is now the most recent
    c.put("fresh", 99)          # force the next eviction

    assert c.get("k2") is None, "touching k1 must protect it"
    assert c.get("k1") == 1, "recently touched key must survive"
    print("eviction-order properties hold")

A FIFO cache fails this on the third assertion: it evicts k1 even though k1 was just touched. A random-eviction cache fails it intermittently. The agent's own unit tests, written to match the new behavior, pass. The property check does not care about the agent's narrative.

Then add the randomized capacity and round-trip checks:

import random

def check_capacity_and_roundtrip(make_cache, capacity=16, steps=2000, seed=7):
    rng = random.Random(seed)
    c = make_cache(capacity)

    for _ in range(steps):
        k = f"key-{rng.randrange(64)}"
        if rng.random() < 0.6:
            c.put(k, rng.randrange(2**31))
            assert c.get(k) is not None, "round-trip failed"
        else:
            c.get(k)
        assert len(c) <= capacity, "capacity invariant violated"

    print("capacity and round-trip invariants hold")

Note the len(c) requirement. If the agent's cache does not expose its size, that is the first thing to fix. A data structure you cannot measure is not reviewable.

Layer 2: Pinned fixtures

Properties prove general invariants. Fixtures pin real behavior. You need both, because a cache can satisfy every LRU property and still corrupt a specific workload.

The rule: fixtures are append-only and hash-pinned. The manifest lives in the repo:

{
  "fixtures": [
    {"path": "corpus/prod_trace.bin", "sha256": "9f2c4a...", "note": "captured 2026-08-14"},
    {"path": "corpus/edge_1k.bin",    "sha256": "ab12d9...", "note": "synthetic, 1024 keys"}
  ]
}

When an agent patch changes behavior, the fixture diff shows the semantic change. That is the point. A patch that "optimizes" eviction and also rewrites a fixture has a story to tell; the diff is the story. If the fixture changed but the patch description did not mention it, the patch is rejected.

The trap is the agent that "fixes" a failing test by mutating the fixture. Hash-pinning does not prevent that. It makes it visible.

Layer 3: Flaky quarantine

A flaky test destroys the signal of the first two layers. If the gate fails randomly, the agent learns to ignore the gate. The fix is not to delete the test. It is to quarantine it.

Flaky tests move to a quarantine/ directory. They do not block merges. They carry a ticket and an expiry date. When the date passes, the test is either repaired or deleted. A quarantine without an expiry is a graveyard with a nicer name.

I covered the expiry mechanism in an earlier post, so I will keep this short: the quarantine keeps the gate honest, and the expiry keeps the quarantine from becoming a permanent hiding place for real regressions.

The loop, with MonkeyCode

This is where the workflow comes together. The agent in this loop runs against MonkeyCode's free model access, and the harness runs on the free server option — infrastructure I control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The free model access changes the loop economics. Because each revision attempt does not consume a metered budget, I can iterate: generate a patch, run the three-layer gate, send the failure report back, repeat. The gate is identical every round. The agent does not get to argue with the property check; it gets to read the assertion.

The free server option matters for reproducibility. A seeded failure at step 1,412 today can be rerun tomorrow on the same machine. That reproducibility is the basis of the quarantine: a test that fails on the same seed twice in a row is flaky by definition, and now you can prove it.

The workflow, numbered:

1. The agent proposes a patch (MonkeyCode free model access).
2. Run the agent's own unit tests. They will pass. Record that fact; do not act on it.
3. Run the property checks. Failures here stop the patch immediately.
4. Run the fixture suite. Any fixture diff must be explained in the patch description.
5. Run the suite three times on the same seed. Any test that fails once goes to quarantine with a ticket and an expiry date.
6. Merge only if layers 1–4 are clean and layer 5 contains no new quarantines.

Decision table

Condition	Verdict
Unit tests pass, properties fail	Reject. The agent's tests describe the bug.
Properties pass, fixture diff unexplained	Reject. Behavior changed off-contract.
All pass, one flake in three runs	Quarantine the flake, merge, repair before expiry.
All pass, zero flakes, mutation survives	Run the mutation gate (see my earlier posts).
All pass, zero flakes, no fixture diff	Merge. This is the only case that merges.

Limitations

This strategy is not for every patch. A one-line constant change does not need property checks; the gate overhead would exceed the patch value. A prototype that will be deleted next sprint does not need fixtures. And property checks cannot prove the absence of bugs — they prove the presence of violations, on the sequences you generated.

Who should not use this: teams without CI budget, teams that cannot review a fixture diff, and teams that treat quarantine as deletion. The quarantine is a debt ledger. If you never pay the debt, the ledger lies to you.

In my experience, the three layers cost about an hour per patch. They catch more real faults than the agent's unit tests do, and they produce failure reports the agent can actually act on. If your gate has a fourth layer I am missing, I would like to read about it.