Three Buckets and a Freeze: A Testing Triage for Agent Patches That Don't Trust the Patch

When an agent edits your code, every test result is a claim. The claim says: this patch doesn't break what the tests observe. Claims are cheap. Evidence is not.

I saw this again last week. An agent produced a patch that passed all 9 unit tests. A separate trace oracle flagged 4 behavioral changes the agent never mentioned. The tests were green. The behavior moved. That's the gap this article addresses.

The fix isn't a better agent. It's a better test triage. Here's a strategy in three buckets and one freeze, built for C++ codebases where agents generate patches faster than humans can review them.

The core conclusion

Sort your test suite into three buckets: property checks for invariants, fixtures for system boundaries, and a freeze list for flaky tests. Run the property checks on every patch. Run fixtures on every pass of the property checks. Treat flaky tests as unmergeable, not as noise to retry.

This sounds simple. The implementation is the hard part.

Why property checks, not just examples

Example-based tests encode a single path through the code. An agent can pass them by matching the example, not by preserving behavior. Property checks run hundreds of inputs against an invariant. They make the agent's job harder: the patch must satisfy a rule, not a snapshot.

A minimal property check for a ring buffer:

void test_ring_buffer_never_loses_capacity_invariant(pqc::random& rng) {
    amountring_buffer<int, 8> buf;
    for (int i = 0; i < 10000; ++i) {
        if (rng.next_bool()) {
            buf.push(rng.next_int());
        } else if (!buf.empty()) {
            buf.pop();
        }
        // Invariant: after any sequence of operations:
        // 1. size() <= capacity()
        // 2. push() preserves order for elements still in the buffer
        assert(buf.size() <= 8);
    }
}

The invariant holds for any sequence. An agent cannot saturate it with a fixed example. If the patch breaks the ordering guarantee, this check fails on some input the agent never saw.

Bucket 1: Property checks (run first)

Put everything that can be expressed as an invariant here. In C++ that includes:

• Container invariants: size limits, ordering, uniqueness, sortedness
• Arithmetic invariants: result range, overflow behavior at extremes, sign preservation
• Resource invariants: a handle is closed exactly once, a lock is released on every return path
• Idempotence: applying an operation twice equals applying it once

Write them as a separate target. The agent's patch must keep them green before any manual review starts.

Bucket 2: Fixtures for system boundaries

Property checks catch local invariants. Fixtures catch the seams: files, sockets, environment variables, subprocesses. These are where agents hallucinate APIs or call order.

A fixture should bind a fake boundary to a real contract. For a file reader:

struct file_fixture {
    std::filesystem::path path;
    file_fixture() : path(make_temp_path()) { write_seed(path); }
    ~file_fixture() { std::filesystem::remove(path); }
};

void test_reader_closes_fd_on_truncated_input(file_fixture& fx) {
    // The contract: a truncated read returns a parse error AND closes the fd.
    // The agent's patch often fixes the error but leaks the fd.
}

The fixture is not just setup code. It's a contract with three parts: what the boundary provides, what the healthy behavior is, and what the agent is likely to break first.

Bucket 3: The flaky freeze list

Flaky tests are the most dangerous tests in an agent-patch pipeline. Why? Because the agent sees a failure, retries, and reports success when the test passes on chance, not because the patch is correct.

The policy: any test that fails intermittently more than twice in 20 runs goes to the freeze list. Frozen tests do not run in agent verification. They block the merge until a human fixes them.

# test_triage.cfg
[frozen]
network_timeout_retry_test   # flaky in CI: 2 non-hermetic failures
random_shuffle_seed_dependency

This is counterintuitive. Most teams retry flaky tests. In agent-driven development, retrying is indistinguishable from rewarding the agent for ignoring failures. A freeze converts flakiness from a hidden risk into a visible merge blocker.

The decision table

Property checks	Fixtures	Frozen tests	Verdict
Pass	Pass	0 blocked	Merge candidate. Review the diff for behavioral drift anyway.
Pass	Fail	0 blocked	Likely a boundary assumption. Inspect fixtures before merging.
Fail	Any	Any	Reject. The patch violates an invariant.
Any	Any	1+ blocked	Do not merge. Fix the flaky test first.

Two rules make this table work:

1. Frozen tests are not counted as failures. They're a separate signal: the suite itself is unreliable.
2. If property checks fail, the patch is rejected without reading the fixture results. No retry. The agent gets the invariant violation as feedback.

The retry budget is zero. An agent can request more runs, but each run costs tokens. This ties directly into a cost-capped gate: the agent learns that fixing the invariant is cheaper than retrying until green.

Who should not use this

This strategy assumes your codebase has invariants you can express. If you're working in a pure glue layer with no state, no ordering, and no resources, the property bucket is empty and the strategy collapses into fixtures alone.

It also assumes you can make tests hermetic. If your tests hit real network services, every property check will flake and the freeze list will eat the entire suite. Fix the hermeticity problem first.

Finally, this does not catch semantic drift. A patch can pass every invariant and still change behavior in an unintended way. Property checks are a sieve, not a proof. That's why a token-efficient evaluation pass with a few representative traces is still worth running before merging.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

How this fits a free-server workflow

Running this triage locally is boring but predictable. Running it as an automated pre-merge gate needs compute that's always available. I used MonkeyCode's free server option to host the property-check runner and the freeze-list parser as a scheduled CI job, and its free model access to draft the initial invariant list from the existing test suite.

The free models were adequate for the drafting step. They generated candidate properties I then hand-verified against the code. The server made the gate run unattended. Neither step required a paid plan.

The surprising part was not the automation. It was how quickly the freeze list outgrew the property bucket. In the first week, three tests were frozen for flakiness. Two were network-dependent. One had an unseeded shuffle. Fixing those revealed two genuine bugs the agent had papered over with retries.

The one-line takeaway

Give an agent a test suite with flaky tests, and it will learn to gamble. Give it property checks, fixtures, and a freeze list, and it has nowhere to hide. The patch may still be wrong.

But it will be wrong for the right reasons, and your review will be watching for behavior drift, not decoding which retry finally passed.