Three Gates for Agent Patches: Properties, Fixtures, and a Flaky Freeze

An AI-generated patch that passes its own tests is not evidence. It is a hypothesis. The tests were written by the same model that wrote the code, so they share its blind spots: they check examples, not invariants; they pass on the agent's machine, not yours; and when they fail, the default response is to re-run until green.

This article proposes a three-gate testing strategy for agent patches: property checks, deterministic fixtures, and a freeze on flaky tests. The code is minimal and runnable. The workflow is the artifact, and it is designed for C++ codebases where the agent's unit tests are the cheapest form of confidence and the least trustworthy.

Why the agent's tests are not enough

An agent generates a patch and a test file in one pass. The test file encodes the same assumptions as the patch. If the patch misreads the API contract, the test usually misreads it the same way. Unit tests answer "does this example work?". They do not answer "does this invariant hold for every input?". The gap between those two questions is where the bugs live.

Two results from my recent work illustrate the gap. A thread pool passed 64/64 tasks while ThreadSanitizer found a race in the destructor. An SPSC queue passed its unit tests while the server's TSan run did not. The tests were green. The code was not.

Gate 1: Property checks

Replace "the agent's test cases" with invariants that must hold for many generated inputs. A property test generates inputs, runs the invariant, and reports a failing case. The counterexample is the deliverable; the agent can consume it directly in the next prompt.

class BumpAllocator {
    std::vector<std::byte> storage_;
    std::size_t offset_ = 0;
public:
    explicit BumpAllocator(std::size_t n) : storage_(n) {}

    void* allocate(std::size_t size) {
        offset_ = (offset_ + alignof(std::max_align_t) - 1)
                & ~(alignof(std::max_align_t) - 1);
        if (offset_ + size > storage_.size()) return nullptr;
        void* p = storage_.data() + offset_;
        offset_ += size;
        return p;
    }
};

// Property: every allocated block is aligned and never overlaps a live block.
bool bump_allocator_property(std::mt19937& rng) {
    BumpAllocator alloc(8192);
    std::vector<std::pair<std::uintptr_t, std::size_t>> live;

    for (int op = 0; op < 500; ++op) {
        if (live.size() < 16 && rng() % 3 != 0) {
            std::size_t size = 1 + (rng() % 128);
            void* p = alloc.allocate(size);
            if (p == nullptr) continue;  // exhausted, not a violation
            auto addr = reinterpret_cast<std::uintptr_t>(p);
            if (addr % alignof(std::max_align_t) != 0) return false;
            live.emplace_back(addr, size);
        } else if (!live.empty()) {
            live.pop_back();
        }

        for (std::size_t i = 0; i < live.size(); ++i)
            for (std::size_t j = i + 1; j < live.size(); ++j) {
                auto [a, sa] = live[i];
                auto [b, sb] = live[j];
                if (a < b + sb && b < a + sa) return false;
            }
    }
    return true;
}

Run it with a fixed seed first, then with several seeds in CI. When it fails, shrink the failing sequence by hand or with a minimizer.

Write properties for the contract, not the implementation. Good candidates: "push followed by pop returns the pushed value", "the cache never exceeds capacity", "serialize then parse is the identity", "allocations are aligned and disjoint". If you cannot name three invariants for a patch, you do not understand the patch well enough to merge it.

Gate 2: Deterministic fixtures

Agent tests often pass because the environment is forgiving. Wall-clock time, random seeds, port allocation, locale, and the working directory leak into test behavior. A fixture pins them.

struct PinnedEnvironment {
    unsigned seed;
    std::filesystem::path tmp;

    explicit PinnedEnvironment(unsigned s)
        : seed(s),
          tmp(std::filesystem::temp_directory_path() /
              ("agent_patch_" + std::to_string(::getpid()))) {
        std::filesystem::create_directories(tmp);
        setenv("TZ", "UTC", 1);
        tzset();
        std::srand(seed);
    }

    ~PinnedEnvironment() {
        std::filesystem::remove_all(tmp);
    }
};

Every test that touches time, randomness, the filesystem, or the network goes through the fixture. If the agent's test reads std::chrono::steady_clock::now() directly or creates its own temp files, treat that as a defect in the test, not a detail. The goal is not to make tests fast. It is to make a failure reproducible, because a failure that cannot be reproduced cannot be prompted back to the agent.

Gate 3: The flaky freeze

The third gate is a policy, not a library. When a test fails once in CI, the default move is to re-run it. Re-running teaches the team that red is negotiable. Worse, it teaches the agent that flaky output is acceptable, because the flaky test still merges.

The freeze: a test that fails once is quarantined. It does not run again until a human commits a root-cause fix. Re-running the same binary to "check if it was flaky" is forbidden in the acceptance loop. Diagnosis can happen locally; acceptance requires a fix.

#!/usr/bin/env bash
# quarantine.sh -- one failure freezes the test until a fix lands.
set -euo pipefail
TEST_BIN="$1"
QUARANTINE=".quarantine"

if grep -qxF "$TEST_BIN" "$QUARANTINE"; then
    echo "QUARANTINED: $TEST_BIN (root cause required)"
    exit 1
fi

if "$TEST_BIN" >/tmp/test.log 2>&1; then
    exit 0
fi

echo "$TEST_BIN" >> "$QUARANTINE"
echo "FROZEN: $TEST_BIN failed once. Fix the root cause, then remove it from .quarantine."
exit 1

Note what the script does not do. It does not re-run. It does not let the test back in without an explicit removal from the quarantine list. The removal is the human's signature.

The order of the gates

1. Ask the agent for a patch and its tests.
2. Write three invariants for the patch's contract.
3. Run the property checks under pinned fixtures.
4. Run the agent's unit tests under the same fixture.
5. Run TSan and ASan if the patch touches memory or concurrency.
6. If any test fails once, freeze it and send the log back to the agent as the next prompt.

Order matters. Properties before unit tests, because a counterexample is more informative than a failed assertion. Fixtures before everything, because a test that depends on the environment is not a test. Sanitizers after the deterministic gates, because you want the deterministic bugs gone before the nondeterministic ones get your attention.

Agent patch arrives with	Gate that catches it	What you do
Unit tests that pass locally	Property checks	Feed the counterexample back as the next prompt
Tests that pass on the agent's machine	Pinned fixtures	Fix the hidden dependence on time, seed, or cwd
A test that failed once in CI	Flaky freeze	Quarantine, root-cause, then unquarantine
A race that passes 64/64 tasks	TSan after gates 1-3	Fix the destructor, not the test

Where the agent fits

None of this is specific to a particular agent. Disclosure: This article was prepared as part of MonkeyCode's product outreach. In the context of this workflow, the agent is MonkeyCode's free model access, and the runs use the free server option.

I am not quoting throughput or latency figures, because I have not measured them for this article. The relevant property is simpler: the agent accepts a follow-up prompt, so a failed gate produces the next prompt instead of a meeting. That is the loop the three gates are built to feed.

Limitations

The strategy has costs. Property checks need invariants, and writing invariants is real work; for a one-line patch it is overkill. Fixtures do not fix nondeterminism in the code under test; they only expose it. The flaky freeze will occasionally quarantine a test that failed because of a genuine bug, which is correct behavior, but it will also annoy everyone when the root cause is a slow CI machine. And none of the three gates replace sanitizers; TSan and ASan still run on every patch that touches memory or concurrency.

Who should not use this

Do not use this workflow for prototypes, throwaway scripts, or patches that will be deleted next week. The overhead is justified when the patch survives in the codebase for months. If the team is one person and the CI is a laptop, reduce the freeze to a personal rule: never re-run a failing test without writing down why it failed.

The next time an agent hands you a green test suite, ask what invariant it did not check. If the answer is "none", you have not finished reviewing the patch. You have only finished reading its confidence.