Before You Trust an Agent Patch: Properties, Fixtures, and a Flaky Freeze

An agent patch that passes its own tests is not a verified patch. It is a hypothesis with a green checkmark. The tests an agent writes encode what the agent believed, not what the code must guarantee. This article is a three-layer verification contract for agent-written patches: property checks for invariants, deterministic fixtures for edge cases, and a hard freeze on flaky tests. Each layer is cheap to run. Each one catches a different class of fault.

The example is a small C++ function. The contract transfers to Python, Go, or anything with a test runner.

The failure mode

Agent-generated patches fail in a recurring pattern: plausible logic, narrow tests, and a green suite that misses the invariant. Consider split_csv. The agent rewrote it and produced these unit tests:

TEST(SplitCsv, Basic) {
    auto fields = split_csv("a,b,c");
    ASSERT_EQ(fields.size(), 3);
}

TEST(SplitCsv, Empty) {
    auto fields = split_csv("");
    ASSERT_EQ(fields.size(), 1);
}

Both pass. The implementation, however, drops empty fields:

std::vector<std::string> split_csv(const std::string& line) {
    std::vector<std::string> fields;
    std::string cur;
    for (char c : line) {
        if (c == ',') {
            if (!cur.empty()) fields.push_back(cur); // fault
            cur.clear();
        } else {
            cur += c;
        }
    }
    if (!cur.empty()) fields.push_back(cur); // fault
    return fields;
}

split_csv("a,,b") returns two fields instead of three. split_csv("a,") returns one field instead of two. The unit tests never express the invariant that one comma means one field boundary.

Layer 1: property checks

A property check is an assertion about the input/output relationship that must hold for every input, not for a chosen example. For split_csv, three properties cover the contract:

1. Round-trip: joining the fields with , reproduces the input.
2. Cardinality: an input with n commas produces n + 1 fields.
3. Purity: no output field contains a comma.

Here is a minimal property harness with no external dependency. It generates 10,000 random strings and asserts all three properties:

#include <cassert>
#include <random>
#include <string>
#include <vector>

std::vector<std::string> split_csv(const std::string& line);

std::string join(const std::vector<std::string>& fields, char sep) {
    std::string out;
    for (size_t i = 0; i < fields.size(); ++i) {
        if (i) out += sep;
        out += fields[i];
    }
    return out;
}

int main() {
    std::mt19937 rng(20260827);
    std::uniform_int_distribution<int> len(0, 64);
    std::uniform_int_distribution<int> kind(0, 2); // 'a', 'b', ','

    for (int trial = 0; trial < 10000; ++trial) {
        std::string s(len(rng), 'a');
        for (char& c : s) {
            int r = kind(rng);
            c = (r == 0) ? 'a' : (r == 1) ? 'b' : ',';
        }

        auto fields = split_csv(s);

        assert(join(fields, ',') == s);              // round-trip
        size_t commas = 0;
        for (char c : s) if (c == ',') ++commas;
        assert(fields.size() == commas + 1);          // cardinality
        for (const auto& f : fields)
            assert(f.find(',') == std::string::npos); // purity
    }
}

The buggy implementation fails on trial 1. The round-trip property catches "a,,b". The cardinality property catches "a,". Two lines of assertion logic outperform a dozen hand-written examples.

Property checks are not a replacement for unit tests. They replace the belief that example-based tests cover the input space. The generator explores combinations a human reviewer would never type: 40 consecutive commas, a single comma, a comma at every position.

Layer 2: deterministic fixtures

Random generation has a blind spot. A given seed may never hit a specific edge case. Fixtures close that gap. A fixture is a named input with an expected result, stored in a table:

struct Fixture {
    const char* name;
    std::string input;
    size_t expected_fields;
};

const Fixture fixtures[] = {
    {"empty",        "",                      1},
    {"single_comma", ",",                     2},
    {"two_commas",   ",,",                    3},
    {"trailing",     "a,",                    2},
    {"leading",      ",a",                    2},
    {"nul_byte",     std::string("a\0b", 3),  1},
    {"huge",         std::string(1000000, 'a'), 1},
};

The fixture table is the part of the suite a human should read. It documents intent: empty input, boundary positions, embedded NUL bytes, size limits. When the next agent patch changes behavior, the table tells you which behaviors are load-bearing.

A useful rule: every bug found by a property check becomes a fixture. Add the minimal failing input to the table before fixing the code. This prevents regression and gives the next reviewer a concrete example instead of a probabilistic one.

Layer 3: the flaky freeze

The third layer is a process rule, not a code rule. Flaky tests destroy the signal that layers 1 and 2 produce. If a test fails without a code change, it is quarantined, not deleted. The quarantine entry carries an expiry date:

{
  "quarantined": [
    {
      "test": "split_csv.huge",
      "first_flake": "2026-08-27",
      "expires": "2026-09-03",
      "reason": "timeout on loaded CI runner"
    }
  ]
}

The freeze has two parts. First, no new tests are merged while the flaky rate exceeds the threshold — for example, 1% over the last 7 days. Second, a quarantined test that expires without a root-cause fix is deleted, not extended. A flaky test without an expiry date is a permanent lie in your suite.

The gate runner mechanics are a separate topic. Here the freeze is one layer of a larger contract. The flaky rate is a number, so the freeze is objective:

flaky_rate = flakes_in_7_days / total_runs_in_7_days

If the rate is above threshold, the merge gate stops accepting agent patches. The gate runner does not judge whether the patch is good. The suite is untrustworthy, so no patch can be trusted.

Merge order

Run the layers in this order — cheapest and most informative first:

1. Property checks — fast, catch invariant violations.
2. Fixtures — deterministic, catch documented edge cases.
3. Existing unit suite — slower, catches regressions in known behavior.
4. Flaky gate — blocks merge if the suite is unreliable.

The order matters. Property checks fail fast with a minimal counterexample. Fixtures tell you which documented behavior broke. The unit suite tells you what regressed. The flaky gate tells you whether any of the previous three results are believable.

Where this runs

The full loop — patch generation, property harness, fixture table, and flaky gate — needs two things: a model endpoint and a CI runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are the two availability facts this workflow relies on; generation and verification both run on the free tier. The contract itself is independent of the tool that produced the patch. If you already have a model endpoint and a runner, the same three layers apply unchanged.

Limitations

Property checks sample the input space; they do not prove absence of bugs. A property that encodes the wrong invariant will pass on a correct implementation. The cardinality property above, for example, assumes the CSV dialect has no quoted commas. If the agent adds quoting, the property must change with it.

Fixtures rot. A table without a reviewer is a list of opinions. The flaky freeze can be gamed by lowering the threshold or extending expiries. And the random loop is reproducible only if the seed is fixed — use a constant seed in CI, never std::random_device.

Who should not use this

Teams without a CI runner get nothing from the freeze; there is no gate to block. Patches that touch code with no stable input/output relationship produce vacuous property checks. One-off scripts do not justify the contract's overhead. Use this where the patch is long-lived and the failure cost is real.

Next time an agent hands you a green suite, run the three layers before you merge. The green checkmark is the hypothesis. The contract is the test.