Finley Zhou

Posted on Aug 27

The Agent Patch Gate: Properties, Fixtures, and a Flake Freeze

#testing #cpp #ai #agents

An agent patch arrives with tests, and the tests pass. That is the baseline, not the verdict.

The agent wrote the code and the tests from the same context, so blind spots survive in both. In a previous post on this account, mutation testing showed 2 of 4 injected faults survived an agent's passing suite. The tests were honest. They were just narrow.

This post is a three-layer gate for that situation: property checks that encode invariants, fixtures that encode failure history, and a freeze that stops flaky tests from laundering a bad patch. I run it on every agent patch I review, and the whole loop fits on a free server.

Layer 1: Property checks, not example checks

An example test proves one path. A property check proves a rule across thousands of paths.

Take a ring buffer. The agent's tests will cover push(1); push(2); pop() == 1. That is fine. It does not cover wrap-around, full-buffer policy, or size bookkeeping after 50,000 mixed operations.

The property harness below encodes three rules: size never exceeds capacity, FIFO order survives any interleaving, and a full buffer rejects pushes. It replays 50,000 random operations against a reference model.

#include <cassert>
#include <random>
#include <vector>

void check_ring_buffer_properties() {
  std::mt19937 rng(0xC0FFEE);         // fixed seed: reproducible failures
  std::vector<int> model;             // reference model
  RingBuffer<int> rb(8);

  for (int i = 0; i < 50'000; ++i) {
    if (std::uniform_int_distribution<int>(0, 1)(rng)) {
      int v = i;
      bool ok = rb.push(v);
      if (rb.full()) {
        assert(!ok);                  // rule: reject when full
      } else {
        assert(ok);
        model.push_back(v);
      }
    } else {
      int out = -1;
      bool ok = rb.pop(out);
      if (model.empty()) {
        assert(!ok);                  // rule: pop on empty fails
      } else {
        assert(ok);
        assert(out == model.front()); // rule: FIFO order
        model.erase(model.begin());
      }
    }
    assert(rb.size() == model.size()); // rule: size invariant
  }
}

Run it under sanitizers. That is where memory bugs surface, not in the logic.

clang++ -std=c++20 -fsanitize=address,undefined -O1 \
  -I src tests/property_ring_buffer.cpp -o /tmp/prop
/tmp/prop && echo "properties: PASS"

The fixed seed matters. A failing iteration must be reproducible, or the agent will "fix" a ghost.

Layer 2: Fixtures are the project's memory

Properties catch unknown paths. Fixtures catch known wounds.

Every bug you fix should leave behind a fixture: a recorded operation sequence that once broke the code. The harness replays it against the current implementation. If the fixture fails, the regression is back.

Fixture	Operation sequence	Bug it encodes
`empty_pop`	`pop` on empty buffer	Underflow returned garbage
`full_then_push`	fill to capacity, push again	Full policy was ignored
`wrap_around`	2×capacity pushes with interleaved pops	Linear indexing, no wrap
`single_element`	push, pop, repeat	Size bookkeeping drift

Store fixtures as plain files, one operation per line.

# tests/fixtures/ring_buffer/wrap_around.ops
push 1
push 2
push 3
pop
push 4
pop
pop
pop

Replaying them is a loop, not a framework.

./build/replay_fixtures tests/fixtures/ring_buffer && echo "fixtures: PASS"

Agent patches must pass every fixture. The agent did not write these; the project's failure history did. The corpus grows every time a bug escapes, so the gate gets stricter without any new tooling.

Layer 3: The flake freeze is a veto

A flaky test from an agent patch is a red flag, not a maintenance task. Run the agent's test suite ten times on the free server and compare exit codes.

for i in $(seq 1 10); do
  ./build/agent_tests > /tmp/run_$i.log 2>&1
  echo $?
done | sort | uniq -c

Ten identical exit codes means stable. Anything else means the outcome depends on luck. Freeze the test: mark it non-blocking, file a ticket, and reject the patch until the root cause is identified.

The freeze here is a veto. It says the agent's test cannot vote on this patch. A freeze without an expiry date becomes permanent tech debt, but the first decision is simpler: flaky equals unmerged.

The full pipeline

The agent generates the patch and its tests. MonkeyCode's free model access is enough for this step; the gate will judge the patch quality.
Run the property harness under ASan/UBSan. Any assertion failure rejects the patch with a named rule.
Replay the fixture corpus. Any failure rejects the patch with a named regression.
Run the agent's tests ten times. Any flake freezes the test and rejects the patch.
Only then does a human review the diff — for design, not for obvious bugs.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The whole loop is cheap. Properties run in seconds, fixtures are tiny files, and ten test runs fit in a coffee break. MonkeyCode's free server option covers it without a paid plan. The expensive resource is not compute; it is the decision of what to trust.

Limitations

This gate is not a substitute for review. It is a filter that makes review cheaper.

Do not use it as the only defense for distributed systems; integration and load testing still own that layer. Do not expect fixtures to predict new failure classes; they only remember old ones. And ten runs will miss rare flakes — the freeze is a signal, not a proof.

Use it where the code has invariants you can name and a history you can record. That covers most C++ data structures, parsers, and protocol handlers. It covers very little UI code, where the interesting properties are visual and behavioral.

If you are about to merge an agent patch on the strength of its own tests, run these three layers first. The properties are the point. The fixtures are the memory. The freeze is the veto.

DEV Community