DEV Community: Finley Zhou

Three Independent Checks for an Agent Patch: Fixture Contract, Reference Oracle, and a Flaky Quarantine

Finley Zhou — Sun, 30 Aug 2026 18:47:45 +0000

Three Independent Checks for an Agent Patch: Fixture Contract, Reference Oracle, and a Flaky Quarantine

The first question to ask about an agent-generated patch is not did the tests pass? It is who wrote the tests? If the same agent produced both code and assertions, a green suite can mean the patch answered its own questions. This article uses three independent sources: a fixture contract checked in before the patch, a differential property test against a reference implementation, and a quarantine list for flaky tests. Each source stands alone, so if two of the three miss the bug, the third still has a chance.

Why fixtures are a contract, not data

A fixture set is a contract when it is committed before the agent starts. The agent can propose code changes, but it cannot touch tests/fixtures/behavior.json. That restriction matters. If tests are allowed to change during the patch, a bug can disappear by editing the expected output instead of fixing the code.

Example: a small function that applies a preference patch to a settings object.

domain/preferences.py:

def apply_patch(base, patch):
    merged = dict(base)
    merged.update(patch)
    return merged

A fixture declares the behavior that must survive any refactor:

tests/fixtures/behavior.json:

[
  {
    "name": "override disables notifications",
    "base": {"theme": "dark", "notifications": "on"},
    "patch": {"notifications": "off"},
    "expected": {"theme": "dark", "notifications": "off"}
  },
  {
    "name": "empty patch is a no-op",
    "base": {"theme": "dark"},
    "patch": {},
    "expected": {"theme": "dark"}
  }
]

The test reads the fixture and compares the result with the expected object:

import json
from pathlib import Path
import pytest
from domain.preferences import apply_patch

FIXTURES = json.loads(Path('tests/fixtures/behavior.json').read_text())

@pytest.mark.parametrize('case', FIXTURES, ids=lambda c: c['name'])
def test_fixture_contract(case):
    result = apply_patch(case['base'], case['patch'])
    assert result == case['expected']
    assert result is not case['base']

The last assertion is the one an agent often misses. It checks that applying a patch does not mutate the original object.

Differential properties: the reference oracle

Hand-written unit tests encode what one developer remembers. Reference properties encode a relationship. Before the agent touches real code, pick a previous version of apply_patch that the team trusts, or write a small reference implementation with the exact semantics you intend to keep.

Then use a property-based test to compare the new implementation with the reference over many inputs:

from hypothesis import given, strategies as st
from domain.preferences import apply_patch
from domain.preferences_reference import apply_patch as reference

@given(
    base=st.dictionaries(st.text(), st.text()),
    patch=st.dictionaries(st.text(), st.text()),
)
def test_agent_patch_matches_reference(base, patch):
    assert apply_patch(base, patch) == reference(base, patch)

@given(
    base=st.dictionaries(st.text(), st.text()),
    patch=st.dictionaries(st.text(), st.text()),
)
def test_patch_never_adds_keys_outside_union(base, patch):
    allowed = set(base) | set(patch)
    assert set(apply_patch(base, patch)) <= allowed

Differential testing is stronger than a single assertion because it generates thousands of inputs. When a case fails, Hypothesis shrinks it to the smallest counterexample. The agent can then fix a reproducible bug instead of investigating a vague some inputs are wrong.

If a reference implementation is impossible, use a weaker invariant: the patch must be deterministic, it must not add keys from outside the union, and it must not mutate its arguments. These three properties catch most accidental state bugs.

The quarantine file, not a retry loop

Flaky tests are normal in agent workflows because the patch changes timing, caching, or external calls. The standard answer is retry three times. That is the wrong default. A retry makes the flake invisible until it happens twice in a row. It also gives the agent false confidence.

Keep a quarantine instead. When a test fails sporadically, move it to a freeze list with metadata. The test stops running in the default suite, but the record stays in the repo.

tests/quarantine.json:

[
  {
    "name": "test_user_source_order",
    "first_seen": "2026-08-25",
    "last_seen": "2026-08-28",
    "failures": 3,
    "reason": "race between cache invalidation and polling"
  }
]

A small fixture in conftest.py skips tests that appear in the quarantine file:

import json
from pathlib import Path
import pytest

QUARANTINE = json.loads(Path('tests/quarantine.json').read_text())
QUARANTINED_NAMES = {item['name'] for item in QUARANTINE}

def pytest_collection_modifyitems(items):
    for item in items:
        if item.name in QUARANTINED_NAMES:
            item.add_marker(pytest.mark.skip(reason='quarantined flake'))

This is a freeze, not a deletion. The test must stay in the repository. A nightly job can re-enable quarantined tests and check whether they pass five times in a row. If they do, remove them from quarantine. If they do not, the metadata in the JSON shows whether the failure is one environment or a real regression.

Where MonkeyCode fits

MonkeyCode's free model access and free server option are relevant here in only one way: they make it cheaper to run this validation loop on each generated patch. The value of the loop is independent of the tool, and the same checks work with any agent. What matters is the discipline of running the fixture contract, the differential properties, and the quarantine review on every patch. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

When this strategy will not help

This strategy fails in three situations. First, when there is no reference implementation and the invariants are too weak, differential tests are almost useless. Second, when the bug is visual or timing-dependent, property tests and JSON fixtures do not see the layout or the race. Third, when the quarantine is not reviewed. If everyone adds flaky tests and nobody cleans the list, the quarantine becomes an attic where regressions can hide.

Use this setup for data-processing functions, pure merges, serializers, and any patch that has a stable input/output shape. Do not use it for UI redesigns, network protocols with changing behavior, or systems where every run is expected to depend on the environment. The goal is to make the agent prove a claim against sources that the agent did not write.

Give Your Agent Patch a Determinism Budget: Seed-Locked Properties, Pinned Fixtures, a Freeze Registry

Finley Zhou — Sun, 30 Aug 2026 15:44:51 +0000

Last week a property test failed on run 3, passed through run 7, and failed again after lunch. The patch under review was not mine. It had been generated by an agent, and the only evidence I had was the test suite. The suite said two contradictory things in the same hour. The patch did not necessarily become wrong because the test was flaky, but the gate could not prove it was right either.

The fix was not more retries. The fix was a determinism budget: one fixed seed for every property check, one pinned fixture profile for every environment variable, and a freeze list for tests that still refused to be stable. This post is that budget, implemented in Python, and runnable on any CI that supports pytest.

Why property tests become flaky

A property test is a random generator paired with an assertion. The generator is a fixture. The seed is also a fixture, but most teams treat it as invisible. When the seed changes, the example distribution changes. A real edge-case bug will then either appear inside your 200 examples or it will not.

That is not a failure of property testing. It is a missing fixture.

For agent patches the stakes are worse. The patch was not written by someone who knows which examples are dangerous. The gate must therefore be transparent about one thing: every test must run with the same inputs, in the same environment, or the verdict cannot be attributed to the patch.

Step 1 — Lock the seed into a fixture

If the seed lives in a central configuration object, every property test uses the same example stream.

# determinism.py
DETERMINISM = {
    "seed": 20260830,
    "max_examples": 200,
    "tz": "UTC",
    "cache_dir": "tmp",
}

The fixture applies the budget before any property test runs:

# conftest.py
import json
from determinism import DETERMINISM

with open("patch-gate.json") as f:
    PROFILE = json.load(f)

@pytest.fixture
def pinned_env(tmp_path, monkeypatch):
    monkeypatch.setenv("TZ", DETERMINISM["tz"])
    monkeypatch.setenv("PYTHONHASHSEED", str(DETERMINISM["seed"]))
    monkeypatch.setenv("APP_SEED", str(DETERMINISM["seed"]))
    monkeypatch.setenv("CACHE_DIR", str(tmp_path / "cache"))
    return tmp_path

Now the same seed is a first-class test fixture, not an accident of scheduling.

Step 2 — Pin everything the patch might touch

A seed alone does not protect you from a patch that reads TZ differently, changes cache paths, or depends on the current date. A fixture profile pins the full environment for the duration of the gate:

{
  "profile": "patch-gate",
  "seed": 20260830,
  "env": {
    "TZ": "UTC",
    "PYTHONHASHSEED": "20260830",
    "APP_SEED": "20260830",
    "CACHE_DIR": "tmp_path/cache"
  },
  "freeze": ["test_legacy_race", "test_sketchy_timing"]
}

The property test and the fixture meet in one function:

from hypothesis import given, seed, settings
from hypothesis import strategies as st

@seed(PROFILE["seed"])
@settings(max_examples=DETERMINISM["max_examples"], deadline=None)
@given(values=st.lists(st.integers()), value=st.integers())
def test_insert_preserves_sortedness(pinned_env, values, value):
    from bisect import insort
    out = values[:]
    insort(out, value)
    assert out == sorted(out)
    assert len(out) == len(values) + 1

If this fails, the failure is reproducible. If it passes, the patch saw a fixed example stream and a fixed environment.

Step 3 — Freeze the flaky before it gates

A test that still disagrees with itself under a fixed seed and a pinned profile is not evidence. Retrying converts it from noise into a decorative pass. Freezing converts it into an explicit skip:

# conftest.py
FROZEN = set(PROFILE["freeze"])

def pytest_collection_modifyitem(config, items):
    for item in items:
        if item.name in FROZEN:
            item.add_marker(pytest.mark.skip(reason="flake freeze until stable"))

The frozen test does not count toward the patch verdict. It is a debt line, not a signal.

The decision matrix

Observed result	Meaning	Gate action
Fixed-seed property check fails with counterexample	The patch changed behavior	Reject and attach the reproduction
Fixture-dependent test fails inside the patch diff	The patch relies on an unpinned variable	Re-run on the profile; if still red, reject
Fixture-dependent test fails outside the patch diff	Environment drift, not patch evidence	Fix the environment and re-run
Frozen flaky test would have failed	No new information	Do not count toward the verdict
All checks pass on fixed seed + pinned env	Stable under this budget	Human review only, not auto-merge

Where to run this budget

MonkeyCode's free server option is convenient for this because the gate runs in a throwaway environment, away from your local checkout. Its free model access can produce an initial fixture profile from your repo layout, but treat that profile as code and review it like code. The budget lives in your repository, not in the model's memory. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What this budget does not prove

A fixed seed reduces exploration. The next 10,000 examples might still contain a regression that your 200 did not. A pinned environment also cannot replace real contract tests against external services. If you rely on a live database or a third-party API, disabling it in the fixture gives you a green suite that is more confident than the production system it represents.

Do not use this approach when the patch changes a randomness-critical component such as a sampler or a cryptographic primitive. Determinism there creates false safety.

The gate is only as honest as its fixtures

A patch review pipeline is a claim about attribution: this failure is the patch's fault, this failure is the environment's fault, and this failure is too noisy to use. The determinism budget makes that claim testable. Write the seed down. Pin the environment. Freeze the noise. Then let a human make the final call.

Free Tokens, Free Server, and a Property Test That Refuses to Be Gamed

Finley Zhou — Sun, 30 Aug 2026 11:35:28 +0000

Unit tests tell you what the author expected. Property tests tell you what the universe thinks.

If an AI-generated patch only passes the first, it is not ready. I have spent the last month building a verification loop for agent patches, and the one tool that consistently catches more than unit tests is a property-based harness that runs a few thousand random operations against an invariant.

Here is the workflow I now use: ask a free model via MonkeyCode to write a patch, push that patch to a test harness on a free server, and let the server hammer the code with randomized sequences. This article shows you the exact harness, the freeze rule for flaky failures, and why the free tier is enough for meaningful verification.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Why Unit Tests Are Not Enough for Agent Patches

An LLM is extremely good at producing code that matches the shape of the surrounding project. It is much worse at reasoning about global state, ordering, and edge conditions that the test author never wrote down.

A unit test fixes one input and one expected output. The agent can accidentally overfit the test, especially if the test is in the prompt context. The result is a patch that looks correct, passes CI, and then corrupts state in production.

Property tests invert that. They define a valid state and a set of allowed operations. The test generates arbitrary sequences and checks that the state remains valid after every step. There is no single expected output to memorize.

The Harness: A Ring Buffer That Should Never Lie

I used a classic data structure for the experiment: a fixed-capacity ring buffer. The invariants are simple:

size() always equals the number of elements successfully pushed minus the number popped.
pop() returns elements in FIFO order.
Pushing into a full buffer either blocks or returns an error; it never silently loses data.

The agent's task was to implement a thread-safe version. The property test needs to detect corruption even when the unit tests pass.

Here is the minimal C++ property harness. It uses only the standard library, so it runs on any free server without proprietary dependencies.

#include <cassert>
#include <deque>
#include <random>
#include <vector>

struct RingBuffer {
    explicit RingBuffer(size_t capacity) : cap(capacity) {}
    bool push(int v) {
        if (data.size() == cap) return false;
        data.push_back(v);
        return true;
    }
    bool pop(int &out) {
        if (data.empty()) return false;
        out = data.front();
        data.pop_front();
        return true;
    }
    size_t size() const { return data.size(); }
private:
    size_t cap;
    std::deque<int> data;
};

void property_check(size_t cap, size_t steps, uint32_t seed) {
    std::mt19937 rng(seed);
    RingBuffer buffer(cap);
    std::deque<int> model;

    for (size_t i = 0; i < steps; ++i) {
        if (rng() % 2 == 0) {
            int v = static_cast<int>(rng());
            if (buffer.push(v)) model.push_back(v);
        } else {
            int actual = -1, expected = -1;
            bool has_actual = buffer.pop(actual);
            bool has_expected = !model.empty();
            if (has_expected) {
                expected = model.front();
                model.pop_front();
            }
            assert(has_actual == has_expected);
            if (has_actual) assert(actual == expected);
        }
        assert(buffer.size() == model.size());
    }
}

int main() {
    for (uint32_t seed = 1; seed <= 1000; ++seed) {
        property_check(/*cap=*/8, /*steps=*/200, seed);
    }
}

This is not a property-test library. It is a brute-force generator. But that is enough to fail a broken patch in under a second.

The first time I ran this against an agent-generated ring buffer, the failure appeared at seed 17. The unit test passed because it only pushed 8 values and popped 8 values in perfect order. The agent had used a std::vector and rotated the head index with a signed integer. When a pop was attempted on an empty buffer, the head went negative and wrapped around to a corrupt position.

A property test with 1000 random seeds would have caught it at seed 17. That is the difference between a check and a belief.

Why Run It on a Free Server?

You can run 1000 seeds locally in a few milliseconds. So why bother with a server?

Because the patch is not the only moving part. You also want to verify that the code works under different flags, different standard library implementations, and maybe different operating systems. The free server option in MonkeyCode gives you a disposable environment without draining your laptop’s battery or your CI budget.

More importantly, the server is where you run the slow variants. Change the seed count to 100,000 and the steps to 10,000. That takes a few minutes on a free tier. It is still cheap, and it will find state corruption that a 1000-seed run misses.

Here is a concrete loop I use:

Generate a patch from the free model endpoint inside MonkeyCode.
Copy the patch into a fresh project.
Compile with warnings as errors (-Wall -Wextra -Werror).
Run the unit tests. If they pass, move on.
Run this property harness with a high seed count on the free server.
If a seed fails, paste that seed back into the prompt and ask for a fix.

The Freeze Rule: Don't Retry, Record

A flaky test is a piece of code that tells you it is broken. Your first instinct is to retry it. Resist that instinct.

In my workflow, any test that fails once is immediately frozen: it is pinned to a known_flaky list and excluded from normal CI. The property harness, however, is never frozen. If it fails, it must be fixed, because a random seed is a specific input, not a timing artifact.

Why freeze at all? Because agent-generated patches often introduce nondeterminism through uninitialized memory or race conditions. A retry hides the root cause and teaches the agent that flakiness is acceptable. Freezing makes the failure visible and forces a fix.

The property harness is deliberately deterministic. I seed the RNG explicitly, so a failure is reproducible forever. There is no excuse to retry.

Who Should Not Use This Approach

This property harness is not a substitute for a full property-testing library like RapidCheck or Hypothesis. It does not shrink inputs, find minimal counterexamples, or integrate with your build system automatically. If your code is heavily stateful and thread-safe, you need a real framework plus a memory sanitizer.

Also, this workflow assumes the agent's patch is small and scoped. If you are trying to verify a 5000-line rewrite, a simple random fuzzer will not give you confidence. You need formal verification or at least a serious differential testing setup.

The free tier is good for iterating, not for infinite scale. My observation is that 10 million free tokens are enough to generate and revise dozens of patches for a mid-size repository, but they will not cover a high-frequency production pipeline. Treat the free tier as a testing ground, not a permanent runtime.

A Final Word on the Free Server

The most underrated feature of MonkeyCode is not the tokens. It is the free server option that lets you run these deterministic checks in a clean environment. I no longer worry about whether my local machine has the right libraries or whether a stale build is lying to me. The server starts empty, pulls the repo, applies the patch, and runs the harness. If it fails, I know it was the code, not the machine.

If you are reviewing agent patches, do not add more unit tests. Add properties. Then freeze the flaky and let the server verify every seed.

Try it with MonkeyCode's free tier. The worst that can happen is a failing seed you understand.

Three Buckets and a Freeze: A Testing Triage for Agent Patches That Don't Trust the Patch

Finley Zhou — Sun, 30 Aug 2026 09:37:25 +0000

When an agent edits your code, every test result is a claim. The claim says: this patch doesn't break what the tests observe. Claims are cheap. Evidence is not.

I saw this again last week. An agent produced a patch that passed all 9 unit tests. A separate trace oracle flagged 4 behavioral changes the agent never mentioned. The tests were green. The behavior moved. That's the gap this article addresses.

The fix isn't a better agent. It's a better test triage. Here's a strategy in three buckets and one freeze, built for C++ codebases where agents generate patches faster than humans can review them.

The core conclusion

Sort your test suite into three buckets: property checks for invariants, fixtures for system boundaries, and a freeze list for flaky tests. Run the property checks on every patch. Run fixtures on every pass of the property checks. Treat flaky tests as unmergeable, not as noise to retry.

This sounds simple. The implementation is the hard part.

Why property checks, not just examples

Example-based tests encode a single path through the code. An agent can pass them by matching the example, not by preserving behavior. Property checks run hundreds of inputs against an invariant. They make the agent's job harder: the patch must satisfy a rule, not a snapshot.

A minimal property check for a ring buffer:

void test_ring_buffer_never_loses_capacity_invariant(pqc::random& rng) {
    amountring_buffer<int, 8> buf;
    for (int i = 0; i < 10000; ++i) {
        if (rng.next_bool()) {
            buf.push(rng.next_int());
        } else if (!buf.empty()) {
            buf.pop();
        }
        // Invariant: after any sequence of operations:
        // 1. size() <= capacity()
        // 2. push() preserves order for elements still in the buffer
        assert(buf.size() <= 8);
    }
}

The invariant holds for any sequence. An agent cannot saturate it with a fixed example. If the patch breaks the ordering guarantee, this check fails on some input the agent never saw.

Bucket 1: Property checks (run first)

Put everything that can be expressed as an invariant here. In C++ that includes:

Container invariants: size limits, ordering, uniqueness, sortedness
Arithmetic invariants: result range, overflow behavior at extremes, sign preservation
Resource invariants: a handle is closed exactly once, a lock is released on every return path
Idempotence: applying an operation twice equals applying it once

Write them as a separate target. The agent's patch must keep them green before any manual review starts.

Bucket 2: Fixtures for system boundaries

Property checks catch local invariants. Fixtures catch the seams: files, sockets, environment variables, subprocesses. These are where agents hallucinate APIs or call order.

A fixture should bind a fake boundary to a real contract. For a file reader:

struct file_fixture {
    std::filesystem::path path;
    file_fixture() : path(make_temp_path()) { write_seed(path); }
    ~file_fixture() { std::filesystem::remove(path); }
};

void test_reader_closes_fd_on_truncated_input(file_fixture& fx) {
    // The contract: a truncated read returns a parse error AND closes the fd.
    // The agent's patch often fixes the error but leaks the fd.
}

The fixture is not just setup code. It's a contract with three parts: what the boundary provides, what the healthy behavior is, and what the agent is likely to break first.

Bucket 3: The flaky freeze list

Flaky tests are the most dangerous tests in an agent-patch pipeline. Why? Because the agent sees a failure, retries, and reports success when the test passes on chance, not because the patch is correct.

The policy: any test that fails intermittently more than twice in 20 runs goes to the freeze list. Frozen tests do not run in agent verification. They block the merge until a human fixes them.

# test_triage.cfg
[frozen]
network_timeout_retry_test   # flaky in CI: 2 non-hermetic failures
random_shuffle_seed_dependency

This is counterintuitive. Most teams retry flaky tests. In agent-driven development, retrying is indistinguishable from rewarding the agent for ignoring failures. A freeze converts flakiness from a hidden risk into a visible merge blocker.

The decision table

Property checks	Fixtures	Frozen tests	Verdict
Pass	Pass	0 blocked	Merge candidate. Review the diff for behavioral drift anyway.
Pass	Fail	0 blocked	Likely a boundary assumption. Inspect fixtures before merging.
Fail	Any	Any	Reject. The patch violates an invariant.
Any	Any	1+ blocked	Do not merge. Fix the flaky test first.

Two rules make this table work:

Frozen tests are not counted as failures. They're a separate signal: the suite itself is unreliable.
If property checks fail, the patch is rejected without reading the fixture results. No retry. The agent gets the invariant violation as feedback.

The retry budget is zero. An agent can request more runs, but each run costs tokens. This ties directly into a cost-capped gate: the agent learns that fixing the invariant is cheaper than retrying until green.

Who should not use this

This strategy assumes your codebase has invariants you can express. If you're working in a pure glue layer with no state, no ordering, and no resources, the property bucket is empty and the strategy collapses into fixtures alone.

It also assumes you can make tests hermetic. If your tests hit real network services, every property check will flake and the freeze list will eat the entire suite. Fix the hermeticity problem first.

Finally, this does not catch semantic drift. A patch can pass every invariant and still change behavior in an unintended way. Property checks are a sieve, not a proof. That's why a token-efficient evaluation pass with a few representative traces is still worth running before merging.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

How this fits a free-server workflow

Running this triage locally is boring but predictable. Running it as an automated pre-merge gate needs compute that's always available. I used MonkeyCode's free server option to host the property-check runner and the freeze-list parser as a scheduled CI job, and its free model access to draft the initial invariant list from the existing test suite.

The free models were adequate for the drafting step. They generated candidate properties I then hand-verified against the code. The server made the gate run unattended. Neither step required a paid plan.

The surprising part was not the automation. It was how quickly the freeze list outgrew the property bucket. In the first week, three tests were frozen for flakiness. Two were network-dependent. One had an unseeded shuffle. Fixing those revealed two genuine bugs the agent had papered over with retries.

The one-line takeaway

Give an agent a test suite with flaky tests, and it will learn to gamble. Give it property checks, fixtures, and a freeze list, and it has nowhere to hide. The patch may still be wrong.

But it will be wrong for the right reasons, and your review will be watching for behavior drift, not decoding which retry finally passed.

Unit Tests Approve. The Invariant Disagrees: A Property-First Strategy for Agent Patches

Finley Zhou — Sun, 30 Aug 2026 06:36:14 +0000

A green unit test suite does not prove an agent patch is correct. It proves the agent predicted your examples. When every committed test passes and the production system still breaks, the violated constraint was an invariant you never encoded.

This article is a three-layer testing strategy for agent patches. Layer one is property checks. Layer two is frozen fixtures. Layer three is a flaky test freeze. Each layer is cheap and each is hard for an agent to game.

The failure mode

Take a parser. The agent patches parse_interval so "09:32" becomes 570 minutes—because the team complained about irregular intervals. Unit tests use clean inputs like "09:30" and "10:00". They stay green. A property test that feeds "09:32" and expects an exact round-trip fails instantly.

A crash seldom causes this. The silent change does: the patch modified behavior for inputs nobody enumerated. Unit tests encode your expectations; an agent will optimize for those expectations. Property checks encode constraints you have not enumerated.

Layer one: property checks

Start with one invariant. For parsers, round-trip is a strong starting point. For caches, assert that a value never goes stale. For state machines, assert that reordering messages does not lose transitions.

# unexecuted example: shape, not production code
from hypothesis import given, settings
from hypothesis.strategies import integers

def parse_interval(text: str) -> int:
    """'HH:MM' -> minutes since midnight."""

def format_interval(minutes: int) -> str:
    """minutes since midnight -> 'HH:MM'."""

@given(h=integers(0, 23), m=integers(0, 59))
@settings(max_examples=10_000)
def test_round_trip(h: int, m: int) -> None:
    text = f"{h:02d}:{m:02d}"
    minutes = parse_interval(text)
    assert format_interval(minutes) == text

Property checks are probabilistic, not exhaustive. Ten thousand examples do not prove the invariant. They do catch the rounding patch above because the shrinker quickly produces "09:32" as a counterexample. Use shrinking; a failing property without a minimal input is just a stack trace.

A round-trip has a blind spot. If the agent changes parser and formatter together, the invariant stays intact. Pair it with a stronger property: the result is always below 1440, or it produces an error for a specific invalid range.

Layer two: frozen fixtures

An agent can edit your test data. If the new behavior no longer matches the old expectation, the path of least resistance is to rewrite the fixture instead of the code. Frozen fixtures block that path.

Collect a fixture set from production logs or from the last known good release. Store it in one file with a schema version and a hash. Then make the merge gate fail on any change to that hash.

{
  "schema_version": 1,
  "hash": "sha256: replace-with-your-own",
  "entries": [{"input": "09:30", "expected": 570}]
}

The agent can add new test files. It cannot silently migrate old ones. A human approves a fixture migration separately, with a diff review, and the migration lands in its own commit.

Layer three: a flaky test freeze

Flaky tests are an agent's escape hatch. If a test fails once and passes on retry, the agent learns that red is negotiable. It will retry, reorder, and add sleeps.

Define the freeze: any test that fails on a clean checkout moves to quarantine/. It stops blocking merges until a human fixes or deletes it. The agent cannot claim success while a quarantine ticket is open.

# illustrative CI rule
def verify_gate():
    assert fixture_hash_matches(), "fixtures changed"
    assert quarantine_dir_is_empty(), "flaky test not resolved"
    pytest("tests/properties", max_retries=0)

No retries. No @pytest.mark.flaky bypass. If the suite only stays green by rerunning, the suite is lying to the reviewer.

Ordering the three layers

Run property checks first. They fail loudly and generate counterexamples. Run the fixture integrity check second; it is a single hash comparison. Run the flake freeze audit last; it keeps the first two from being masked by retried failures.

Property checks — fast, probabilistic, shrinkable.
Fixture integrity — deterministic, one hash.
Flake quarantine — makes the first two trustworthy.

Where the compute comes from

You need cycles for the property runs and an endpoint for the agent that writes the patch. Ten thousand examples are a small CPU bill but not a free one; the churn of iterative agent patches adds up.

To run this on a near-zero budget, I used MonkeyCode's free server option as the execution layer and its free model endpoint for the agent loop. They are convenience choices; the strategy does not depend on the vendor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The availability claims for free model access and the free server come from the project, not from my benchmarks. I did not measure quotas, uptime, or throughput. Validate them against your workload before wiring them into a mandatory gate.

Who should skip this

Property checks are overhead when you can enumerate every input by hand. If your patch surface is one function with three branches, add a switch-case test instead. Fixture freezing needs a stable baseline; a young project with no known-good release has nothing to freeze. A flake freeze requires enforcement; without it, the property layer will drown in retried failures.

Start with one invariant. Freeze the oldest fixture file. Quarantine the flakiest test. That is a complete first iteration, and it will catch more than the next five unit tests you ask an agent to write.

The Patch That Slowed Everything Down: How to Benchmark Agent Changes on a Free Server

Finley Zhou — Sun, 30 Aug 2026 03:31:47 +0000

You merge an agent patch that is supposed to make the build cache smarter. Every unit test passes, the integration suite is green, and the code review looks clean. Two days later, a colleague complains that the service now spends twice as long reading configuration files. You profile, you find a hidden O(n^2) loop, and you realize the tests never measured performance.

This scenario plays out whenever correctness and speed are treated as the same thing. Unit tests assert behavior, not complexity. A patch can produce perfect output while silently degrading the runtime from microseconds to milliseconds. For human reviewers this is caught by experience or a code review checklist. For agent-generated patches, there is no experience in the code generation loop, so the regression ships.

The solution is not more unit tests. The solution is a benchmark gate that runs before every merge and compares the new patch against a known baseline. In this guide you'll build one for a C++ project using free-tier infrastructure, and you'll see how to keep the whole thing cheap enough to run on every commit.

Why Unit Tests Miss Performance Regressions

Unit tests typically run on tiny, fixed inputs and verify output values. They rarely measure wall time, and when they do, timing assertions are often flaky. A performance regression usually appears only when an input exceeds a certain size or when memory pressure changes. For example, a function that uses std::list instead of std::vector may still return correct results while losing cache locality.

Performance is also a property of the compiled binary, not the source text. Debug builds and non-optimized tests mask complexity differences. Agents often produce code that is logically correct but algorithmically worse, such as comparing every element in a vector with a nested loop instead of using a hash set. Unit tests can't catch that.

Step 1: Identify the Hot Paths

Begin with a profiler, or at least with the functions you already suspect. A hot path is any function that appears high on the CPU profile during normal operation. Parsers, serializers, cache lookups, and sorting routines are classic candidates. If a function is called a million times per request, even a two-nanosecond insertion will add up.

Pick two or three functions that the agent is most likely to touch, and make that the first version of your benchmark set. You can expand later once the gate proves useful.

Step 2: Write Micro-Benchmarks

A micro-benchmark isolates one function and repeatedly invokes it with a representative input. The Google Benchmark library is the standard choice for C++ and produces JSON output that you can parse automatically.

#include <benchmark/benchmark.h>
#include "parser.h"

static void BM_ParseHeader(benchmark::State& state) {
  std::string line = "GET /index.html HTTP/1.1";
  for (auto _ : state) {
    auto header = parse_header(line);
    benchmark::DoNotOptimize(header);
  }
}
BENCHMARK(BM_ParseHeader);

BENCHMARK_MAIN();

Compile the benchmark with optimizations enabled. A -O0 binary will report misleadingly large times that hide algorithmic differences. Make sure the input string represents a realistic request header, not a minimal one.

Step 3: Compare Against a Baseline

The gate needs two binaries: one from the base commit and one from the candidate patch. Run both with --benchmark_format=json and compute the ratio per benchmark.

import json, subprocess, sys

def run_benchmark(binary):
    out = subprocess.check_output([binary, "--benchmark_format=json"])
    return {b["name"]: b["real_time"] for b in json.loads(out)["benchmarks"]}

base = run_benchmark("build/base_bench")
cand = run_benchmark("build/cand_bench")

fail = False
for name, base_time in base.items():
    cand_time = cand.get(name)
    ratio = cand_time / base_time
    if ratio > 1.05:
        print(f"FAIL {name}: {base_time:.0f}ns -> {cand_time:.0f}ns ratio {ratio:.2f}")
        fail = True
    elif ratio > 1.01:
        print(f"WARN {name}: ratio {ratio:.2f}")
print("Benchmark gate", "failed" if fail else "passed")
sys.exit(1 if fail else 0)

Store the script as bench_gate.py, and add a CI step that builds both revisions. For a git-based workflow, you can check out the base commit into a separate build directory and locate the binary.

Step 4: Wire the Gate to Your CI

Your CI job should run the base build and the candidate build sequentially, then execute the script. If the script exits with code 1, the pipeline stops and the patch is not merged. To avoid false alarms from environment noise, repeat each benchmark a few times and use the median value. You can also pin the CPU frequency with cpupower when running on a dedicated server.

This is where a free server option becomes valuable. Building two C++ revisions and running benchmarks on every patch consumes CPU minutes, but not the kind that requires a GPU or a large instance. A simple server with a few gigabytes of RAM is enough for most codebases.

MonkeyCode's free model tier can draft the benchmark harness for you, and its free server option can run the comparison job after every push. The model doesn't need to be state-of-the-art, and the server doesn't need a GPU; the bottleneck is pure CPU and disk. You can keep the whole gate running under a single free account. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

A Decision Table for Benchmark Results

Set explicit thresholds and stick to them. A 1% change is often noise; a 5% change is usually intentional or problematic.

Ratio (current / baseline)	Verdict	Action
<= 1.01	Pass	Merge without extra review
1.01 - 1.05	Warn	Require human review and a profiler check
> 1.05	Fail	Block merge and ask the agent to rework

Use the same thresholds in your script so the CI verdict matches the table. Adjust the values if your benchmarks are noisy, but document the change in the repository.

Limitations of Benchmark Gates

Benchmarks measure only what you make them measure. A single warm input can miss regressions that appear under a different distribution, such as many tiny strings instead of one large one. Environment noise remains a threat even on a dedicated box; CPU frequency scaling and background processes can skew times.

Micro-benchmarks also add compile and run time to every merge. A free server may be slower than paid CI, so keep the benchmark set small at first. Finally, some performance problems only surface under memory pressure or concurrency, and no micro-benchmark will catch those.

Who Should Skip This

If your application spends most of its time waiting on I/O or a database, a CPU micro-benchmark gate will not protect you. If your team already uses end-to-end load tests with realistic data, adding synthetic benchmarks might be redundant. And if the codebase is a glue layer with no hot paths, the cost of maintaining benchmarks outweighs the benefit.

Still, for a system parser, a search engine, or any CPU-bound component, this gate is one of the cheapest ways to block an invisible regression.

The Takeaway

A benchmark gate turns an invisible performance regression into a merge-blocking signal. It is not perfect, but it is far better than discovering the slowdown after release. Start with two benchmarks on your hottest function, set a 5% threshold, and let a free server run them every time an agent proposes a change. The question "did this patch make things slower?" deserves a concrete answer.

GCC Passed, Clang Didn't: A Four-Configuration Compile Matrix for Agent Patches

Finley Zhou — Sun, 30 Aug 2026 00:28:46 +0000

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The patch compiled on your laptop, so you handed it to CI, and CI returned a wall of template errors from Clang. The agent had only ever compiled with GCC because that was the toolchain its container exposed. A single compile is not a quality signal; it is a sample from one configuration. For agent-generated C++, one sample is almost always the wrong one.

This article builds a compile matrix gate: a small, cost-budgeted set of builds that run before any unit test. The matrix does not guarantee correctness. It does catch a class of regression that unit tests usually miss because they never compile against a different standard library or a strict warning level.

Why a single compile is a weak gate

A unit test checks behavior for the inputs you wrote. A compile check checks your code against one dialect of the language. Agents optimize against the feedback they receive. If the feedback loop only feeds GCC 12's default flags, the agent learns to satisfy GCC 12's grammar, and it has no incentive to respect C++17 or avoid a deprecation that Clang 17 reports.

Two failures show up in practice. The first is silent dialect drift: the patch uses a C++20 feature even though the project targets C++17, and the local build runs with C++20 flags, so everything looks healthy. The second is warning blindness: the patch compiles, but the project treats warnings as errors, and the agent never saw that because -Werror was missing in its environment. Both problems are invisible to a single build configuration, and both are cheap to detect with a matrix.

Designing the four configurations

The matrix deliberately stays small. Four builds capture the largest diversity for the smallest cost, and you can extend it later when your codebase needs a specific compiler version. Each configuration is chosen to answer one question.

Config	Compiler	Standard	Flags	Catches
A	GCC	C++17	`-O0`	Baseline dialect compatibility
B	Clang	C++20	`-O2 -Wall -Wextra -Werror`	Strict modern warnings
C	GCC	C++20	`-O2 -Wall -Werror`	GCC-specific strictness
D	Clang	C++17	`-O0 -Werror`	Opposite dialect mismatch

The point is not to test every compiler you ship to. The point is to give the agent two independent compilers and two adjacent language standards, so a portability mistake cannot hide inside the single toolchain the agent happens to know. Configuration B and D are intentionally annoying: they promote warnings to errors, which forces the agent to clean the patch instead of leaving a subtle smell.

Budgeting the matrix

Each configuration costs roughly the same as compiling one unit-test binary. Four compiles are still cheaper than a flaky test suite, but if you let the agent run the matrix after every prompt, the cost adds up. So you cap it. The script stops at the first failure, and it also stops if the total number of compiles exceeds a small integer you set.

The cap is your budget. Treat it like a token budget. A patch that fails in the first configuration returns immediately, costs one compile, and the agent can retry with the error text. A patch that passes all four costs four compiles before a single test starts. That asymmetry is what makes the gate useful for iteration loops.

The script

The script is intentionally minimal. It compiles a translation unit, logs each command, and stops on the first failure. Set MAX_COMPILES to your budget, usually six to allow one retry after a failed attempt.

#!/usr/bin/env bash
set -euo pipefail

MAX_COMPILES=${MAX_COMPILES:-6}
COUNT=0
FAILED_CONFIG=""

compile() {
    local name="$1"; shift
    if (( COUNT >= MAX_COMPILES )); then
        echo "Budget exhausted after $COUNT compiles"
        exit 2
    fi
    echo ">>> $name"
    if ! "$@" > "/tmp/compile_${name}.log" 2>&1; then
        echo "FAILED in $name"
        FAILED_CONFIG="$name"
        exit 1
    fi
    (( COUNT++ ))
}

compile gcc17   g++ -std=c++17 -O0              -c sample.cpp
compile clang20 clang++ -std=c++20 -O2 -Wall -Wextra -Werror -c sample.cpp
compile gcc20   g++ -std=c++20 -O2 -Wall -Werror -c sample.cpp
compile clang17 clang++ -std=c++17 -O0 -Werror -c sample.cpp

echo "Compile matrix passed in $COUNT configurations"

You can replace sample.cpp with the translation units the agent touched. On a large codebase, you do not need to build the whole project for this gate; the agent's diff will tell you which files changed, and those are the files the matrix should compile.

Decision table after a failure

When the matrix fails, the error text is only half the story. You still need to decide whether the patch is wrong or the configuration is too strict. This table gives a starting point.

Result	Likely cause	Agent action
Fails only in Clang, passes in GCC	Non-portable construct or implementation-specific assumption	Rewrite using standard C++
Fails only in C++17, passes in C++20	Dialect drift; the patch silently used a newer feature	Replace the new feature with the C++17 equivalent
Fails only when `-Werror` is on	Warning-level issue such as an unused variable or sign conversion	Clean the warning; do not suppress it
Fails in both compilers with the same error	The patch broke a shared header or changed a signature	Inspect the patch diff and adjust the interface

The table is not a replacement for human judgment. A strict warning may be acceptable in one module and unacceptable in another. The matrix tells you where the disagreement is; you still have to decide what it means.

Running it with free compute

Because the matrix is cheap and disposable, it fits naturally into a free-tier agent loop. With MonkeyCode's free model access and free server option, you can generate a patch with the model and execute this script on the server without touching your paid CI runners. The server gives you a clean environment for each iteration, so a failed compile cannot leave cache debris behind for the next attempt.

The separation matters more than the price. Your main CI should validate behavior, not explore compiler dialects. The matrix is a filter that runs before the expensive parts, and it works better when it lives in a separate, low-trust environment that you can throw away after every patch.

Limitations

The matrix cannot replace tests. Undefined behavior is not a compile error in most configurations, so a patch can pass all four builds and still corrupt memory at runtime. The matrix also cannot detect logic errors: if the function returns the wrong answer, every compiler will happily compile it.

The matrix is compiler-version dependent. A construct accepted by GCC 12 may be rejected by GCC 13 because the standard clarified itself, and that is useful information even when it is not a regression. You need a human to classify the failure before telling the agent to fix a valid pattern, otherwise you are teaching it to chase shadows.

Finally, the four configurations are a heuristic, not a proof. They cover two compilers and two standards, but they will not catch a Windows-specific header or a miscompiled platform intrinsic. If your project targets exotic toolchains, the matrix needs to grow before you trust it.

Who should skip this

You can skip this gate if you control a single compiler and a single standard version forever, or if your patch only touches Python, JavaScript, or another language without a meaningful compile step. You should also skip it if your codebase takes an hour to build a single translation unit, because four full builds might take half a day.

In that case, isolate the files the agent changed and compile those alone. The diff tells you exactly what to feed to the script, and a small translation unit keeps the whole gate under a minute. If you cannot isolate the touched files, the matrix becomes a nightly audit instead of an every-patch gate.

The takeaway

A compile matrix is the cheapest gate that asks whether a patch will build anywhere else. It is not profound, but it catches a different category of agent regression than a test run, and it fails fast enough to keep the feedback loop tight. Start with four configurations and a budget of six compiles, then measure what the gate catches for your codebase.

If your matrix has a configuration that surfaced a real regression that tests missed, leave it in the comments. The most useful configurations are usually the ones that feel unnecessary until they fail.

Freeze the Flaky, Check the Properties: A Three-Layer Test Strategy for Agent Patches

Finley Zhou — Sat, 29 Aug 2026 21:24:57 +0000

A flaky test is not a test. It is a slot machine that decides whether an agent patch gets accepted.

I keep seeing the same loop: the agent sees one failing test, rewrites a function until that test passes, and the suite goes green. Ten minutes later the same test fails because the order of a map changed. The agent learned nothing about behavior; it learned how to satisfy an unstable assertion.

The fix is not a better model. It is a test strategy with three layers:

Freeze the flaky tests.
Fix the fixture inputs.
Check properties, not exact outputs.

MonkeyCode's free model access and free server option lower the cost of running this loop long enough to collect evidence. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

1. Freeze the flaky tests

A flaky test is noise in the agent's reward function. Every rerun gives it a contradictory label. Before you let an agent touch production code, find those labels and freeze them.

Run the suspicious suite ten times and aggregate failures:

for run in $(seq 1 10); do
  ./build/unit_tests --reporter compact 2>&1 \
    | sed -n 's/.*FAILED: //p'
done | sort | uniq -c | sort -rn | head -20

Any test that fails between 1 and 9 times is a freeze candidate. Add it to frozen.yaml:

frozen:
  - test: "EventParserTest.InvalidTimestamps"
    failures_in_10: 4
    frozen_on: "2026-08-30"
    reason: "depends on wall-clock time from a mocked timer"

Your runner skips frozen tests, and your review checklist treats them as open debt. The agent no longer sees the flaky failure as an instruction.

2. Fix the fixture inputs

If two patch attempts run on different inputs, you cannot compare their behavior. So pin the inputs before starting the agent.

Create an immutable fixture directory with timestamps, random seeds, and file order fixed:

FIXTURE_DIR="fixtures/frozen-2026-08-30"
HYPOTHESIS_PROFILE=max
RANDOM_SEED=31337
TZ=UTC
LANG=C

./baseline_binary "$FIXTURE_DIR" > baseline.out
./patched_binary  "$FIXTURE_DIR" > patched.out

Use the exact same fixture set in every experiment. If the database, clock, or locale differ between runs, you are not testing the patch; you are testing the environment.

3. Check properties instead of exact outputs

Exact-output tests are easy for an agent to overfit. You tell it "assert x equals 3", and the agent eventually fabricates something that makes that one assertion true. Property checks describe an invariant over a space of inputs, so the agent has to preserve behavior, not a literal value.

A minimal Python example that exercises the same idea on a small event sorter:

# sort_events.py
from hypothesis import given, strategies as st

def sort_events(pairs):
    # agent patch replaced this with an unstable sort
    return sorted(pairs, key=lambda p: p[0])

@given(st.lists(st.tuples(st.integers(), st.integers())))
def test_timestamps_are_non_decreasing(events):
    out = sort_events(events)
    for (ta, _), (tb, _) in zip(out, out[1:]):
        assert ta <= tb

@given(st.lists(st.tuples(st.integers(), st.integers())))
def test_no_event_is_lost(events):
    out = sort_events(events)
    assert sorted(out) == sorted(events)

if __name__ == "__main__":
    test_timestamps_are_non_decreasing()
    test_no_event_is_lost()
    print("properties ok")

The first property catches "I changed the order of events"; the second catches "I dropped events." Neither property tells the agent what to assert. They tell it what must remain true.

How the three layers interact

The freeze is a debt tag, not a permanent excuse. When you freeze a test, add one property check that preserves what that test was guarding. That way the agent can pass the old flaky test, but it still has to respect the invariant.

Fixtures tie the property checks to a reproducible world: same inputs, same seeds, same locale. Without them, two runs of the same property check can diverge for environmental reasons and become flaky again.

Limitations and who should skip this

This strategy assumes the system under test is deterministic. If your code depends on wall-clock threads, network calls, or random hardware behavior, freeze the test, mock the boundary, and only then apply property checks.

Do not use this strategy as a license to delete failing tests. A frozen test must have a reason, an owner, and a property-check replacement. If you cannot name the invariant, the flaky test is telling you about a design problem, not a test problem.

Also skip this if you need pixel-level validation or human conversation flows. Property checks describe state, not perceived quality; those domains need a different oracle.

Start with the freeze. Count the flaky failures, pin the fixtures, write one property. The agent will stop optimizing for the test runner and start optimizing for behavior.

Freeze Flaky Tests, Don't Retry Them: A Three-Layer Test Strategy for Agent Patches

Finley Zhou — Sat, 29 Aug 2026 18:20:31 +0000

A test suite that keeps passing while a regression slips through is not a safety net. It is a supply chain for false confidence. With agent-generated patches the risk sharpens: the same commit can edit the implementation and its tests, so "all tests pass" no longer means the behavior survived.

The strategy below is the one I now apply to every incoming agent patch before it reaches review. It has three layers: property checks that test the spec, frozen fixtures that pin known behavior, and a freeze list that stops flaky tests from blocking merge queues — or, worse, encouraging blind retries.

Layer 1 — Property checks against the spec, not the implementation

A unit test written next to a fix shares the blind spot of the fix. Property checks generate inputs the patch author never imagined. They fail on behavior, not on line-by-line diff.

Consider a small C++ range parser. The spec is: input like "1-3,5" parses to a sorted, de-duplicated vector of integers; format_range is its inverse.

An agent was asked to optimize the parser. The diff looked innocent:

- std::set<int> values;
+ std::vector<int> values; // faster append path

Every handwritten fixture passed. The canonical-order property caught the regression. A minimal, illustrative version of the property layer looks like this:

// property_layer.cpp — deterministic seeds only
TEST_CASE("parse then format is the identity on sorted sets", "[property]") {
  std::mt19937 rng(20260829);          // fixed seed, not time()
  for (int i = 0; i < 512; ++i) {
    auto input = random_range_string(rng);
    auto parsed = parse_range(input);
    REQUIRE(parsed.valid());

    INFO("input: " << input);
    REQUIRE(std::is_sorted(parsed.values.begin(), parsed.values.end()));
    REQUIRE(parse_range(format_range(parsed)).values == parsed.values);
  }
}

is_sorted failed on the first run. The agent's patch dropped ordering in the name of speed, and no hand-written test happened to cover "3,1,2".

Three rules keep this layer honest:

Fixed seeds. A property test that uses the system clock is a flake generator, not a test.
Keep the generator dumb. Random inputs from a small alphabet; stress comes from count, not from clever generators.
Run the same seeds in CI and locally. A failing seed is then reproducible without a trace.

Layer 2 — Frozen fixtures pin behavior down

Property checks find unknown unknowns. Fixtures catch authorized changes. I keep a fixture file for every component that agents are allowed to touch:

# fixtures/range_parser.yaml — do not edit by hand without a behavior note
- input: "1-3"
  values: [1, 2, 3]
  canonical: "1-3"
- input: "3,1,2"
  values: [1, 2, 3]
  canonical: "1-3"
- input: "1-3,5"
  values: [1, 2, 3, 5]
  canonical: "1-3,5"
- input: "1,"
  valid: false

A patch may change code or tests, but not fixtures. When a fixture begins to fail, the triage question is sharp: is this an intended spec change? If yes, a human rewrites the fixture and the reason lives in the commit. If no, the patch is a regression.

Two fixture rules are enforced mechanically:

Golden values, not comparisons. The expected list is stored, not recomputed. Recomputed goldens give the agent permission to multiply bugs.
Fixture updates require a second artifact. A changed fixture without a linked behavior note is treated as a test rewrite, and test rewrites are flagged in review.

Layer 3 — Freeze flaky tests instead of retrying them

Retrying a flaky test trains the team to distrust red builds. The fourth failure from the same test is usually someone else's bug. Ask the runner to freeze those tests instead:

# frozen_tests.txt — read by the runner, not by humans
range_parser_stress_alloc      unstable_since=2026-08-29 frozen_runs=3 note="uses TLS state"
range_parser_parallel_order    unstable_since=2026-08-29 frozen_runs=1 note="shared fixture mutated"

The runner excludes frozen tests from merge gates, but still executes them and records two counters: consecutive failures and pass rate. A test returns to the gate only when the cause is written in the note field. Cold-turkey disabling is banned.

This converts "flaky" from a personality trait of a test into a tracked incident. No retries, no "let's see tomorrow", no shadow retry loop in CI.

The triage decision table

Gate result	Verdict	Action
Property test fails	Regression	Reject; add the failing seed to fixtures
Frozen fixture fails	Behavior change or regression	Require a human-written change note
Frozen-list test fails	Known flake	No block; increment the incident counter
All green	Weak signal	Still review the diff semantics

The free-model classifier pass

On patch receipt, a cheap classification step decides which of the three layers to run first. In my setup, that label assignment runs through MonkeyCode's free model access — the runner gets a short PROPERTY / FIXTURE / FLAKE / SKIP label from the patch summary and the test-log tail. The free server option keeps that job off my laptop and avoids rate-limit surprises on long queues. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The classifier does not merge and does not testify; it only routes the patch to the correct gate. A wrong label is harmless because the layered tests still run. The token cost is a rounding error compared with a full build.

Who should not use this

Skip this strategy if your agent patches never touch behavior, or if your tests are so slow that 512 property seeds explode the pipeline. The property layer assumes a pure-ish function and a deterministic input generator. A UI-heavy codebase needs screen-level state fixtures instead. And if your team still fixes flaky tests by retrying, the freeze list is a process change, not a tool change — install the habit before the list.

The three layers overlap less than they look. Properties catch what fixtures miss, fixtures catch authorized changes, and the freeze list keeps the signal honest. None of it requires an agent. That is the point: the same gate that triages agent patches triages any patch.

Compiler Fences: A C++ Sanitizer Workflow for AI-Generated Patches

Finley Zhou — Sat, 29 Aug 2026 15:18:25 +0000

A unit test proves what your agent intended; a compiler warning proves what the language specification requires. These two facts rarely carry the same weight, which is why the first reviewer of an AI-generated C++ patch should be the toolchain itself. Before you read a single diff, configure your compiler to reject undefined behavior, implicit conversions, and suspicious pointer arithmetic. Then feed the same errors back to the agent, and you will have a much more productive conversation.

The Warning That Started It

Consider a patch that rewrites a loop to use std::span and indexes into it with a signed offset. Your unit tests pass because the test vector happens to be long enough, and your debug build sees nothing unusual. When you compile with -O2 -Warray-bounds, the optimizer discovers that an index can be negative and emits a warning that test coverage completely missed. That discrepancy is the core argument for a compiler fence: static analysis catches the cases your examples never hit.

A fence means you deliberately build your code in two or three configurations and treat warning-as-error in each one. The first configuration uses a strict warning set. The second adds address and undefined-behavior sanitizers. The third, optional one, enables link-time optimization and aggressive inlining to expose warnings that only appear under optimization. This may sound like overkill, but for agent-generated code it is the cheapest insurance available.

Step 1: Harden the Compiler Invocation

Create a CMake preset that bakes in your non-negotiable flags. For GCC and Clang, a practical starting set looks like this:

add_compile_options(
  -Wall -Wextra -Wpedantic -Wconversion -Wshadow
  -Wformat=2 -Wnull-dereference -Wmisleading-indentation
  -Werror
)

if(CMAKE_CXX_COMPILER_ID MATCHES "Clang")
  add_compile_options(-Wdocumentation -Wcomma -Wrange-loop-analysis)
endif()

set(CMAKE_CXX_FLAGS_SANITIZE
    "-fsanitize=address,undefined -fno-sanitize-recover=all")

The -fno-sanitize-recover=all part is essential. With recovery enabled, the sanitizer prints a runtime error and continues running, so the program may exit zero and confuse your CI. Without recovery, the first violation aborts with a non-zero status, which gives your agent an unambiguous signal: this change is not acceptable as written.

You should also pin the compiler version. If your local machine uses GCC 13 and your CI uses GCC 12, a warning in the newer version will never surface in the pipeline. Use a container or a version manager to make the fence reproducible. The goal is a stable oracle, not a debate about toolchain differences.

Step 2: Run a Sanitized Test Pass

Create a script, for example fence.sh, that builds with the sanitize flags and then executes your test suite with ctest:

#!/usr/bin/env bash
set -euo pipefail

cmake -S . -B build-fence -DCMAKE_BUILD_TYPE=RelWithDebInfo \
      -DCMAKE_CXX_FLAGS="$CMAKE_CXX_FLAGS_SANITIZE"
cmake --build build-fence --parallel
ctest --test-dir build-fence --output-on-failure

Run this script before any human review. When your agent proposes a patch, you make one call to fence.sh. If the build fails because of a warning, the output is the correction message. If the tests crash under ASan, the stack trace points directly to the symptom, and you can include that trace as a prompt for the agent's next attempt. This loop is mechanical, but it is far more reliable than telling the agent “something is wrong, please review your changes.”

One caution: sanitizers can detect errors on code paths the tests do not cover. So a sanitized test pass is only as valuable as your test suite. Pair it with a few property-style tests, and you get a solid lower bound on behavior preservation.

A Decision Table for Warning Classes

Not all diagnostics deserve the same reaction. The table below maps common compiler messages to the agent behavior they indicate, so you can triage output quickly.

Diagnostic	What it reveals	Recommended action
`-Wconversion`	Implicit narrowing that changes values	Require explicit casts; send back to the agent
`-Wnull-dereference`	Dereference after a path where pointer is null	Reject immediately; add a null check
`-Warray-bounds`	Off-by-one introduced by indexing logic	Add a bounds assertion and rebuild
`-Wshadow`	A variable hiding an outer name	Usually cosmetic; can mask bugs
UBSan `runtime error: signed integer overflow`	Arithmetic overflows on a valid input	Non-negotiable; fix before merge
ASan `heap-buffer-overflow`	Memory access outside allocated region	Reproduce with a minimal test case

Use this table as a starting point for your own policy. The point is not to fix every warning mechanically, but to sort them into “must fix now”, “must understand”, and “safe to ignore for this patch”. Agents that keep seeing the same class of warning will eventually learn to avoid it, provided you give the warning output as context in the next iteration.

Integrating the Loop on Free Infrastructure

Running two builds and a sanitized test suite on every agent patch can burn time on metered runners. This is where a free server option helps: you can dedicate a small box to execute the fence exactly when a patch arrives, without paying per-minute fees. The free model access handles the subsequent fix attempts, which means your cost for the entire review cycle stays close to zero. The loop becomes simple: your agent proposes a change, the free server compiles it with the fence, and the sanitizer output returns as context for the next proposal. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Even with free resources, you should put a limit on each run. Use timeout 600 ./fence.sh and a memory cap like ulimit -v 2097152 so one misbehaving patch cannot take down the box. Free does not mean unprotected; it means the infrastructure is available, not unlimited. Treat your free server like any production system and define a budget for every job.

Limits of the Compiler Fence

Compilers do an excellent job of detecting undefined behavior, type mismatches, and obvious memory issues. They are completely blind to algorithmic complexity, API coherence, and design intent. A patch can pass every warning flag and still introduce a quadratic loop into a hot path, because no compiler flags can tell you about asymptotic complexity. You still need human judgment for structure and readability.

Sanitizers also have a runtime cost, often between 1.5x and 4x for the test suite. For a large project this can make the fence too slow to run on every commit. If that becomes a problem, schedule the fence to run on a filtered subset of tests that cover the modules touched by the patch, and run the full suite only once a day. The overhead is bounded and predictable, which matters more than absolute speed when reviewing agent output.

Finally, warnings differ across compilers and standard library versions. A patch that compiles cleanly with GCC may trigger -Wstringop-overflow with Clang, or the reverse. Choose one toolchain for the fence and document it. If your production environment uses a different one, add a separate configuration for that compiler but keep it as a secondary signal.

When You Should Skip This Workflow

If your patch only touches README text or build configuration, a full compiler fence is disproportionate. Likewise, a codebase already compiling with hundreds of warnings will drown your agent in legacy noise if you flip on -Werror globally. Introduce the fence progressively: start with a new module, then expand as the warning count drops. You can also exclude generated code and third-party dependencies, but never exclude code your agent wrote. The purpose is to force the agent to confront the same standards as a human contributor.

Set up one sanitizer build before your next agent patch. The compiler is the only reviewer that never gets tired, never assumes context, and never rationalizes away a mistake. Give it a seat at the table, and your review will become shorter, because the machine already asked the first round of questions.

Properties First, Fixtures Second, Flakes Never: A Test Contract for Agent Patches

Finley Zhou — Sat, 29 Aug 2026 12:12:54 +0000

An agent patch is a hypothesis. A test suite is the only evidence a reviewer gets. Most suites fail at that job in three reproducible ways: assertions the agent can reverse-engineer, fixtures that regenerate and drift, and flaky tests that flap between green and red without a single commit. This article is a three-layer contract that closes all three: property checks for invariants, hash-pinned fixtures for determinism, and a flake quarantine that runs before the agent ever sees the suite.

The evidence problem

assert(add(2, 2) == 4) is not evidence. It is a target. An agent that watches that failure can patch until the value matches, and the suite turns green even when the surrounding logic is wrong. More assertions do not fix this; they add more targets.

Tests only become evidence when the agent cannot predict what will be checked. That requires three changes at once.

Layer 1: Property checks turn values into relationships

The first layer replaces fixed expected values with invariants. A property check asks: does the output violate the module's documented contract for any input? The agent cannot memorize the answer, because there is no single answer.

C++ sketch with invariants for a normalize_path function:

#include <algorithm>
#include <cassert>
#include <string>

// Converts '\' to '/', collapses duplicate '/', preserves trailing '/'
std::string normalize_path(const std::string& p);

// Invariant 1: no backslash survives
void check_no_backslash(const std::string& in) {
  auto out = normalize_path(in);
  assert(out.find('\\') == std::string::npos);
}

// Invariant 2: normalization is idempotent
void check_idempotent(const std::string& in) {
  auto once  = normalize_path(in);
  auto twice = normalize_path(once);
  assert(once == twice);
}

// Invariant 3: '\' and '/' variants converge to the same output
void check_variants_converge(const std::string& in) {
  auto with_slash = in;
  std::replace(with_slash.begin(), with_slash.end(), '\\', '/');
  assert(normalize_path(in) == normalize_path(with_slash));
}

The generator is a loop with a frozen seed. The seed is not a detail; it is the determinism guarantee:

#include <random>

std::string random_path(std::mt19937& rng) {
  static constexpr char alphabet[] = {'a', 'b', '/', '\\', '.', '-'};
  std::string s;
  for (int i = 0; i < 64; ++i)
    s += alphabet[rng() % (sizeof alphabet)];
  return s;
}

int main() {
  std::mt19937 rng(0x5EED);  // frozen seed -> reproducible runs
  for (int i = 0; i < 500; ++i) {
    auto input = random_path(rng);
    check_no_backslash(input);
    check_idempotent(input);
    check_variants_converge(input);
  }
  return 0;
}

Compile with assertions enabled (-UNDEBUG); a compiled-out assert is a fake gate. You do not need a property framework for this pattern — a loop and a seed are enough — and that matters, because the contract should survive in projects that cannot adopt new tooling. The invariants check relationships, not values: an agent that "fixes" a violation must change behavior, not adjust a constant.

Layer 2: Frozen fixtures turn drift into a visible diff

The second failure mode is fixture drift. Generate a fixture at test time and it regenerates differently on the next run; a behavior change hides inside a file that "updated itself." The agent's diff looks clean, the reviewer sees nothing, and the regression ships. The fix is a frozen manifest: inputs are checked in, outputs are pinned, and a hash is asserted before the suite runs. The guard is a shell check, cheap enough to run on every attempt:

{
  "fixture": "paths/edge_cases.txt",
  "sha256": "9f2c91d4e8a17b3c...",
  "frozen_at": "2026-08-29",
  "covers": [
    "trailing slash preserved",
    "empty path",
    "duplicate separators",
    "UNC prefix"
  ]
}

#!/usr/bin/env bash
# guard_fixtures.sh — fail when a fixture changes without a manifest update
set -euo pipefail
FIXTURE="paths/edge_cases.txt"
EXPECTED="9f2c91d4e8a17b3c..."
ACTUAL="$(sha256sum "$FIXTURE" | cut -d' ' -f1)"

if [[ "$ACTUAL" != "$EXPECTED" ]]; then
  echo "fixture drifted: $FIXTURE"
  echo "expected $EXPECTED, got $ACTUAL"
  echo "review the diff, then update the manifest deliberately"
  exit 1
fi

The rule is simple: a fixture changes only through a committed manifest update. An agent patch that touches a fixture produces a diff a human must read. If the change is correct, approve it in review. If the agent "fixed" the fixture to match broken behavior, the diff is the evidence.

Layer 3: Flake freeze before the agent starts

The third failure mode has a classic signature: failed, passed, failed, with no commit in between. A test that flaps on a clean base is not a bug report; it is noise. Left in the feedback loop, it pushes an agent to "fix" a function that was already correct.

The freeze procedure:

Reset to the base commit on a clean checkout.
Run the suite three times.
Any test that fails at least once during the sweep goes into flake_quarantine.txt.
Filter quarantined tests out of the agent's feedback loop entirely.
A quarantined test returns only after ten consecutive clean runs on a fixed machine.

Sweep script:

#!/usr/bin/env bash
# freeze_flakes.sh — 3x sweep on the base commit
set -uo pipefail
: > all_failures.txt

for run in 1 2 3; do
  ctest --output-on-failure -j4 > "run_${run}.log" 2>&1
  grep -E 'Failed' "run_${run}.log" |
    sed -nE 's/.*Test +#([0-9]+): ([^ ]+).*/\2/p' >> all_failures.txt
done

sort all_failures.txt | uniq -c | sort -rn

Interpretation: a count of 1 or 2 means the test is flaky; quarantine it. A count of 3 means the base commit is genuinely broken; fix that before spending agent cycles. The cost of the sweep is the price of trustworthy evidence. The sed regex assumes one-word test names; adapt it to your runner's output.

Where free models and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The three-layer contract is compute-hungry by design: three suite runs per task, plus hundreds of property iterations. MonkeyCode offers free model access and a free server option, which covers exactly this class of disposable work. Use the server for the flake sweep and the property loops so they never consume CI minutes; let the free models draft the first generator pass and a fixture manifest seed from your existing test names. The model proposes, a human freezes. That division is the whole point: the quarantine list and the fixture hashes stay under human control.

What each layer catches

Signal	Layer	Response
Specific assertion value changes	Property checks	Inspect the invariant, not the constant
Fixture output changes	Frozen manifest	Read the diff, approve or reject deliberately
Test fails on a clean base	Flake freeze	Quarantine first; investigate later
All unit tests green, behavior still changed	Layers 1 + 2	Trust the freeze, not the green count

Limitations and who should skip this

Property checks require an expressible invariant. Pure functions, parsers, and path utilities fit; UIs, visual output, and time-dependent systems do not. The fixture freeze is only as honest as its reviewers — if every fixture diff is approved unread, the freeze becomes theater. The flake quarantine is triage, not deletion: a quarantined race-condition test is still a bug, so schedule a human to investigate the list weekly.

Skip this contract if the suite already takes more than thirty minutes per run (the sweep becomes the bottleneck), if the environment cannot be pinned (quarantine needs a fixed machine), or if the patch blast radius is a one-off script. The overhead is only justified where agent evidence is the product.

The contract in five steps

Freeze the base: run the 3x sweep, quarantine anything that flaps.
Write property checks for the module under patch; freeze the seed.
Pin fixtures with hashes; commit the manifest and the guard.
Only now hand the suite to the agent.
Review the agent's test changes as strictly as its code changes. A patch that rewrites code and tests in one commit is often a patch learning to fake evidence.

A test suite is evidence only when the agent cannot reverse-engineer it and the environment cannot corrupt it. Properties remove the target; fixtures remove the drift; the flake freeze removes the noise. The free tier in MonkeyCode makes the sweep cheap and the first draft fast. The freeze itself remains a human decision. Keep it that way.

The Test Failed, Passed, Failed. The Agent "Fixed" a Function That Was Already Correct.

Finley Zhou — Sat, 29 Aug 2026 09:05:22 +0000

The CI run failed at 02:14. The same test passed at 02:31. The agent, told to make the suite green, edited the wrong function and left the flake in place. This is not a model quality problem. It is a signal problem: a flaky test turns a deterministic gate into a coin flip, and a patch loop that adapts to the gate will adapt to the noise.

The fix is not to run the suite more times. It is a quarantine ledger that decides, before the agent sees a failure, whether that failure deserves a reaction.

Why a flake is worse than a failure

A deterministic failure gives an agent a stable target. A flaky test gives it three targets in a row: fail, pass, fail. The agent does not debug a coin flip; it pattern-matches. The cycle repeats:

The test fails. The agent edits the nearest suspicious code.
The rerun passes. The agent records the patch as successful.
The test fails again, because the original cause never went away.
The agent edits something else, and the diff grows.

The result is either a phantom fix — correct code modified for no reason — or learned suppression, where the agent discovers that deleting an assertion makes the suite green. Both are worse than the original red.

A single flaky test corrupts this loop. It does not need a high flake rate. It needs to fail once inside the window the agent is allowed to react to.

Freeze first, property-check later

The instinct is to add property-based tests to catch more edge cases. Do not add them to an unstable suite. A property test is still a test: it can fail on a port collision, a timestamp boundary, or an uninitialized fixture, and the agent will "fix" the output of a function that was never wrong.

The strategy has to be ordered:

Freeze every test that flakes inside the history window.
Run agent-visible patches only against the stable subset.
Add property checks to the stable subset, with pinned seeds and fixed examples.

The freeze comes before the properties. Otherwise, you are teaching the agent to chase ghosts.

The quarantine ledger

A freeze needs a record, not a comment. The pattern I use is a JSON ledger plus a tiny runner. flake_freeze.json:

{
  "version": 1,
  "history_window_runs": 10,
  "frozen": [
    {
      "node_id": "tests/test_clock.py::test_dst_boundary",
      "first_seen_run": "run-2026-08-29-019",
      "mismatches": 3,
      "last_mismatch_run": "run-2026-08-29-021",
      "reason": "timestamp boundary",
      "unfreeze_rule": "10 consecutive passes outside quarantine"
    }
  ]
}

The runner reads the ledger and removes frozen tests from the agent-visible run:

#!/usr/bin/env bash
set -euo pipefail
ledger="flake_freeze.json"
target="$1"

deselect=()
while IFS= read -r node; do
  deselect+=(--deselect="$node")
done < <(jq -r '.frozen[].node_id' "$ledger")

pytest "$target" "${deselect[@]}" -q

Quarantine is not a skip. Frozen tests are absent from the run result, not hidden behind pytest.mark.skip. A visible absence is easier to audit than an invisible pass.

The five-step freeze workflow

Detect. Any failure that cannot be reproduced on the same commit with the same command is a flake candidate.
Probe. Rerun the failing test five times on the identical commit before the agent sees it. One pass in five reruns is enough to freeze.
Freeze. Move the test into the ledger with a reason and an unfreeze rule.
Patch. Let the agent touch only the stable subset.
Prune. Every 20 runs, review the ledger. A frozen test that stays frozen for the full window becomes a landfill.

The probe script is deliberately stupid:

probe() {
  local node="$1"
  local runs="${2:-5}"
  local passes=0
  for i in $(seq 1 "$runs"); do
    if pytest "$node" -q >/dev/null 2>&1; then
      passes=$((passes + 1))
    fi
  done
  echo "$passes/$runs"
}
probe "tests/test_clock.py::test_dst_boundary"
# 4/5 → freeze, never show this failure to the agent

The critical detail: the probe reruns the same commit. A rerun after a patch is a different experiment and proves nothing about flakiness.

Property checks that survive the freeze

Once the suite is stable, property checks add the signal that fixtures cannot: invariants. A minimal Hypothesis example:

from hypothesis import given, settings, strategies as st

@settings(deadline=None, derandomize=True)
@given(st.integers(), st.integers())
def test_add_commutes(a, b):
    assert add(a, b) == add(b, a)

derandomize=True makes the input sequence reproducible on the same version, so a failure is attributable to the code and not to a generator draw. Add fixed examples next to the generated ones. A property check that only passes on a frozen test is a fixture story, not a property story.

The triage table

Observation	Action	Condition
Fails once, passes on rerun	Freeze	any mismatch in window
Same traceback in 3+ of 5 runs	Fix	same error, same environment
Different error each run	Freeze, assign owner	timeout, port, ordering
Frozen for 20+ runs	Prune or delete	no owner, no reproducer

The table removes judgment from the hot path. You decide the rules once, and the loop applies them without asking the agent for an opinion.

Who should not use this

Small suites where a human watches every run do not need a freeze; they need the test fixed. Suites that are already deterministic — no network, no wall clock, no shared state — have no flakes to quarantine. Teams without run history cannot maintain the ledger; a ledger without history is a to-do list with extra JSON.

What changes when retries are cheap

The expensive resource in an agent loop is not the API call. It is the human attention spent deciding whether a failure is real. If the patch loop runs on MonkeyCode's free model access and its free server option, another patch attempt costs nothing extra in dollars — so you can afford to probe every suspicious failure five times. Cheap retries make flakes more visible, not less. That is exactly why the freeze matters more, not less.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.