Properties, Fixtures, and a Flaky Freeze: A Three-Layer Gate for Agent Patches

An agent patch earns trust in exactly three layers: property checks, fixtures, and a flaky freeze. Each layer answers a different question. Properties ask whether invariants hold across the input space. Fixtures ask whether known behavior stayed identical. The freeze asks whether the gate signal is clean enough to trust the first two. One layer alone is a false sense of security.

The tests an agent writes are usually the prompt's examples, replayed as assertions. They prove the patch works on inputs the agent already saw. They say nothing about the space around those examples. A verification gate has to cover that space without a human writing a thousand cases by hand.

Why agent patches fail differently

Agent-written code fails in three patterns that human code rarely shows in the same combination:

1. Overfitting. The patch and its tests are generated from the same prompt. Passing tests only confirm the patch matches its own examples.
2. Edge hallucination. Boundary inputs are handled by plausible guesswork. The happy path works; the empty string, the trailing zero, the negative value do not.
3. Nondeterminism. The patch introduces ordering or timing dependence that a single test run cannot expose.

Each pattern needs a different countermeasure. That is the three-layer gate.

Layer 1: Property checks — invariants, not examples

Property-based testing has been around since QuickCheck (Claessen and Hughes, 2000). The idea is simple: instead of asserting on fixed inputs, generate many inputs and assert that an invariant holds for all of them. For a C++ gate, you do not need a framework. A fixed-seed generator and a loop are enough.

// propcheck.h — minimal property harness, no dependencies
#pragma once
#include <iostream>
#include <random>
#include <string>

template <typename InputGen, typename Fn>
int check_property(const char* name, int trials, InputGen gen, Fn fn) {
  std::mt19937 rng(20260827);  // fixed seed: the gate must be reproducible
  for (int i = 0; i < trials; ++i) {
    auto input = gen(rng, i);
    if (!fn(input)) {
      std::cerr << "[FAIL] " << name << " trial=" << i
                << " input=\"" << input << "\"\n";
      return 1;
    }
  }
  std::cout << "[PASS] " << name << " (" << trials << " trials)\n";
  return 0;
}

The fixed seed is deliberate. A gate that fails on trial 731 today and trial 402 tomorrow is useless for merge decisions. Reproducibility matters more than coverage in the gate; coverage gets a separate nightly run with varying seeds.

A concrete example: an agent rewrote parse_duration, which converts strings like "1h30m" into seconds. The agent's tests covered "1h", "30m", and "1h30m". The properties covered the space around them.

// gate.cpp — properties for the parse_duration patch
#include <cstdint>
#include <random>
#include <string>
#include "parse_duration.h"
#include "propcheck.h"

std::string random_duration(std::mt19937& rng, int trial) {
  std::uniform_int_distribution<int> h(0, 23);
  std::uniform_int_distribution<int> d(0, 59);
  std::string s;
  if (trial % 3 != 0) s += std::to_string(h(rng)) + "h";
  if (trial % 3 != 1) s += std::to_string(d(rng)) + "m";
  s += std::to_string(d(rng)) + "s";
  return s;
}

int main() {
  int failed = 0;

  // Property 1: round-trip through the canonical formatter.
  failed += check_property("roundtrip", 2000, random_duration,
    [](const std::string& s) {
      int64_t secs = parse_duration(s);
      if (secs < 0) return true;  // invalid input: nothing to round-trip
      return parse_duration(format_duration(secs)) == secs;
    });

  // Property 2: the value equals the sum of its tokens.
  failed += check_property("additive", 2000, random_duration,
    [](const std::string& s) {
      int64_t secs = parse_duration(s);
      if (secs < 0) return true;
      int64_t sum = 0, n = 0;
      for (char c : s) {
        if (c >= '0' && c <= '9') { n = n * 10 + (c - '0'); continue; }
        sum += n * (c == 'h' ? 3600 : c == 'm' ? 60 : 1);
        n = 0;
      }
      return secs == sum;
    });

  return failed == 0 ? 0 : 1;
}

Property 2 is the one that matters. It re-derives the expected value from the string and compares it to the patch's result. Unit-scale confusion — treating minutes as seconds, or hours as minutes — passes the agent's own examples and breaks this property on the first generated input.

Layer 2: Fixtures — pin down known behavior

Property checks catch invariant violations. They do not catch wrong-but-consistent behavior. If the patch is uniformly off by a factor of sixty, the round-trip property still passes. Fixtures exist for that gap.

A fixture is a checked-in corpus of real inputs with expected outputs, verified once by a human or a reference implementation. The agent's example tests are not fixtures; they are the prompt echoed back. The corpus has to come from outside the prompt.

# fixtures/durations.txt — input<TAB>expected_seconds
1h<TAB>3600
90m<TAB>5400
1h30m<TAB>5400
0s<TAB>0
1h0s<TAB>3600
-1m<TAB>-1

The runner is deliberately dumb:

// fixture_runner.cpp
#include <fstream>
#include <iostream>
#include <sstream>
#include <string>
#include "parse_duration.h"

int main(int argc, char** argv) {
  std::ifstream in(argv[1]);
  std::string line;
  int failed = 0, count = 0;
  while (std::getline(in, line)) {
    if (line.empty() || line[0] == '#') continue;
    std::istringstream ss(line);
    std::string input;
    int64_t expected;
    if (!(ss >> input >> expected)) continue;
    ++count;
    int64_t got = parse_duration(input);
    if (got != expected) {
      std::cerr << "[FAIL] fixture input=\"" << input
                << "\" expected=" << expected << " got=" << got << "\n";
      ++failed;
    }
  }
  std::cout << "[PASS] " << (count - failed) << "/" << count << " fixtures\n";
  return failed == 0 ? 0 : 1;
}

The fixture file is part of the repository, not part of the prompt. When behavior intentionally changes, the fixture changes in the same commit, and the diff is reviewable. A patch that changes fixture outputs without a matching explanation is rejected.

Layer 3: The flaky freeze — keep the signal clean

A gate with a flaky test has no signal. The property and fixture layers are only meaningful if the test run they sit in is deterministic. Two rules keep it that way.

Rule one: freeze pre-existing flakes. A test that fails intermittently for environmental reasons gets skipped with a recorded reason and a review date. A freeze without an expiry is a permanent blind spot, so the expiry is part of the freeze.

Rule two: never freeze a flake introduced by the patch. If a test was stable before the patch and flakes after it, that is a rejection signal. The patch introduced nondeterminism — ordering, timing, or uninitialized state. Freezing it would bless the regression.

The distinction is the whole point. The freeze protects the gate from noise the team already knows about. It does not protect the patch from scrutiny.

Reading the three signals together

Signal	Likely cause	Action
Property check fails	Invariant broken by the patch	Reject; report the generated input
Property passes, fixture fails	Wrong-but-consistent behavior	Reject, or require a reviewed fixture update
New test flakes after the patch	Patch introduced nondeterminism	Reject; do not freeze
Pre-existing test flakes	Environment or legacy flake	Freeze with expiry; keep the signal clean
All layers pass	Consistent with invariants and known behavior	Merge; run a nightly seed-varied property pass

The table is the gate. No single row decides a merge; the rows are read in order. Property checks run first because they are fast and dependency-free. Fixtures run second because they are deterministic but need I/O. The stability probe — running the agent's own tests five times — runs last, because it only makes sense on a patch that already passed the first two layers.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The gate above is cheap to run, which is exactly why it fits a free tier. The candidate patch comes from an agent session on MonkeyCode's free model access. The gate runs on MonkeyCode's free server option. That separation matters: verification runs on a different machine from generation, and rejected candidates never touch your main CI pipeline.

The workflow is three steps:

1. Generate the candidate patch with the free model access.
2. Push the patch to the free server and run the gate: properties, fixtures, then the stability probe.
3. Merge only when all three layers pass. Reject with the failing input or fixture line when they do not.

The free server option is a fit for a gate this size. A large fixture corpus or long property runs may exceed what a free tier is meant for — check the current limits before relying on it.

Limitations and who should not use this

The gate verifies consistency, not correctness against an external spec. Property checks need a specifiable invariant; if the patch has no algebraic structure, there is nothing to check. Fixed-seed runs are reproducible but can miss seed-dependent failures, which is why the nightly varied-seed pass exists. Fixtures rot when behavior changes and nobody updates the corpus; the gate then blocks legitimate changes instead of catching bad ones.

Do not use this approach when the patch is a one-line change and the existing suite is already green. Do not use it when there is no reference behavior to fixture. Do not use it if the team will not maintain the corpus — a stale fixture file is worse than no gate, because it produces confident false failures.

The three layers exist because agent patches fail in three different ways. Properties cover the input space. Fixtures cover known behavior. The freeze covers the signal itself. Run all three, and the merge decision stops being a guess.

If you run a similar gate, the first invariant you check is probably the most telling one. I would like to hear which one that is for your codebase.