Agent Patches Pass Their Own Tests. Here's the Three-Layer Gate That Doesn't Trust Them.

Agent-generated patches pass their own tests for a boring reason: the same model wrote both sides. The tests inherit the code's assumptions, so a green suite only proves the patch matches the model's model of the problem. It does not prove the patch matches the problem.

The fix is not more tests. It is three layers of tests that break the code-test correlation:

1. Property checks — invariants that must hold for any input, not just the examples the agent saw.
2. Curated fixtures — edge cases collected from production incidents and written by a human.
3. A flaky freeze — a registry that removes unreliable tests from the signal until they are fixed or expire.

Each layer answers one question. Property checks ask whether the patch violates a spec. Fixtures ask whether it preserves behavior you already paid for. The freeze asks whether the test signal is deterministic enough to trust.

The gate is cheap to run. That is the point. Here is the exact setup.

Why the agent's tests are not evidence

When a model writes a patch and its tests in a single pass, both come from the same distribution. The tests use the same examples, the same assumptions, the same blind spots. A passing suite measures self-consistency, not correctness.

A minimal CSV parser shows the pattern. The agent's patch handled every example in its own test file. The tests were green. The parser still dropped empty fields, because the agent's examples never contained one.

The agent's tests passed. The parser was wrong.

That is the baseline problem: agent tests are correlated with agent code. Any gate that trusts them inherits the correlation. The three layers below are designed to break it.

Layer 1: property checks

Property checks replace examples with invariants. Instead of asserting that parse_csv_line("a,b") returns ["a", "b"], you state a rule that must hold for every input in a large generated space.

For the parser, the most useful property is round-trip stability:

# csvline.py — the parser the agent rewrote
def parse_csv_line(line: str) -> list[str]:
    return line.rstrip("\r\n").split(",")

def serialize_csv_line(fields: list[str]) -> str:
    return ",".join(fields)

# tests/test_properties.py
from hypothesis import given, settings, strategies as st

# The format contract: fields contain no commas, newlines, or control chars.
field = st.text(
    alphabet=st.characters(blacklist_categories=("Cc", "Cs"), blacklist_characters=","),
    max_size=16,
)

@given(st.lists(field, min_size=1, max_size=8))
@settings(deadline=500)
def test_round_trip(fields):
    assert parse_csv_line(serialize_csv_line(fields)) == fields

@given(st.text(max_size=512))
def test_never_raises_on_arbitrary_text(text):
    parse_csv_line(text)  # must not raise

The round-trip test catches the empty-field bug. Hypothesis generates ["a", "", "b"], serialization produces "a,,b", and a parser that skips empty fields returns ["a", "b"] instead. The assertion fails.

The second property is a crash guard. A parser that raises on arbitrary text will fail in production on data you never saw. This property is deliberately boring. Boring properties catch real faults.

Two rules keep property checks reliable in a gate:

• Fix the seed. Run Hypothesis with --hypothesis-seed=20260828 so a failure is reproducible. A gate that fails differently on every run is a flaky gate.
• Keep the generator inside the contract. The field strategy excludes commas and control characters. If the generator produces inputs the format cannot represent, the property test fails for the wrong reason.

One case is deliberately excluded: the empty line. serialize_csv_line([]) returns "", and parse_csv_line("") returns [""]. The ambiguity is real, so the property starts at min_size=1 and the empty line is pinned by a fixture instead.

Property checks need an oracle: you must be able to state the invariant. When you cannot, layer 2 takes over.

Layer 2: curated fixtures

Some behavior has no clean invariant, but you still know the answer. You know it because production taught you, usually by breaking. Those lessons belong in a fixture manifest, written by a human and reviewed on a schedule.

{
  "fixtures": [
    {"name": "empty_field", "input": "a,,b", "expected": ["a", "", "b"]},
    {"name": "trailing_comma", "input": "a,b,", "expected": ["a", "b", ""]},
    {"name": "spaces_kept", "input": " a ,b", "expected": [" a ", "b"]},
    {"name": "empty_line", "input": "", "expected": [""]}
  ]
}

The runner is deliberately small:

# tests/test_fixtures.py
import json
from csvline import parse_csv_line

with open("fixtures.json") as f:
    FIXTURES = json.load(f)["fixtures"]

def test_fixtures():
    for fx in FIXTURES:
        assert parse_csv_line(fx["input"]) == fx["expected"], fx["name"]

The rule is simple: every production incident that touched this module becomes a fixture. The agent never sees the incident report. It only sees the code, so the fixture tests something the agent did not optimize for.

Fixtures are the memory of the system. Property checks catch what you can state. Fixtures catch what you can only remember. Agent patches are optimized against the code in front of them, not against your history. Fixtures close that gap.

Layer 3: the flaky freeze

A flaky test destroys an agent-patch gate faster than a missing test. If a test fails 5% of the time, you cannot tell whether the agent's patch caused the failure. You retry, it passes, and you learn nothing. The gate becomes a coin flip.

The freeze is a registry. When a test flakes, it is skipped with a recorded reason, a flake count, and an expiry date. The gate treats a frozen test as absent until the date arrives.

{
  "frozen": [
    {
      "test": "test_fixture_crlf",
      "reason": "CI runner converted CRLF to LF in the fixture file; needs a binary fixture or a normalization step",
      "flakes": 4,
      "runs": 60,
      "expires": "2026-09-14"
    }
  ]
}

The expiry date is not optional. A permanent freeze is a permanent hole in the gate. When the date passes, the gate fails until you fix the test or delete it. That forces a decision instead of letting the hole age silently.

A flaky property check is usually a bad property, not a bad environment. When a Hypothesis test flakes, the invariant is probably stated too loosely. Fix the property. Do not freeze it twice.

The gate, end to end

The three layers run in order, as one script:

#!/usr/bin/env bash
set -euo pipefail

echo "== layer 1: property checks =="
pytest tests/test_properties.py -q --hypothesis-seed=20260828

echo "== layer 2: fixtures =="
pytest tests/test_fixtures.py -q

echo "== layer 3: frozen flaky tests =="
python gate/freeze.py --registry gate/frozen.json --expired-action=fail

echo "== gate passed =="

The freeze runner is a small script:

# gate/freeze.py
import json
import sys
from datetime import date

args = sys.argv[1:]
with open(args[args.index("--registry") + 1]) as f:
    registry = json.load(f)

expired = [
    t for t in registry["frozen"]
    if date.fromisoformat(t["expires"]) < date.today()
]

if expired and "--expired-action=fail" in args:
    for t in expired:
        print(f"expired freeze: {t['test']} expires {t['expires']}")
    sys.exit(1)

The order matters. Property checks run first because they are the cheapest source of real information. Fixtures run second because they are deterministic by construction. The freeze check runs last because it only makes sense after the other layers have produced signal.

This is the workflow I use for agent patches. Patches are generated through MonkeyCode's free model access, and the gate runs on MonkeyCode's free server option, so the whole loop costs nothing. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The cost point matters more than the vendor. When model access and compute are free, the gate becomes the default for every patch, including the ones that look trivial. The trivial-looking ones are exactly the ones that ship the empty-field bug.

Limitations

The gate has real limits. Property checks require an oracle; if you cannot state the invariant, the layer is silent. Fixtures decay; a fixture that encoded a 2024 decision will fail a 2026 change that was deliberate. The freeze can hide genuine breakage if the expiry date is ignored.

There is also a class of faults this gate does not see. It measures behavior, not performance. A patch can pass all three layers and still be ten times slower. It cannot detect a patch that is correct but architecturally wrong. No test layer replaces a code review that reads for structure.

Free model access and free server options are availability claims, not contracts. Verify the current terms before you build a workflow around them. The gate itself does not depend on them; it runs on any CI.

Who should not use this

Three teams should skip this approach. Teams with no CI at all: a gate that does not run automatically is a document, not a gate. Teams whose bottleneck is test runtime: add profiling before adding more test layers. Teams with a formal spec: use a model checker or a proof tool instead of probabilistic testing.

For everyone else, start with one module. Write one property. Add one incident fixture. Freeze one flaky test. The gate grows from there.

If you run a similar gate, I would like to see your fixture manifest. The failures are usually more interesting than the passes.