The Agent Wrote Its Own Tests. Your Gate Shouldn't Trust Them.

An agent patch is a hypothesis. The tests it writes with that patch are evidence collected by the same team that made the hypothesis. Independent confirmation has to come from somewhere.

Another review pass over the agent's own claims is not the cheapest way to get that confirmation. The pattern that works is simpler: read the diff shape, then assign each changed path to one of three oracles — property checks, fixture contracts, or a flaky-test freeze. Each oracle catches a different failure class, and each runs on a small server.

Self-written tests share blind spots

An agent that misreads a spec tends to encode the same misreading in the implementation and in the test. A wrong assumption about sorting appears in both places. A mocked API client that reimplements the bug looks green in both places. A timing change can pass on a lucky CI run and then flake for weeks.

Humans do the same thing. The difference is that an agent can produce a large volume of confident, wrong tests without blinking. So the gate should treat agent-written tests as claims, not as proof.

Three verification modes

Change pattern	Weak signal	Stronger oracle
Pure transformation: slug, format, render, validate	Snapshot equality	Property checks and metamorphic relations
I/O boundary: API client, repository, adapter	Mock that reimplements the system	On-disk fixture contracts with a checksum lock
Timing: retry, lock, lease, timeout	One green run	Time-boxed flaky freeze with an expiry date

The decision is deliberately boring. A pure function gets properties. A boundary gets a contract. A timing change gets a freeze. The boredom removes the reviewer's mood from the equation.

Property checks for pure transformations

For a pure transformation, ask what has to be true about the result regardless of the input. Idempotence is usually the easiest property to write and the hardest to fake.

from hypothesis import given, strategies as st
from core import slugify

@given(st.text(max_size=200))
def test_slugify_is_a_fixed_point(value):
    once = slugify(value)
    twice = slugify(once)

    assert once == twice
    assert once == slugify(twice)

This does not prove the slug is human-readable. It proves the function reaches a stable answer and stops mutating it. Pair it with a safety property:

@given(st.text(max_size=200))
def test_slug_has_no_url_unsafe_characters(value):
    assert not any(c in slugify(value) for c in ' /?#%')

Property checks are not magic. A property that is too weak can be true for a broken implementation. Write at least one fixed-point, one inversion, and one safety property per pure module.

Fixture contracts lock the boundary

I/O code is where agents are most likely to copy the wrong field names. A contract test replays a captured response and checks that the parser preserves what the outside world actually sent.

import hashlib
import json
from fixtures import FIXTURES

FIXTURE = FIXTURES / 'orders_page_1.json'
LOCKFILE = {
    'orders_page_1.json': 'sha256:9f2c9c8a1b0f3d4e5a6b7c8d9e0f1a2b3c4d5e6f7a8b9c0d1e2f3a4b5c6d7e8f'
}

def test_fixture_has_not_been_silently_edited():
    digest = hashlib.sha256(FIXTURE.read_bytes()).hexdigest()
    assert f'sha256:{digest}' == LOCKFILE['orders_page_1.json']

def test_parse_orders_contract():
    raw = json.loads(FIXTURE.read_text())
    page = parse_orders(raw)

    assert page.next_token == raw['meta']['next_token']
    assert set(page.item_ids) == {item['id'] for item in raw['items']}

Compute the checksum when you commit the fixture. Never let the agent regenerate the fixture to match the parser. The checksum is what turns a stored fixture from documentation into a contract.

Flaky freeze is a time-boxed debt

When the patch touches retry or lock logic, the worst thing to do is run the timing test once and call it green. A better cheap thing is to quarantine it with an expiry date, an owner, and a reason.

quarantine = {
    'tests': [
        {
            'test': 'tests/test_retry.py::test_connection_reset',
            'frozen_until': '2026-09-16',
            'owner': 'platform',
            'reason': 'agent patch widened the backoff window',
        }
    ]
}

A quarantine is not a permanent skip. The gate should fail any test that is still frozen after its expiry. Every frozen test either gets re-enabled or gets promoted to a real fix within the window.

A small script assigns the oracle

Instead of asking a reviewer to re-decide every time, classify by path markers:

import argparse
import json
import subprocess

PURE = ('slug', 'parse', 'format', 'render', 'transform', 'validate')
BOUNDARY = ('repository', 'client', 'adapter', 'gateway', 'api', 'store')
TIMING = ('retry', 'backoff', 'lock', 'lease', 'clock', 'timeout', 'sleep')

def classify(path):
    lowered = path.lower()
    if any(m in lowered for m in TIMING):
        return 'timing'
    if any(m in lowered for m in BOUNDARY):
        return 'boundary'
    return 'pure'

def main(base):
    out = subprocess.check_output(['git', 'diff', '--name-only', base], text=True)
    paths = [line for line in out.splitlines() if line]
    plan = [{'path': p, 'oracle': classify(p)} for p in paths]
    print(json.dumps({'oracles': plan}, indent=2))

if __name__ == '__main__':
    main('origin/main...HEAD')

Run it from CI:

python oracle_plan.py

The output tells the pipeline which pytest marker to run: property, contract, or quarantine. You can also use it as a human checklist. The review question stops being 'do these tests look good?' and becomes 'does this changed path get the right oracle?'

Where to run it

This gate needs Python, pytest, and a git checkout. It does not need a database or a container registry, which makes it a good fit for constrained infrastructure. If you want a low-friction place to start, MonkeyCode's free model access can draft the first property templates and its free server option can run the gate on a low-traffic repository. Disclosure: This article was prepared as part of MonkeyCode's product outreach. A free server simply executes the same commands; it does not make a weak oracle stronger.

Limits and who should skip this

Path-based classification is a heuristic. A client file can contain a pure parser, and a pure file can hide a timing side effect. Review the plan once, then trust it until a failure proves otherwise.

Skip this workflow if your team cannot maintain fixture checksums or cannot enforce quarantine expiry. The tools only add rigor if someone is willing to read the plan. For cosmetic changes, a normal test run is enough; the triage overhead is not free.

The real test of an agent-patch gate is not how many tests it runs. It is whether the tests can fail for a reason the agent never considered. Property checks, fixture contracts, and expiring freezes are three cheap ways to manufacture that reason.