Opinion: Repo Tests Are a Compromised Oracle for AI Patches — Probe the Contract the Model Never Saw

A green CI badge measures one thing only: the tests you already wrote still pass. It does not measure whether an AI-generated patch preserved the behaviors you never wrote down. When the model read those tests as part of its prompt context, the badge is scoring a target the patch was already aiming at. My position is direct: for free-model patches, the repo test suite is a compromised oracle, and the only gate that survives is a contract probe the model never saw.

Generation stopped being the bottleneck the day free model access arrived. A patch now costs tokens instead of budget, and the free server option gives you a disposable runtime where that patch can execute before any human reads a line. That shift moves the entire risk to verification, which is exactly where most teams are least prepared. They reach for the test suite that already lives in the repository, and that is the mistake.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The suite is compromised in a mechanical sense, not a moral one. When you hand a model your repository, the tests become part of the context it optimizes against, so a patch that satisfies them is pattern matching, not evidence. The suite also encodes yesterday's assumptions about the system, and a model that reads those assumptions can reproduce them faithfully. Meanwhile the invariants no test ever captured — idempotency, isolation, timeout behavior — break silently behind a green badge.

I argued recently that the test suite should pick the patch instead of the reviewer, and I still believe that with one correction. The suite must not be the one the model read, because an oracle that the judged party can see is no oracle at all. The fix is a small standalone contract probe that lives outside the repository, never enters the prompt, and asserts the behaviors your team once debugged at 3 a.m.

The probe

The artifact is a single Python script that exercises a running instance and emits a JSON report. It uses only the standard library, so it runs anywhere, and it treats every check as a comparison between baseline and patched instances. A probe that fails on the baseline is a pre-existing issue, not a patch regression, and the diff makes that distinction visible.

#!/usr/bin/env python3
"""contract_probe.py — behavioral invariants checked outside the repo.

Usage:
    python3 contract_probe.py http://staging:8080 > baseline.json
    python3 contract_probe.py http://patched:8080 > patched.json
    diff baseline.json patched.json   # the only diff that should matter
"""
import json, sys, time, urllib.request, urllib.error

BASE = sys.argv[1].rstrip("/") if len(sys.argv) > 1 else "http://127.0.0.1:8080"

def call(method, path, payload=None, headers=None, timeout=5.0):
    req = urllib.request.Request(
        BASE + path,
        data=json.dumps(payload).encode() if payload is not None else None,
        headers={"Content-Type": "application/json", **(headers or {})},
        method=method,
    )
    start = time.monotonic()
    try:
        with urllib.request.urlopen(req, timeout=timeout) as resp:
            body = resp.read().decode()
            return resp.status, json.loads(body) if body else None, time.monotonic() - start
    except urllib.error.HTTPError as e:
        return e.code, None, time.monotonic() - start
    except urllib.error.URLError:
        return 0, None, time.monotonic() - start

def probe_idempotency():
    key = "probe-20260821-001"
    first = call("POST", "/api/orders", {"sku": "A-1", "qty": 2}, {"Idempotency-Key": key})
    second = call("POST", "/api/orders", {"sku": "A-1", "qty": 2}, {"Idempotency-Key": key})
    return {"duplicate_create_rejected": first[0] == second[0] and first[1] == second[1],
            "first_status": first[0], "second_status": second[0]}

def probe_authz_boundary():
    a = call("GET", "/api/tenants/a/records", headers={"Authorization": "Bearer tenant-b-token"})
    b = call("GET", "/api/tenants/b/records", headers={"Authorization": "Bearer tenant-a-token"})
    return {"cross_tenant_read_blocked": a[0] == 403 and b[0] == 403,
            "a_status": a[0], "b_status": b[0]}

def probe_timeout_budget():
    status, _, elapsed = call("GET", "/api/reports/slow", timeout=3.0)
    return {"slow_endpoint_within_budget": status != 0 and elapsed < 3.0,
            "status": status, "elapsed_s": round(elapsed, 3)}

if __name__ == "__main__":
    report = {"base": BASE, "idempotency": probe_idempotency(),
              "authz": probe_authz_boundary(), "timeout": probe_timeout_budget()}
    print(json.dumps(report, indent=2))

The endpoints and invariants above are placeholders; the structure is the point. Your probes should assert the behaviors that break when a model rewrites a handler with clean-looking code. Duplicate submissions, cross-tenant reads, unbounded dependency timeouts, unstable list ordering, expired records still being served — those are the contract.

What the probe catches that the suite misses

Invariant	Typical repo suite	Contract probe
Idempotent create	mocked or absent	real duplicate request
Cross-tenant isolation	mocked auth context	real foreign token
Timeout budget	fake clock	real wall clock
List ordering	fixture order	concurrent writes
Retention window	not covered	old record query

The pattern is consistent: repo tests verify that code does what the fixtures expect, while the probe verifies that the running system still honors the promises users rely on. Those promises are rarely written as tests, which is precisely why a model can break them without failing anything.

The workflow

1. Write the probe and commit it to a separate repository, or at minimum a path that never enters the model prompt. Secrecy from the model is the entire mechanism, and a probe the model can read is just another test it can satisfy.
2. Run the probe against your current staging instance and save baseline.json. This snapshot defines what "unchanged behavior" means before any patch exists.
3. Generate the patch with the free model and deploy it to a disposable instance on MonkeyCode's free server. No diff reading yet, no review; just execution against a real runtime.
4. Run the same probe against the patched instance and save patched.json.
5. Diff the two reports and accept the patch only when no probe regressed. A regression that maps to a deliberate, documented behavior change needs explicit human approval before it counts as intentional.
6. Read the code diff only after the probe diff is clean, and read it for maintainability rather than correctness. Correctness was already decided by execution.

This workflow costs a few minutes per patch and removes the most dangerous assumption in AI-assisted development: that a model which read the tests can be judged by them.

Who should not use this

The probe is not a universal gate, and pretending otherwise would be dishonest. Teams with no running service, or a prototype that changes shape daily, will spend more time maintaining the probe than the patch saves. Teams whose integration tests already cover these invariants will see redundant reports, although in practice most suites do not cover idempotency and authz boundaries at all. The probe also cannot judge anything that needs a human eye: UI behavior, data quality, naming, or whether the feature should exist in the first place.

The approach has a hard limit that deserves emphasis: it verifies behavior, not intent. A patch can pass every probe and still delete a feature nobody remembered to assert, which is why the probe diff gates the code review instead of replacing it.

The position

Free model access made patch generation nearly free, and the free server made execution cheap, so the scarce resource is no longer tokens or infrastructure. The scarce resource is an independent oracle — something that can judge a patch without having been part of the context that produced it. Your repo tests do not qualify, because the model read them; your contract probe does, because it never will. Stop treating a green suite as evidence for AI patches, and start treating an unseen probe diff as the only evidence that matters.