The Agent Cannot Own the Oracle

The pull request looked green in the dashboard. New unit tests sat beside the generated patch. Reviewers trusted those checkmarks without reading the oracles.

This scene is reconstructed, not a named incident. An agent had authored both the fix and the proof. Production still drifted during the following on-call morning.

The suite did not fail on that path. The oracle had moved with the patched code. That merge failure is now a common pattern.

An agent patch is not a human bugfix. It often emits tests that describe the new code. Those tests encode fresh behavior as if it were truth.

They cannot detect a silent contract break. They only prove the patch matches itself. That is grading an exam after rewriting the key.

Yesterday's characterization suite still knew the old answers. Today's generated tests forgot those answers on purpose. Green then means agreement, not preservation.

You need a rule that predates the diff. Oracles must exist before the agent types. Anything the agent authors stays evidence, never law.

Split the suite into lanes

Treat the test tree as traffic, not as a pile. Characterization traffic replays pinned outputs from last week. Property traffic checks invariants with frozen random seeds.

Flake traffic stays behind a closed gate. Those tests do not vote on agent merges. A human reopens that gate after a root-cause note.

A fourth lane holds human regression tests only. Agent diffs may extend fixtures, not rewrite that lane. Unmarked tests are not a lane at all.

The decision lives in one small table. Keep it in the repo as executable data. Do not keep it in a wiki paragraph.

# lane_policy.py — executable merge policy, not documentation
LANES = {
    "characterization": {
        "may_agent_add": False,
        "oracle": "pinned_fixture_hash",
        "votes_on_merge": True,
    },
    "property": {
        "may_agent_add": False,
        "oracle": "invariant_plus_frozen_seed",
        "votes_on_merge": True,
    },
    "flake_frozen": {
        "may_agent_add": False,
        "oracle": None,
        "votes_on_merge": False,
    },
    "human": {
        "may_agent_add": False,
        "oracle": "reviewer_written_assert",
        "votes_on_merge": True,
    },
}

The agent may propose fixture bytes under review. It may not invent a new property statement. It may not unskip a frozen flake to buy green.

Mark every collected test

Pytest markers make the lanes visible to CI. Put the markers on modules, not on luck. Collection must fail closed when a mark is missing.

# conftest.py
import pytest

ALLOWED = frozenset({
    "characterization",
    "property",
    "flake_frozen",
    "human",
})

def pytest_collection_modifyitems(config, items):
    unmarked = []
    for item in items:
        present = ALLOWED.intersection(item.keywords)
        if len(present) != 1:
            unmarked.append(item.nodeid)
    if unmarked:
        joined = "\n".join(unmarked[:40])
        raise pytest.UsageError(
            "each test needs exactly one lane mark\n" + joined
        )

Run collection before any patch lands on main. The command is short and boring on purpose. Boring gates survive agent volume.

pytest --collect-only -q

A reconstructed billing helper shows the characterization lane. The fixture predates the agent session. The hash is the oracle, not the new function body.

# tests/test_invoice_characterization.py
import hashlib
import json
from pathlib import Path

import pytest

from billing import render_invoice

pytestmark = pytest.mark.characterization

GOLDEN = Path(__file__).with_name("invoice_v3.json")
PINNED = "e3b0c44298fc1c149afbf4c8996fb924"  # replace with real sha256

def test_render_invoice_matches_pinned_bytes():
    payload = json.loads(GOLDEN.read_text())
    rendered = render_invoice(payload).encode("utf-8")
    digest = hashlib.sha256(rendered).hexdigest()
    assert digest == PINNED

If the agent “fixes” formatting, this test fails first. That failure is the feature, not the nuisance. Humans then choose a hash bump or a revert.

Freeze seeds, not hopes

Property tests die when seeds wander between runs. An agent will reroll Hypothesis and call the flake a fix. Pin the seed in the test, not in chat history.

# tests/test_invoice_properties.py
from decimal import Decimal

from hypothesis import given, settings, seed, strategies as st
import pytest

from billing import allocate_tax

pytestmark = pytest.mark.property

@seed(20260920)
@settings(max_examples=80, deadline=None)
@given(
    amount=st.decimals(min_value="0.01", max_value="100000", places=2),
    rate=st.decimals(min_value="0.00", max_value="0.25", places=4),
)
def test_tax_plus_net_equals_gross(amount: Decimal, rate: Decimal):
    net, tax = allocate_tax(amount, rate)
    assert net + tax == amount
    assert tax >= 0
    assert net >= 0

The invariant is older than the patch. The seed is older than the patch. The agent may change allocate_tax only while both still hold.

Flakes do not join that vote. Mark them frozen and skip them in merge CI. Record the last failure hash in the skip reason.

# tests/test_invoice_flakes.py
import pytest

pytestmark = pytest.mark.flake_frozen

@pytest.mark.skip(reason="frozen: clock-skew on tax day, see issue 1841")
def test_month_end_cutoff_against_live_clock():
    raise AssertionError("must not run during agent merge")

An agent that deletes the skip is proposing policy. Policy is not a coding task. The lane gate must see that deletion as a diff crime.

Gate the diff, not the vibes

Collection policy is not enough under a busy agent. You also inspect the changed files. New tests without a lane mark fail the job.

Changed property signatures fail the job. Unskipped flakes fail the job. New asserts inside tests/ fail unless a human lane file already existed.

# tools/lane_gate.py
#!/usr/bin/env python3
"""Fail closed on agent diffs that steal the oracle."""
from __future__ import annotations

import subprocess
import sys
from pathlib import Path

LANE_MARKS = (
    "pytest.mark.characterization",
    "pytest.mark.property",
    "pytest.mark.flake_frozen",
    "pytest.mark.human",
)

def git_changed(ref: str = "origin/main") -> list[str]:
    out = subprocess.check_output(
        ["git", "diff", "--name-only", f"{ref}...HEAD"],
        text=True,
    )
    return [line for line in out.splitlines() if line]

def is_test(path: str) -> bool:
    return path.startswith("tests/") and path.endswith(".py")

def main() -> int:
    changed = git_changed()
    failures: list[str] = []
    for path in changed:
        if not is_test(path):
            continue
        text = Path(path).read_text(encoding="utf-8")
        if "flake_frozen" in text and "pytest.mark.skip" not in text:
            failures.append(f"unfrozen flake: {path}")
        if path not in _preexisting() and not any(m in text for m in LANE_MARKS):
            failures.append(f"unmarked new test file: {path}")
        if "@given" in text and "@seed(" not in text:
            failures.append(f"property test missing frozen seed: {path}")
    if failures:
        print("lane gate failed:")
        print("\n".join(failures))
        return 1
    return 0

def _preexisting() -> set[str]:
    out = subprocess.check_output(
        ["git", "ls-tree", "-r", "--name-only", "origin/main"],
        text=True,
    )
    return set(out.splitlines())

if __name__ == "__main__":
    sys.exit(main())

Wire it as a required check. Keep it on the default runner you already trust. Do not let the generating box certify its own homework.

# .github/workflows/lane-gate.yml
name: lane-gate
on: pull_request
jobs:
  gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - run: python tools/lane_gate.py
      - run: pytest --collect-only -q
      - run: pytest -m "not flake_frozen" -q

The last command is the merge vote. Frozen flakes are collected, then excluded. Property and characterization lanes still have to pass.

Where a generator still helps

The generator belongs on the writing side of this wall. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option for that writing step.

Those two facts are the only product claims used here. No model names, quotas, or timings are attached. The lane gate does not run inside the generator.

A practical loop stays small and local. Draft the patch on the free server if you lack spare GPUs. Fetch the diff onto a laptop that already holds the golden fixtures.

Run python tools/lane_gate.py before you open the pull request. Run the non-flake pytest target after the gate. If characterization hashes move, stop and write a human note.

The note must name the contract you are changing. “Agent said so” is not a name. Revert when the note cannot point at a ticket.

What this will not catch

Lane marks do not replace production probes. A pinned invoice hash will miss a new optional field. A tax invariant will miss a wrong jurisdiction table.

Property tests need a true invariant. If you cannot state one in one sentence, skip that lane. Weak properties become another vacuous green in disguise.

Do not use this gate on a toy repo with no baseline. There is nothing to characterize and nothing to freeze. A green lane gate on empty tests is theater.

Do not use it to block human spikes on a throwaway branch. The policy is for merge trains that ship. Apply it where a bad oracle can bill a customer.

Teams without fixture discipline will hate the first week. That pain is unpaid debt, not a tool defect. Pay it before you scale agent volume.

Keep the sentences of policy as short as the tests. Oracles predating the diff is the whole method. Everything else is plumbing around that sentence.

If you already draft patches on a spare box, run this gate on the same checkout before the PR exists.