The Agent Writes the Harness. A Human Owns the Oracle File.

A green suite is not evidence that an agent patch is correct if that same patch also wrote the tests. Split the claim from the runner. Keep every claim in a human-owned oracle file. Run the harness on a machine the agent cannot edit.

That split is the merge rule. The oracle file is the spec. The harness is untrusted code that tries to satisfy the spec. If a patch needs to edit both in one diff, the spec moved. Treat that as a product change, not as a test fix.

This article proposes a small, reproducible gate. It does not report production incident rates. Examples below are labeled as such.

Why a single workspace lies

An agent working in one checkout can satisfy tests by changing production code, the tests, the fixtures, or the clock. CI that runs in that same tree only proves the tree is self-consistent. It does not prove the tree still means what the team meant last week.

The usual counter is “review the test diff.” Reviewers miss silent oracle drift. A renamed assertion, a loosened bound, a fixture that now contains the answer: all still look like diligence.

So stop asking the agent to own the meaning of pass. Ask it only to own the procedure that checks a frozen meaning.

Oracle versus harness

Define two directories and never let one patch own both.

1. oracles/ is human-owned. It holds invariant ids, bounds, seeds, and the hash of every input blob the invariant is allowed to read.
2. harness/ is agent-writable. It holds pytest files, builders, and adapters that call production code.
3. oracles/MANIFEST.sha256 is generated only from oracles/ and is checked in.
4. Merge fails if the patch touches oracles/ unless a human label oracle-change is present on the pull request.

The harness may add examples. It may not delete an invariant id, raise a bound, or retarget a hash. Those edits are spec edits.

Artifact: a frozen oracle file

Proposed format. Not a standard. Keep it boring so the gate can parse it without an LLM.

# oracles/billing.toml — human owned, reviewed as product spec
schema = 1

[[invariant]]
id = "total_cents_non_negative"
entry = "billing.totals:grand_total_cents"
kind = "bound"
min = 0
max = 100000000
seed = 42
blob = "oracles/blobs/cart_samples.json"
blob_sha256 = "4f3c0a1b8e2d9c7a6b5e4d3c2b1a090887766554433221100ffeeddccbbaa998"

[[invariant]]
id = "refund_is_idempotent"
entry = "billing.refunds:apply_refund"
kind = "idempotent"
args = ["refund_id", "amount_cents"]
repeat = 2
blob = "oracles/blobs/refund_samples.json"
blob_sha256 = "9a0b1c2d3e4f5566778899aabbccddeeff00112233445566778899aabbccddee"

[[invariant]]
id = "tax_round_half_even"
entry = "billing.tax:apply_rate"
kind = "exact_table"
blob = "oracles/blobs/tax_table.json"
blob_sha256 = "00112233445566778899aabbccddeeff00112233445566778899aabbccddeeff"

Each row is a claim with an id that must survive the patch. kind is a closed enum the harness must implement. blob_sha256 pins the input. If the agent “fixes a fixture,” the hash breaks and the gate fails.

Artifact: a local checker the CI image runs first

Proposed script tools/check_oracle_gate.py. Unexecuted here. Run it on every pull request before pytest.

#!/usr/bin/env python3
"""Fail if a patch edits oracles/ without a label, or if harness drops an id."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

import tomllib

ORACLE_DIR = Path("oracles")
HARNESS_DIR = Path("harness")
ALLOWED_KINDS = {"bound", "idempotent", "exact_table"}


def git(*args: str) -> str:
    return subprocess.check_output(["git", *args], text=True).strip()


def changed_paths(base: str) -> set[str]:
    out = git("diff", "--name-only", f"{base}...HEAD")
    return {line for line in out.splitlines() if line}


def load_invariants() -> dict[str, dict]:
    found: dict[str, dict] = {}
    for path in sorted(ORACLE_DIR.glob("*.toml")):
        data = tomllib.loads(path.read_text())
        for row in data.get("invariant", []):
            iid = row["id"]
            if iid in found:
                raise SystemExit(f"duplicate invariant id: {iid}")
            if row["kind"] not in ALLOWED_KINDS:
                raise SystemExit(f"unknown kind for {iid}: {row['kind']}")
            blob = Path(row["blob"])
            digest = hashlib.sha256(blob.read_bytes()).hexdigest()
            if digest != row["blob_sha256"]:
                raise SystemExit(f"blob hash mismatch for {iid}: {digest}")
            found[iid] = row
    return found


def harness_mentions(iid: str) -> bool:
    needle = iid.encode()
    for path in HARNESS_DIR.rglob("*.py"):
        if needle in path.read_bytes():
            return True
    return False


def main() -> int:
    base = os.environ.get("ORACLE_DIFF_BASE", "origin/main")
    paths = changed_paths(base)
    oracle_touched = [p for p in paths if p.startswith("oracles/")]
    label = os.environ.get("PR_LABELS", "")
    if oracle_touched and "oracle-change" not in label.split(","):
        print("oracle files changed without oracle-change label:")
        print("\n".join(oracle_touched))
        return 2

    current = load_invariants()
    old_txt = git("show", f"{base}:oracles/MANIFEST.sha256")
    old_ids = set(json.loads(old_txt)["ids"])
    missing = sorted(old_ids - set(current))
    if missing:
        print("harness/oracle drop of invariant ids is forbidden:")
        print("\n".join(missing))
        return 3

    uncovered = [iid for iid in current if not harness_mentions(iid)]
    if uncovered:
        print("harness does not mention invariant ids:")
        print("\n".join(uncovered))
        return 4

    manifest = {
        "ids": sorted(current),
        "count": len(current),
    }
    Path("oracles/MANIFEST.sha256").write_text(json.dumps(manifest, indent=2) + "\n")
    print(f"oracle gate ok: {len(current)} invariants")
    return 0


if __name__ == "__main__":
    sys.exit(main())

Command sequence for a reviewer, proposed:

git fetch origin
export ORACLE_DIFF_BASE=origin/main
export PR_LABELS=""   # or "oracle-change" when a human intends a spec edit
python tools/check_oracle_gate.py
pytest harness -q --seed=42

The seed is in the oracle file, not in the agent’s pytest.ini. If the harness.ini changes the seed, the bound and idempotent rows are no longer the same experiment.

Numbered merge workflow

1. Human lands oracles/*.toml and blob hashes on main with a normal review. That commit is the spec.
2. Agent receives the task plus a read-only copy of oracles/. The prompt forbids edits under oracles/.
3. Agent may write harness/ and production code only.
4. CI runs check_oracle_gate.py against origin/main. Any oracles/ path in the diff fails unless oracle-change is set by a human.
5. CI then runs the harness twice: once in the PR workspace, once on a runner that never mounts the agent’s working tree as writable.
6. Merge requires both harness runs to report the same invariant ids and the same per-id outcomes.

Step 5 is the part a laptop CI misses. A workspace-local pass can depend on leftover files, a developer timezone, or an agent-written .env. The second run has to start from a clean image and the oracle blobs only.

Where a free model and a free server actually sit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The model does not write oracles/. That would collapse the split. Use a free model only as a proposer: given a production diff, it lists candidate invariant ids a human might add later. A reviewer copies a candidate into oracles/ in a separate pull request, with the oracle-change label, after rewriting the bounds by hand.

Proposed proposer prompt, to be pasted as text, not as an autonomous merge:

Read the production diff only. Do not read harness/.
List candidate invariants as TOML rows.
Each row needs: id, entry, kind in {bound, idempotent, exact_table},
and a one-line reason that cites a changed function.
Do not emit pytest. Do not suggest edits under oracles/ yourself.

The free server option is the second runner in step 5. Check out main’s oracles/ plus the PR’s production code and harness/, on a machine that the agent session cannot SSH into. If you already operate a spare CI label, use that. MonkeyCode’s free model access and free server option fit this workflow when you need a proposer and a clean runner without standing up another cluster. They do not replace the human label on spec edits.

Do not send oracle blobs that contain production secrets to any remote model. Hash them locally. Send only the TOML rows and the production diff.

Decision table

Signal in the PR	Gate result	Human action
Production code + harness, oracles/ untouched, ids preserved	Continue	Review production risk only
Harness adds a new test file that mentions every existing id	Continue	No spec change
Harness stops mentioning an id	Fail (exit 3 or 4)	Restore the id or open a labeled oracle PR
Blob bytes change, TOML hash not updated	Fail (hash mismatch)	Restore blob or file a spec PR
oracles/ edited, no oracle-change label	Fail (exit 2)	Reject, or apply label after review
oracles/ edited with label, bounds widened	Review as product	Require a changelog line
Model-proposed TOML pasted without hash	Fail	Compute sha256sum on the blob first
Local pytest green, clean runner red	Fail	Treat as environment coupling, not as flake

The last row is not a flaky-test policy. It is an environment-coupling policy. A clean runner that disagrees with the workspace is a spec violation until proven otherwise.

Minimal harness adapter

Proposed pytest module. The adapter is the only file allowed to import tomllib from oracles/.

# harness/test_oracles.py
from pathlib import Path
import hashlib
import importlib
import json
import tomllib

import pytest


def _rows():
    for path in sorted(Path("oracles").glob("*.toml")):
        data = tomllib.loads(path.read_text())
        yield from data.get("invariant", [])


def _load_entry(entry: str):
    mod_name, fn_name = entry.split(":")
    return getattr(importlib.import_module(mod_name), fn_name)


@pytest.mark.parametrize("row", list(_rows()), ids=lambda r: r["id"])
def test_invariant(row):
    raw = Path(row["blob"]).read_bytes()
    assert hashlib.sha256(raw).hexdigest() == row["blob_sha256"]
    fn = _load_entry(row["entry"])
    samples = json.loads(raw)

    if row["kind"] == "bound":
        for sample in samples:
            value = fn(**sample)
            assert row["min"] <= value <= row["max"], row["id"]
        return
    if row["kind"] == "idempotent":
        for sample in samples:
            first = fn(**sample)
            second = fn(**sample)
            assert first == second, row["id"]
        return
    if row["kind"] == "exact_table":
        for sample in samples:
            assert fn(*sample["args"]) == sample["expected"], row["id"]
        return
    raise AssertionError(f"unhandled kind {row['kind']}")

Notice the adapter does not compute new bounds. It only reads them. An agent that wants a weaker test has to change oracles/, which the gate rejects.

Limitations

This gate does not prove the oracle is the right spec. A stale bound that is still hashed will stay green forever. Schedule human oracle review when the product changes, not when the agent feels blocked.

It also does not replace type checks, lint, or security review. It answers one question: did this patch keep the named claims, on a runner the agent does not control.

Closed kind enums will be too small for some domains. Add a kind only in an oracle-change PR. Do not let the harness interpret free-form predicates from model output.

Who should not use this approach: teams with no human willing to own oracles/; repos whose tests must mutate golden files on every UI pixel change; and pipelines that cannot run a second checkout. If the only runner is the agent’s own workspace, the split is theater.

If you need a proposer for candidate rows and a second machine that never mounts the agent workspace writable, MonkeyCode’s free model access and free server option are one way to fill those two slots. Keep the label, the hashes, and the merge decision in your own CI.