Rank Oracles Before You Merge an Agent Patch

A green unit run is not a merge decision. It is one oracle, usually the weakest one the agent can see and rewrite. Rank your oracles instead. Characterization fixtures first. Human-locked properties second. Independent fault probes third. Keep flake freezes on a ledger the agent cannot extend.

This article is a test strategy, not a CI product tour. The artifact is an oracle ladder you can run as a read-only job against an agent branch. If every product name below disappeared, the ladder would still be the method.

Why a single test layer lies

Agent patches fail in a narrow pattern. They keep the happy path green. They weaken an assertion, snapshot a looser fixture, or skip a flaky file. A single pytest invocation cannot tell those edits apart from a real fix.

You need ranked evidence. Lower ranks are cheap and easy to game. Higher ranks are slower and must live outside the agent's write set. Merge only when the branch clears a minimum rank, not when the default job is green.

The oracle ladder

Use this table as the contract. Do not collapse it into one boolean.

Rank	Oracle	Pass means	Fail means	Agent may edit?
R1	Characterization replay	Recorded I/O still matches locked hashes	Behavior drifted or fixture was rewritten	No
R2	Property checks	Invariants hold on generated inputs	A stated rule broke	No
R3	Must-fail probes	Known faults still fail	The suite went numb	No
R4	Flake freeze ledger	Only listed tests are skipped, and only until TTL	New skips, expired skips, or silenced retries	No

R1 without R2 is a frozen snapshot of today's bugs. R2 without R3 is a suite the agent can satisfy with tautologies. R3 without R4 looks strict until a flake is skipped forever. Rank is the point. A pass at R1 does not promote the patch.

Layout the agent cannot own

Keep production code in the branch. Keep oracles in a corpus the scoring job mounts read-only.

oracle-ladder/
  corpus/
    traces/*.jsonl          # recorded request/response pairs
    fixtures.lock           # sha256 of each trace file
    properties.py           # human-locked invariants
    probes/
      seeds.json            # faults that must still fail
    freeze.json             # flaky tests with TTL, never grown by the agent
  runner/
    hash_lock.py
    replay.py
    properties_run.py
    probes_run.py
    freeze_check.py
    report.py
  out/
    ladder-report.json      # written by the runner, not by the agent

The agent branch may change src/. It must not change corpus/ or runner/. If your review UI cannot enforce that path split, the ladder is theater.

Step 1 — Lock characterization traces

Record traces from a known-good build, not from the agent's PR description. One JSON object per line. Hash the files. Scoring compares hashes before it replays bytes.

# runner/hash_lock.py — proposal: run against a frozen corpus only
from __future__ import annotations
import hashlib, json, sys
from pathlib import Path

CORPUS = Path("oracle-ladder/corpus")

def sha256(p: Path) -> str:
    h = hashlib.sha256()
    h.update(p.read_bytes())
    return h.hexdigest()

def main() -> int:
    lock_path = CORPUS / "fixtures.lock"
    expected = json.loads(lock_path.read_text())
    traces = sorted((CORPUS / "traces").glob("*.jsonl"))
    actual = {p.name: sha256(p) for p in traces}
    if actual != expected:
        missing = sorted(set(expected) - set(actual))
        extra = sorted(set(actual) - set(expected))
        changed = sorted(
            k for k in expected if k in actual and expected[k] != actual[k]
        )
        print(json.dumps({"missing": missing, "extra": extra, "changed": changed}))
        return 2
    print("R1-lock: hashes match")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Replay is a separate process. It sends each recorded input into the patched binary and compares canonical JSON, not pretty-printed text.

# runner/replay.py — proposal
import json, sys
from pathlib import Path

def canonical(obj) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"))

def replay_line(app, rec: dict) -> bool:
    got = app.handle(rec["input"])
    return canonical(got) == canonical(rec["output"])

If replay fails, stop. Do not run properties on a binary that already drifted from recorded I/O. Drift is evidence. More tests will not make it a pass.

Step 2 — Run properties the agent did not author

Properties are rules about the domain: idempotence, bounds, error codes, ordering. They are not "the function returns something." If a check cannot fail for a wrong patch, it is not an R2 oracle.

# corpus/properties.py — human-locked examples, not generated into src/
def prop_refund_never_exceeds_capture(capture_cents: int, refund_cents: int, result: dict) -> bool:
    if result.get("status") == "rejected":
        return True
    return 0 <= result["refunded_cents"] <= capture_cents

def prop_retry_is_idempotent(first: dict, second: dict) -> bool:
    return first["id"] == second["id"] and first["amount"] == second["amount"]

Drafting candidates is the only place a coding model belongs. Feed it failing traces, not the agent's patch. Copy nothing into properties.py until a human deletes tautologies and adds counterexamples. Same-agent authorship of code and properties is how R2 collapses into R1.

A free local or remote coding environment can host that drafting loop without putting the corpus on the agent's writable checkout. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are relevant here only as a place to propose properties and to run the ladder on a machine the agent does not control. Do not treat that as a quota, a model list, or a benchmark. The scoring rule is the path split, not the vendor.

Step 3 — Keep a must-fail probe set

Probes are faults you already know. A patch that makes them pass has gone numb. Store seeds next to the corpus, not in the agent's test tree.

{
  "probes": [
    {
      "id": "neg-capture",
      "input": {"capture_cents": -1},
      "must_status": "rejected"
    },
    {
      "id": "refund-gt-capture",
      "input": {"capture_cents": 100, "refund_cents": 250},
      "must_status": "rejected"
    }
  ]
}

# runner/probes_run.py — proposal: every seed must still fail closed
import json, sys
from pathlib import Path

def main(app) -> int:
    seeds = json.loads(Path("oracle-ladder/corpus/probes/seeds.json").read_text())
    limp = []
    for probe in seeds["probes"]:
        got = app.handle(probe["input"])
        if got.get("status") != probe["must_status"]:
            limp.append(probe["id"])
    if limp:
        print(json.dumps({"rank": "R3", "limp_probes": limp}))
        return 3
    print("R3: probes still fail closed")
    return 0

R3 is not fuzzing. It is a regression net for assertion deletion. If you need generation, generate more seeds offline and lock them the same way you lock traces.

Step 4 — Freeze flakes off the branch

Flakes happen. A freeze list is allowed. An agent-grown freeze list is a silent skip of R1–R3.

Rules that stay mechanical:

1. freeze.json lives in corpus/, hashed like traces.
2. Each entry needs test_id, reason, expires_on (UTC date).
3. The runner fails if the branch skip-count exceeds the ledger, if an expiry is past, or if a skipped test is not listed.
4. The agent may not add rows. A human shortens TTL or deletes rows. Growth is a review event, not a patch side effect.

# runner/freeze_check.py — proposal
from datetime import date, datetime, timezone
import json, sys
from pathlib import Path

def main(skipped: list[str]) -> int:
    ledger = json.loads(Path("oracle-ladder/corpus/freeze.json").read_text())
    today = datetime.now(timezone.utc).date()
    allowed = {}
    for row in ledger["freezes"]:
        exp = date.fromisoformat(row["expires_on"])
        if exp < today:
            print(json.dumps({"rank": "R4", "expired": row["test_id"]}))
            return 4
        allowed[row["test_id"]] = row
    extra = sorted(set(skipped) - set(allowed))
    if extra:
        print(json.dumps({"rank": "R4", "unlisted_skips": extra}))
        return 4
    print("R4: freeze ledger honored")
    return 0

Do not retry a frozen test until it goes green. Retries hide the skip and poison R3.

Step 5 — Emit one ranked report

CI should consume a file, not a log vibe. Suggested schema:

{
  "commit": "PLACEHOLDER_SHA",
  "ranks": {
    "R1_lock": "pass",
    "R1_replay": "pass",
    "R2_properties": "fail",
    "R3_probes": "not_run",
    "R4_freeze": "pass"
  },
  "min_rank_required": "R3",
  "merge": false,
  "stopped_at": "R2_properties"
}

Stop on first failing rank. not_run is not pass. A job that skips R3 because R2 failed is honest. A job that paints the rest green is not.

Shell sketch for a scoring host — local machine or a free server you mount the corpus onto:

set -euo pipefail
ROOT=oracle-ladder
python "$ROOT/runner/hash_lock.py"
python "$ROOT/runner/replay.py" --bin ./build/app --traces "$ROOT/corpus/traces"
python "$ROOT/runner/properties_run.py" --bin ./build/app --props "$ROOT/corpus/properties.py"
python "$ROOT/runner/probes_run.py" --bin ./build/app --seeds "$ROOT/corpus/probes/seeds.json"
python "$ROOT/runner/freeze_check.py" --junit out/agent-junit.xml --ledger "$ROOT/corpus/freeze.json"
python "$ROOT/runner/report.py" --out out/ladder-report.json

Mount corpus/ and runner/ read-only. Write out/ elsewhere. If the agent can chmod the mount, you do not have a ladder.

What this does not prove

Characterization fixtures rot when the product changes on purpose. Update them in a human PR that only touches corpus/. Properties proposed by a model are drafts. Rubber-stamping them recreates tautologies under a fancier name. Probes do not explore new fault classes. A freeze ledger with long TTLs is still a skip list. None of this replaces code review of auth, crypto, or data-loss paths.

The method also assumes you can split write sets. Monorepos that let an agent rewrite tests/ in the same commit will flatten R1–R4 back into one green job.

Who should not use this ladder

Do not use it as the sole gate for safety-critical releases. Do not use it when the only tests are non-deterministic UI sessions with no recorded I/O. Do not use it if no human owns properties.py. Do not point an agent at the runner "to help maintain tests." That is how oracles get rewritten.

Teams with a small, stable fixture corpus and a reviewer who will reject property spam will get a clearer merge signal. Teams chasing a single badge will not.

If you already keep traces, lock their hashes and run the ladder on a host the agent cannot write. The rank in ladder-report.json is the review artifact. The green unit job is not.