Two Authors, One Patch: Don't Let the Agent That Wrote the Code Write the Oracle

A patch and its test are two claims, and an agent that writes both has effectively written neither one for you. The test will agree with the code because the same context produced them, not because the behavior was ever specified. If your gate only asks "did the new test pass after the patch", self-confirmation sails through untouched.

The fix is structural, not rhetorical: separate the author of the patch from the author of the oracle, and then prove the separation held by measuring whether each new oracle fails on the unpatched tree. That measurement — base-pass rate — is the number this post is about.

Earlier posts here covered diff-scoped gating and seeded flaky freezes. This one targets a different hole: oracles that were never capable of failing.

1. The failure mode, stated precisely

An oracle is only evidence if it is falsifiable by the pre-patch tree. Three ways that property silently disappears:

1. Same-context confirmation. The patch author knows which line it changed and asserts exactly that line's new output. Fine as a smoke check, useless as a specification.
2. Fixture absorption. The patch also edits the fixture, so the assertion compares the new code to the new expectation.
3. Spec-free tautology. The oracle asserts result is not None because the issue text was never given to the test author — only the diff was.

All three survive a green run. None survive a base-tree replay.

2. The split protocol

Three numbered steps, each mechanically enforceable.

Step 1 — Build a sealed spec. Take the issue text plus the public surface of non-test modules at the base commit: signatures, docstrings, exported names. Strip function bodies. The oracle author must never see the diff, and never see the new implementation.

Step 2 — Author oracles in a fresh session. No shared history with the patch session. Ask for pytest functions only, one behavior per test, no network, no sleeps, no imports of private helpers.

Step 3 — Classify, don't just execute. Run the generated file against a clean worktree of the base branch and against the patched tree. A new oracle that passes on both is base_pass — it was satisfied by unfixed code and cannot be evidence for this patch.

3. A runnable harness

The script below does steps 1–3 and appends verdicts to a ledger. It is plain Python plus git and pytest; nothing else.

#!/usr/bin/env python3
"""oracle_split.py — author oracles under a sealed spec, then classify them."""
import ast, hashlib, json, os, re, subprocess, sys, tempfile, urllib.request
from pathlib import Path

TEST_RE = re.compile(r"(^|/)(tests?|spec)/|(^|/)test_.*\.py$|_test\.py$")
GENERATED = "test_oracle_split_generated.py"

SYSTEM = """You write pytest oracles from a sealed specification.
Rules: one behavior per test; no network; no sleeps; no imports of private
helpers (names starting with _); assert observable behavior only; a test that
asserts 'is not None' alone is invalid. Return only a python file."""

def sh(*cmd, cwd, timeout=1800):
    p = subprocess.run(cmd, cwd=cwd, capture_output=True, text=True, timeout=timeout)
    return p.returncode, p.stdout, p.stderr

def diff_paths(repo, base):
    _, out, _ = sh("git", "diff", "--name-only", f"{base}...HEAD", cwd=repo)
    return [l for l in out.splitlines() if l.strip()]

def public_surface(repo, base):
    """Signatures + first docstring line only. Bodies are never read."""
    surface = {}
    _, out, _ = sh("git", "ls-tree", "-r", "--name-only", base, cwd=repo)
    for path in out.splitlines():
        if not path.endswith(".py") or TEST_RE.search(path):
            continue
        rc, src, _ = sh("git", "show", f"{base}:{path}", cwd=repo)
        if rc != 0:
            continue
        try:
            tree = ast.parse(src)
        except SyntaxError:
            continue
        sigs = []
        for node in ast.walk(tree):
            if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef, ast.ClassDef)):
                sig = node.name
                if hasattr(node, "args"):
                    a = node.args
                    names = [x.arg for x in a.posonlyargs + a.args + a.kwonlyargs]
                    sig += f"({', '.join(names)})"
                doc = (ast.get_docstring(node) or "").splitlines()
                sigs.append(f"{sig}  # {doc[0] if doc else ''}")
        if sigs:
            surface[path] = sigs
    return surface

def sealed_spec(repo, issue_text, base):
    return {
        "issue": issue_text,
        "public_surface": public_surface(repo, base),
        "forbidden": "The diff and the new implementation are withheld. Do not guess them.",
    }

def author_oracles(spec, timeout=240):
    endpoint = os.environ["ORACLE_ENDPOINT"]      # chat-completions compatible
    model = os.environ["ORACLE_MODEL"]
    payload = {"model": model, "temperature": 0,
               "messages": [{"role": "system", "content": SYSTEM},
                            {"role": "user", "content": json.dumps(spec, indent=2)}]}
    req = urllib.request.Request(
        endpoint, data=json.dumps(payload).encode(),
        headers={"Content-Type": "application/json",
                 "Authorization": f"Bearer {os.environ.get('ORACLE_KEY', '')}"})
    with urllib.request.urlopen(req, timeout=timeout) as r:
        return json.load(r)["choices"][0]["message"]["content"]

def run_pytest(cwd, rel_path):
    rc, out, err = sh(sys.executable, "-m", "pytest", rel_path, "-v", "--tb=no",
                      "-p", "no:cacheprovider", cwd=cwd)
    status = {}
    for line in (out + err).splitlines():
        m = re.match(r"^(\S+::\S+)\s+(PASSED|FAILED|ERROR|SKIPPED)", line)
        if m:
            status[m.group(1).split("::", 1)[-1]] = m.group(2)
    return status

def classify(repo, tests_src, base):
    with tempfile.TemporaryDirectory() as tmp:
        base_tree = Path(tmp) / "base"
        sh("git", "worktree", "add", "--detach", str(base_tree), base, cwd=repo)
        try:
            target = base_tree / "tests" / GENERATED
            target.parent.mkdir(parents=True, exist_ok=True)
            target.write_text(tests_src)
            base_status = run_pytest(base_tree, f"tests/{GENERATED}")
        finally:
            sh("git", "worktree", "remove", "--force", str(base_tree), cwd=repo)
        live = Path(repo) / "tests" / GENERATED
        live.write_text(tests_src)
        try:
            patch_status = run_pytest(repo, f"tests/{GENERATED}")
        finally:
            live.unlink(missing_ok=True)
    verdicts = {}
    for tid, s in patch_status.items():
        b = base_status.get(tid, "MISSING")
        if s == "PASSED" and b in ("FAILED", "ERROR"):
            verdicts[tid] = "fail_to_pass"    # real evidence
        elif s == "PASSED" and b == "PASSED":
            verdicts[tid] = "base_pass"       # satisfied by unfixed code
        elif s == "FAILED" and b == "PASSED":
            verdicts[tid] = "regression"
        elif s == "FAILED" and b == "FAILED":
            verdicts[tid] = "fails_always"
        else:
            verdicts[tid] = "unknown"
    return verdicts, base_status, patch_status

def oracle_id(tests_src):
    return hashlib.sha256(tests_src.encode()).hexdigest()[:16]

def append_ledger(path, record):
    p = Path(path); p.parent.mkdir(parents=True, exist_ok=True)
    with p.open("a") as f:
        f.write(json.dumps(record, sort_keys=True) + "\n")

if __name__ == "__main__":
    repo, base, issue_file, ledger = sys.argv[1:5]
    issue = Path(issue_file).read_text()
    spec = sealed_spec(repo, issue, base)
    tests_src = author_oracles(spec)
    verdicts, base_status, patch_status = classify(repo, tests_src, base)
    oid = oracle_id(tests_src)
    append_ledger(ledger, {"oracle_id": oid, "base": base,
                           "changed": diff_paths(repo, base),
                           "verdicts": verdicts, "state": "shadow"})
    bad = {k: v for k, v in verdicts.items() if v in ("base_pass", "regression")}
    print(json.dumps({"oracle_id": oid, "counts": {
        s: list(verdicts.values()).count(s) for s in set(verdicts.values())},
        "blocking_findings": bad}, indent=2))
    sys.exit(1 if bad else 0)

Run it as python oracle_split.py . origin/main issue.md tests/oracle_ledger.jsonl. Example report shape (not a measurement from any repo — run it on yours):

{
  "oracle_id": "3f9a1c0d2b7e4a51",
  "counts": {"fail_to_pass": 4, "base_pass": 2, "fails_always": 1},
  "blocking_findings": {"test_parser_returns_list": "base_pass",
                        "test_retry_counts_attempts": "base_pass"}
}

4. The metric that matters: base-pass rate

Track base_pass / (base_pass + fail_to_pass) per patch batch. It is the share of newly authored oracles that the unfixed tree already satisfies. High values mean your spec is too thin, your oracle author is guessing at the diff, or the issue text is a restatement of the code.

Two secondary numbers keep the metric honest: fails_always catches oracles that reproduce a pre-existing bug and can never go green, and unknown catches collection errors, which usually mean the sealed surface omitted something the oracle legitimately needed.

5. Oracle probation instead of a hard gate

A brand-new oracle should not be allowed to block or unblock a merge on its first appearance. Give it a state machine in the ledger:

Ledger state	Gate effect	Promotion rule	Retirement rule
shadow	recorded, never blocks	3 consecutive fail_to_pass verdicts on distinct patches	14 days with no promotion, or 2 base_pass
blocking	failures block the merge	—	2 base_pass, or 1 regression (page a human)
retired	skipped, history kept	—	terminal

The retirement rule is what keeps the suite from rotting into regression guards. An oracle that starts passing on the base tree has stopped describing the fix; it either became redundant or was never specific. Retire it, and keep the hash in the ledger so the diff history stays auditable.

6. Where the second session runs

Running two sessions per pull request is the whole cost of this protocol, and it is a cost worth engineering down. Point ORACLE_ENDPOINT at any chat-completions-compatible endpoint you control; if you are already working inside MonkeyCode, the free model access covers the oracle-authoring call, which is the one call you most want to keep cheap because it must happen on every patch, not just the risky ones.

The job itself is mostly waiting: one model round trip, two pytest subprocesses, a git worktree add and remove. That profile suits the free server option well — run the harness in a disposable container there so oracle authoring never consumes CI minutes on your main runner and never touches a developer laptop's working tree.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

If you already have a gate that runs diff-scoped tests, adding this split is roughly a two-file change: the harness above plus a ledger path. Start in shadow for a week and look at the base_pass count before you let anything block.

7. Limitations, and who should skip this

• The split does not prove spec completeness. Two authors can both miss a behavior. It removes self-confirmation, not uncertainty.
• Sealed specs leak. Signature names and docstrings hint at design. If your public surface is one giant run(config), the oracle author is guessing regardless.
• Probation delays blocking. A genuinely weak oracle blocks nothing for its first three appearances. That is the trade: fewer false merges against slower detection.
• Fixture edits still need their own check. This harness classifies tests, not fixtures; pair it with whatever fixture-integrity rule you already trust.
• Skip it entirely if a human writes the tests before the patch exists, or if a single base-tree pytest run already exceeds your per-PR budget — you cannot afford the two-tree replay, and a half-run version produces unknown verdicts that look like coverage.