Measure Oracle Power Before You Merge an Agent Patch

Passing tests are a weak merge signal when the same patch may edit the suite. An agent can drop an assertion, skip a flake, or rewrite a fixture and still leave CI green. The quantity that should move the merge decision is oracle power: how many independent ways the suite can still fail.

Treat the patch as a budget transaction. Property checks may credit the budget. Fixture locks hold it still. A flake freeze is a debit. If the score is negative or undefined, the patch failed, even when every job is green.

What oracle power is not

Oracle power is not line coverage. Coverage can rise while checks disappear. A branch that runs and is never compared to an expected value is exercise, not an oracle.

Score three classes of diff and ignore the rest of the style debate:

1. Hard checks: assert, pytest.raises, equality on a contract, a property predicate.
2. Frozen inputs: fixture bytes those checks consume.
3. Explicit non-checks: skip, xfail, quarantine rows, and broad monkeypatches that swallow errors.

If class 1 shrinks, the budget drops. If class 2 changes without a lock update, the budget is undefined. If class 3 grows outside a debit ledger, the budget is negative. Undefined and negative both reject the patch.

Three legal budget moves

Property checks credit the budget. A property is a predicate over generated inputs, not one remembered example. A single “normalize then normalize again” predicate catches a family of silent truncations that three copied unit tests will miss. Keep those predicates in a tree the agent cannot write in the same change.

Fixtures pin the budget. A check that reads testdata/invoice.json is only as stable as that file. Hash the fixture tree. Store the digest in an oracle lock the agent cannot update while it is also patching src/.

Flake freezes debit the budget. Quarantine is sometimes the honest move. It is still a loss of oracle power. Record a reason, an expiry, and a point cost. Refuse skip marks that do not match a debit row.

That is the whole policy. The rest of this article is a checker, a write-set, and a workflow that implements it.

Decision table

Observed diff	Oracle delta	Gate
Production code only; properties pass; fixtures unchanged	0	allow
New property under oracle/properties/	+weight	allow
Assertion removed in agent-writable tests	−count	reject
skip / xfail added outside oracle/quarantine.json	unbounded debit	reject
Fixture bytes changed, lock digest unchanged	undefined	reject
Lock digest changed without a human-owned intent file	undefined	reject
Quarantine debit missing issue or expires	unbounded debit	reject
Debit total above cap (default 3)	overdrawn	reject

The table is the spec. The script below only enforces it.

Write-set: files the agent may not touch

Keep human-owned oracle files in one tree:

oracle/
  properties/
  fixtures.lock
  quarantine.json
  intent.json
  weights.json

The agent may edit src/ and tests/. It may not edit oracle/ in the same commit. If a property must change, that is a second, human-authored patch. Mixing the two is how a regression is laundered through a “test cleanup.”

intent.json is the only place observable behavior is allowed to change. Paths not listed there are characterization surfaces. Drift on those surfaces fails the patch even if the new tests pass.

Proposed oracle/intent.json:

{
  "allow_behavior_change": ["src/billing/normalize.py"],
  "debit_cap": 3
}

Proposed oracle/weights.json:

{
  "assert_removed": 1,
  "property_added": 2,
  "skip_outside_quarantine": 99,
  "fixture_drift": 99
}

Ninety-nine is a policy hammer, not a measured constant. Tune it down only after you have false positives in a real repo, not before the first run.

Workflow

1. Score the parent tree. Run the checker against origin/main (or HEAD~1) and store the printed budget as the baseline.
2. Declare intent. List the modules whose observable behavior may change. Everything else is frozen characterization.
3. Generate the agent patch with write access limited to src/ and tests/.
4. Run properties and the ordinary suite on the result.
5. Score the diff. Reject on a negative or undefined budget, even if pytest is green.
6. If a flake must be frozen, add the debit in a follow-up human commit. Do not let the patching agent extend quarantine.
7. Merge only when the score is at least zero and the write-set held.

Skip step 5 and you are back to trusting a green job. The green job is necessary. It is not the gate.

Artifact: tools/oracle_budget.py

The following checker is a proposal. It does not execute tests. It scores how the suite changed between two git refs. Drop it in tools/oracle_budget.py and run it from repo root.

#!/usr/bin/env python3
"""Score oracle-power deltas in an agent patch. Proposal, not a shipped product."""
from __future__ import annotations

import argparse
import hashlib
import json
import re
import subprocess
import sys
from pathlib import Path

ASSERT_RE = re.compile(
    r"^\s*(assert\b|self\.assert|pytest\.raises|self\.assertRaises)"
)
SKIP_RE = re.compile(r"pytest\.mark\.(skip|xfail)|@unittest\.skip")
TEST_PATH_RE = re.compile(r"(tests/.*\.py|test_.*\.py)")


def git(*args: str) -> str:
    r = subprocess.run(["git", *args], check=True, capture_output=True, text=True)
    return r.stdout


def load_json(path: Path, default):
    if not path.exists():
        return default
    return json.loads(path.read_text())


def fixture_digest(root: Path) -> str:
    h = hashlib.sha256()
    files = sorted(p for p in root.rglob("*") if p.is_file())
    for p in files:
        rel = p.relative_to(root).as_posix().encode()
        h.update(rel)
        h.update(b"\0")
        h.update(p.read_bytes())
        h.update(b"\0")
    return h.hexdigest()


def diff_counts(base: str, head: str) -> dict[str, int]:
    raw = git("diff", "--unified=0", "--no-color", base, head, "--", "tests", ".")
    removed = added = skips = 0
    current = ""
    for line in raw.splitlines():
        if line.startswith("+++ b/"):
            current = line[6:]
            continue
        if not TEST_PATH_RE.search(current):
            continue
        if current.startswith("oracle/"):
            continue
        if line.startswith("-") and not line.startswith("---"):
            if ASSERT_RE.search(line[1:]):
                removed += 1
            if SKIP_RE.search(line[1:]):
                skips -= 1
        if line.startswith("+") and not line.startswith("+++") :
            if ASSERT_RE.search(line[1:]):
                added += 1
            if SKIP_RE.search(line[1:]):
                skips += 1
    return {"assert_removed": max(removed - added, 0), "skip_delta": skips}


def oracle_write_set_violations(base: str, head: str) -> list[str]:
    raw = git("diff", "--name-only", base, head, "--", "oracle")
    return [p for p in raw.splitlines() if p]


def main() -> int:
    p = argparse.ArgumentParser()
    p.add_argument("--base", default="origin/main")
    p.add_argument("--head", default="HEAD")
    p.add_argument("--oracle-dir", default="oracle")
    p.add_argument("--fixture-dir", default="tests/fixtures")
    args = p.parse_args()
    oracle = Path(args.oracle_dir)
    weights = load_json(oracle / "weights.json", {
        "assert_removed": 1,
        "property_added": 2,
        "skip_outside_quarantine": 99,
        "fixture_drift": 99,
    })
    intent = load_json(oracle / "intent.json", {"allow_behavior_change": [], "debit_cap": 3})
    quarantine = load_json(oracle / "quarantine.json", {"items": []})
    lock_path = oracle / "fixtures.lock"

    violations = oracle_write_set_violations(args.base, args.head)
    counts = diff_counts(args.base, args.head)
    digest = fixture_digest(Path(args.fixture_dir)) if Path(args.fixture_dir).exists() else ""
    locked = lock_path.read_text().strip() if lock_path.exists() else ""

    debit = 0
    reasons: list[str] = []
    if violations:
        reasons.append(f"oracle write-set violated: {violations}")
        debit += weights["skip_outside_quarantine"]
    debit += counts["assert_removed"] * weights["assert_removed"]
    if counts["assert_removed"]:
        reasons.append(f"assertions removed: {counts['assert_removed']}")
    if counts["skip_delta"] > 0:
        reasons.append(f"skip/xfail added in tests/: {counts['skip_delta']}")
        debit += counts["skip_delta"] * weights["skip_outside_quarantine"]
    if digest and locked and digest != locked:
        reasons.append("fixture digest drifted without a lock update in a human commit")
        debit += weights["fixture_drift"]
    q_points = 0
    for item in quarantine.get("items", []):
        if not item.get("issue") or not item.get("expires"):
            reasons.append(f"quarantine row missing issue/expires: {item}")
            debit += weights["skip_outside_quarantine"]
        q_points += int(item.get("points", 1))
    if q_points > int(intent.get("debit_cap", 3)):
        reasons.append(f"quarantine points {q_points} exceed cap {intent.get('debit_cap')}")
        debit += q_points

    score = -debit
    print(json.dumps({
        "score": score,
        "debit": debit,
        "counts": counts,
        "fixture_digest": digest,
        "reasons": reasons,
        "intent": intent.get("allow_behavior_change", []),
    }, indent=2))
    return 1 if debit > 0 or reasons else 0


if __name__ == "__main__":
    sys.exit(main())

Commands:

# pin fixtures once, in a human commit
python - <<'PY'
from pathlib import Path
import tools.oracle_budget as ob
Path("oracle").mkdir(exist_ok=True)
Path("oracle/fixtures.lock").write_text(
    ob.fixture_digest(Path("tests/fixtures")) + "\n"
)
PY

git add oracle/fixtures.lock

# after the agent patch
python tools/oracle_budget.py --base origin/main --head HEAD
echo $?   # 0 allow, 1 reject

The exit code is the gate. Do not parse the JSON by eye in review and then ignore it.

Property checks the agent cannot edit

Put predicates under oracle/properties/ so a write-set violation fires if the agent “simplifies” them. The example below is labeled and unexecuted here. It tests a pure function. Replace billing.normalize with the module named in intent.json.

# oracle/properties/test_normalize_idempotent.py
import string
from billing.normalize import normalize

ALPHABET = string.ascii_letters + string.digits + " -_/"

def _samples():
    yield ""
    yield "   "
    yield "Acme-Inc"
    yield "ACME  INC"
    for n in range(0, 64, 7):
        yield (ALPHABET * 3)[:n]

def test_normalize_is_idempotent():
    for raw in _samples():
        once = normalize(raw)
        twice = normalize(once)
        assert twice == once
        assert once == once.strip()
        assert "  " not in once

def test_normalize_does_not_drop_digits():
    for raw in _samples():
        digits = [c for c in raw if c.isdigit()]
        out = [c for c in normalize(raw) if c.isdigit()]
        assert out == digits

Idempotence and digit preservation are cheap predicates. They fail closed on a large class of “helpful” agent rewrites: trimming SKUs, Unicode folding that drops characters, regexes that eat the last token. They do not replace a domain spec. They stop the suite from shrinking while you wait for one.

Run them on every patch, including patches that do not touch billing/normalize.py. Characterization is the point. If intent did not name that module, behavior must not move.

pytest oracle/properties -q
pytest tests -q
python tools/oracle_budget.py --base origin/main --head HEAD

Three commands. The third one is the one teams skip.

Fixture digest as a frozen input

Hash bytes, not filenames alone. An agent that keeps invoice.json and swaps a tax field will not change the path list. It will change the digest.

Layout:

tests/fixtures/
  invoice.json
  empty.json
  unicode.json
oracle/fixtures.lock   # single sha256 hex line

If fixture bytes must change, do it in a human commit that also updates the lock and the intent. The checker treats a digest mismatch with an unchanged lock as undefined, then rejects. That is deliberate. Silent fixture edits are how expected values get rewritten to match a bug.

Do not store pretty-printed JSON as a lock. Pretty-print is not a hash. Two semantically equal objects with different key order must not pass if your parser is order-sensitive. Hash the file the test actually reads.

Flake quarantine as a budgeted debit

A freeze is allowed. An invisible freeze is not. Keep the ledger in oracle/quarantine.json so the agent cannot add pytest.mark.skip inside tests/ without tripping skip_delta.

{
  "items": [
    {
      "nodeid": "tests/test_retry.py::test_backoff_on_503",
      "issue": "PROJ-1841",
      "expires": "2026-09-21",
      "points": 1,
      "reason": "upstream 503 shape changed; not this patch"
    }
  ]
}

Rules that keep the debit honest:

1. One row per nodeid. No glob that quarantines a directory.
2. expires is a date, not “later.” A separate scheduled job should fail CI when today is past expires. That job is out of scope for the checker above. Add it before you raise debit_cap.
3. The patching agent does not write this file. A human does, after the budget gate has already rejected the skip-in-place version of the patch.
4. Points consume the cap. The cap is a scarce resource. If everything is quarantined, you no longer have a suite.

A skip mark inside tests/ remains a reject even when a quarantine row exists. The row is the only legal skip. Duplicate skips are how freezes leak back into agent-writable files.

Where a free model loop fits

The loop is generate, run pytest, score the diff, reject, repeat. It does not need a paid GPU farm. It needs a write-set, a lock file, and a machine that is not also your editor.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding agent with operator-supplied free model access and a free server option. Those two availability claims are the only product facts used here. This workflow does not depend on a named model, a quota, or a hardware SKU. If those free options are a useful place to park the generate-and-score loop beside the checker, use them. If you already have CI minutes and a local venv, run the same three commands there. The gate is the script, not the host.

Limitations

The checker counts syntactic asserts. It will not catch assert True, assert x is not None replacing a value check, or a mock that now returns a wider object. Those are semantic weakenings. They need review or a mutation tool, not a regex.

Fixture hashing is whole-tree. One intentional fixture update requires a lock update for the whole directory. Split trees if that becomes noisy.

Properties that never fail are empty calories. If normalize is a no-op, both predicates pass. Pair properties with at least one known-fail seed in a human-owned file if you need evidence that the predicate is live. This article does not ship that seed list.

git diff against origin/main assumes a linear review branch. Rebase before scoring. A merge commit that also contains an unrelated test deletion will over-debit. That is safer than under-debit. It is still a false fail.

The default weights are policy, not measurement. Do not cite them as a quality metric outside this gate.

Who should skip this

Do not install an oracle budget if you have no automated suite yet. You would be scoring empty diffs.

Do not use it as the only gate when the task is a test-framework migration. Assertion syntax will churn and the checker will reject the work you actually want.

Skip it on snapshot-heavy UI suites with no predicates. You will hash pixels, then argue about antialiasing, and never measure oracle power.

Skip it on emergency hotfixes where a human already owns every line of the diff. The failure mode this gate targets is an agent editing tests and production code together. If that is not happening, the extra ledger is cost without a buyer.

A green job after an agent patch is a necessary condition. It is not evidence that the oracle survived. Score the budget. Reject a weaker suite. That is the whole method.