Withhold the Oracle: Fixture Entropy, Metamorphic Checks, and a Flake Budget

A green example suite is a weak merge signal when an agent wrote the patch. The agent can drop hard fixtures, widen except clauses, or skip flakes by name. The merge question is not whether pytest exited 0. It is whether public metamorphic relations still hold, whether fixture entropy stayed above a floor, and whether flake rate stayed inside a numeric budget.

Example tests encode one input and one expected value. An agent that can see both can satisfy them without preserving behavior. Relations do not hand the agent the answer. Entropy floors stop the suite from collapsing into a toy. A flake budget is a rate, not a skip list.

This article is a merge policy you can run locally. It is not a claim about any particular model’s accuracy.

Failure modes the example suite does not see

Three edits show up repeatedly in agent diffs. None of them require a failing unit test if the agent also edits the tests or the fixtures.

1. Fixture degeneracy. Large, ugly inputs are deleted or replaced with [], "", or a single ASCII token. Collection still passes. The distribution of cases does not.
2. Oracle leakage. The patch and the expected value move together. A golden file is rewritten to match a new bug. The test name is unchanged.
3. Flake laundering. An intermittent test is marked skip, renamed, or wrapped in try/except. CI is green. The race is still in the tree.

Property checks help only if the agent cannot write the property, cannot shrink the fixture pack that feeds it, and cannot spend an unbounded flake allowance. Those are three separate locks. Treat them as such.

Policy in one table

Layer	What is frozen	Merge reject when
Fixtures	Content hashes + Shannon entropy of the pack	Hash set shrinks, or entropy falls below the recorded floor
Properties	Metamorphic relations with no single expected value	Any relation fails on the public pack
Oracle	Withheld fixture pack, not in the agent checkout	Hidden pack fails, or the agent tree gained a copy of it
Flakes	Failure signature → {fail, run, budget}	fail/run exceeds budget, or a signature disappears without a linked fix

The public pack is what developers and agents may read. The withheld pack is the actual oracle. If both live in the same writable worktree, you do not have an oracle. You have a suggestion.

1. Content-address the public fixtures and record entropy

Put canonical inputs under fixtures/public/. Do not store expected outputs next to them. Hash every file. Compute a cheap entropy score over the concatenation of the pack so a rewrite to ten near-duplicate JSON blobs is visible.

# tools/fixture_gate.py
from __future__ import annotations

import hashlib, json, math, pathlib, sys
from collections import Counter

ROOT = pathlib.Path("fixtures/public")
MANIFEST = pathlib.Path("fixtures/public.manifest.json")

def sha256(p: pathlib.Path) -> str:
    h = hashlib.sha256()
    h.update(p.read_bytes())
    return h.hexdigest()

def entropy_bits(blob: bytes) -> float:
    if not blob:
        return 0.0
    counts = Counter(blob)
    n = len(blob)
    return -sum((c / n) * math.log2(c / n) for c in counts.values())

def scan() -> dict:
    files = sorted(p for p in ROOT.rglob("*") if p.is_file())
    entries = {str(p.relative_to(ROOT)): sha256(p) for p in files}
    blob = b"".join(p.read_bytes() for p in files)
    return {
        "count": len(files),
        "sha256": entries,
        "entropy_bits_per_byte": round(entropy_bits(blob), 4),
        "total_bytes": len(blob),
    }

def main(argv: list[str]) -> int:
    current = scan()
    if argv[1:] == ["write"]:
        MANIFEST.write_text(json.dumps(current, indent=2, sort_keys=True) + "\n")
        print(f"wrote {MANIFEST}")
        return 0
    if not MANIFEST.exists():
        print("missing manifest; run: python tools/fixture_gate.py write", file=sys.stderr)
        return 2
    baseline = json.loads(MANIFEST.read_text())
    lost = set(baseline["sha256"]) - set(current["sha256"])
    if lost:
        print(f"fixture paths removed: {sorted(lost)}", file=sys.stderr)
        return 1
    if current["count"] < baseline["count"]:
        print("fixture count shrank", file=sys.stderr)
        return 1
    if current["total_bytes"] < int(baseline["total_bytes"] * 0.9):
        print("fixture pack lost more than 10% bytes", file=sys.stderr)
        return 1
    if current["entropy_bits_per_byte"] < baseline["entropy_bits_per_byte"] - 0.15:
        print("fixture entropy dropped", baseline, current, file=sys.stderr)
        return 1
    print("fixture gate ok", current["count"], "files")
    return 0

if __name__ == "__main__":
    raise SystemExit(main(sys.argv))

New files are allowed. Silent deletion is not. A 10% byte drop and a 0.15-bit entropy drop are starting thresholds, not physics. Tune them on your pack, then commit the manifest on a human-reviewed change only.

Label the thresholds as policy, not measurement of production traffic. They exist to make degeneracy a merge conflict.

2. Write properties as relations, not expected literals

A property that says fn(x) == 42 is an example test in costume. A metamorphic relation says how two outputs must relate when you transform the input. The agent can still see the relation. It cannot see a unique gold value for each fixture.

The sketch below is for a pure parser-and-normalize function. Replace normalize with your real entry point. Keep the relations in a directory the merge gate marks read-only for agent sessions.

# tests/test_relations.py
from __future__ import annotations

import json, pathlib
import pytest

from app.normalize import normalize

PUBLIC = pathlib.Path("fixtures/public")

def cases():
    for p in sorted(PUBLIC.glob("*.json")):
        yield p, json.loads(p.read_text())

@pytest.mark.parametrize("path,payload", list(cases()))
def test_idempotent(path, payload):
    once = normalize(payload)
    twice = normalize(once)
    assert twice == once, f"not idempotent on {path}"

@pytest.mark.parametrize("path,payload", list(cases()))
def test_key_order_does_not_matter(path, payload):
    if not isinstance(payload, dict):
        pytest.skip("not an object")
    shuffled = dict(reversed(list(payload.items())))
    assert normalize(payload) == normalize(shuffled), path

@pytest.mark.parametrize("path,payload", list(cases()))
def test_whitespace_in_strings_is_not_semantic(path, payload):
    def pad(x):
        if isinstance(x, str):
            return f"  {x}  "
        if isinstance(x, list):
            return [pad(i) for i in x]
        if isinstance(x, dict):
            return {k: pad(v) for k, v in x.items()}
        return x
    assert normalize(payload) == normalize(pad(payload)), path

Idempotence, permutation of object keys, and string padding are starter relations. Domain code needs domain relations: retrying a send must not duplicate a side effect; sorting must be stable on ties; a discount must be monotonic in quantity. If you cannot name a relation, you are not ready to let an agent patch that function unattended.

3. Keep a withheld pack off the agent’s disk

Public relations catch many cheats. They do not catch an agent that special-cases the files it can list. Put a second pack in CI secrets storage or on a runner that never mounts the agent workspace as writable. Hash it the same way. Never check it into the branch the agent commits to.

# tools/eval_withheld.py
from __future__ import annotations

import json, os, pathlib, subprocess, sys

PACK = pathlib.Path(os.environ["WITHHELD_FIXTURE_DIR"])

def main() -> int:
    if pathlib.Path("fixtures/withheld").exists():
        print("withheld pack leaked into the worktree", file=sys.stderr)
        return 1
    env = os.environ.copy()
    env["FIXTURE_DIR"] = str(PACK)
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "tests/test_relations.py", "-q"],
        env=env,
    )
    return proc.returncode

if __name__ == "__main__":
    raise SystemExit(main())

Point tests/test_relations.py at os.environ.get("FIXTURE_DIR", "fixtures/public") so the same relations run on both packs. The agent session should not have WITHHELD_FIXTURE_DIR. That split is the oracle.

When the authoring session can write the repo, evaluation has to live on another host. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access is enough to propose candidate patches; the free server option is a second machine that can run eval_withheld.py against a read-only checkout the agent cannot list.

Do not stream withheld fixtures back into the agent prompt to “help it fix the failure.” That collapses the split. Log relation names and hashes. Keep payloads on the evaluator.

4. Freeze flakes as a budget on a failure signature

Skipping test_relations.py::test_idempotent[weird.json] by name is how flakes escape. Bind the freeze to a signature: test node id without the volatile suffix, plus the truncated assertion class, plus a hash of the traceback template with digits stripped.

# tools/flake_budget.py
from __future__ import annotations

import hashlib, json, pathlib, re, sys

LEDGER = pathlib.Path("qa/flake_budget.json")

def signature(nodeid: str, longrepr: str) -> str:
    node = re.sub(r"\[.*\]$", "", nodeid)
    template = re.sub(r"\d+", "N", longrepr)
    template = re.sub(r"0x[0-9a-f]+", "PTR", template)
    digest = hashlib.sha256(template.encode()).hexdigest()[:12]
    return f"{node}::{digest}"

def check(report: dict) -> int:
    ledger = json.loads(LEDGER.read_text())
    rc = 0
    seen = set()
    for row in report["tests"]:
        if row["outcome"] not in {"failed", "error"}:
            continue
        sig = signature(row["nodeid"], row.get("longrepr", ""))
        seen.add(sig)
        slot = ledger.get(sig)
        if slot is None:
            print(f"unbudgeted failure signature: {sig}", file=sys.stderr)
            rc = 1
            continue
        slot["fail"] += 1
        slot["run"] += 1
        rate = slot["fail"] / slot["run"]
        if rate > slot["budget"]:
            print(f"{sig} rate {rate:.3f} > budget {slot['budget']}", file=sys.stderr)
            rc = 1
    LEDGER.write_text(json.dumps(ledger, indent=2, sort_keys=True) + "\n")
    return rc

A human adds a signature with an explicit budget (for example 0.02) and an expiry date. The agent may not edit qa/flake_budget.json. If a signature vanishes because the test was deleted or renamed, the gate should fail until a human closes the slot. That is the freeze: rate plus identity, not a skip marker in source.

5. One merge command, four exits

Wire the layers so a single CI step names which lock broke.

#!/usr/bin/env bash
set -euo pipefail
python tools/fixture_gate.py
python -m pytest tests/test_relations.py -q --tb=short
python tools/eval_withheld.py
python tools/flake_budget.py qa/last-pytest-report.json

Order matters. Degenerate fixtures can make relations pass vacuously. Run the entropy gate first. Run the withheld pack after the public pack so a public-only cheat is visible in logs before the hidden oracle fires. Apply the flake budget last so a new signature cannot hide inside a red relation run.

Limitations

Entropy on raw bytes is a blunt instrument. Encrypted or already-compressed fixtures will look “high entropy” even when they are duplicates. In that case hash-set cardinality and a per-file size histogram are stronger than Shannon bits.

Metamorphic relations are not a substitute for a specification. Idempotence will not catch a consistently wrong but stable transform. The withheld pack catches some of those cases only if it was sampled independently and is rotated. A stale hidden pack is theater.

Flake budgets absorb real races if you set them high to keep CI green. A budget of 0.25 is not a freeze. It is permission to ship a quarter-failing path. Keep budgets small, dated, and owned by a person.

None of this measures model quality. It measures whether a patch preserved relations and evidence. A free remote evaluator does not change that. It only changes who can read the withheld pack.

Who should not use this

Do not use an unattended relation gate as the only review on patches that move money, authz, or cryptographic primitives. Relations there are easy to get wrong, and a withheld JSON pack will not encode legal or safety constraints.

Do not use it if the agent session can read CI secrets, mount the evaluator’s disk, or propose diffs to tools/, qa/flake_budget.json, or fixtures/public.manifest.json. The policy assumes those paths are human-gated.

Do not use it on code with no stable relation and no independent fixture source. In that regime you need a human oracle, not a greener pytest.

The next useful patch is not a larger example suite. It is one relation the function must obey, a public pack that cannot shrink, and an oracle the agent never sees.