Proposal Hosts Do Not Score Merges

A passing suite on the same host that produced an agent patch is correlated evidence, not merge evidence. The model, the checkout, and the test process shared a disk. Shared disks leak.

Score the patch on a machine the agent cannot write. Keep three oracles there: fixture hashes, time-bounded properties, and a flake freeze imported from main. None of those files sit in the agent's workspace.

The trust boundary

Agent patches now arrive from cheap remote loops. A free model on a free server is a reasonable place to propose a diff. It is a poor place to certify one.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode offers free model access and a free server option. Those are useful for generating candidate patches. They are not a substitute for a local, model-free verifier.

Remote unit tests can still fail usefully. A red remote job is a cheap reject. A green remote job is only a hint. The hint is allowed to be wrong.

The correlation is mechanical, not mystical. The proposal host can rewrite fixtures, insert skip, and add tautological tests in the same tree it then executes. Pytest green follows from that tree. Independence does not.

What the local re-score actually proves

The local loop does not ask whether some tests passed. It asks three narrower questions.

1. Did any fixture bytes change without an explicit, human-owned allowlist prefix?
2. Do the properties that cover the touched modules still hold under a fixed trial budget and a wall clock?
3. Did the diff add skip, xfail, or equivalent markers that are not already frozen on main?

If any answer is yes in the wrong direction, the score is reject. Tests the agent added in the same patch are recorded as intent. They are not counted as proof.

This is a proposed workflow, not a production study. No pass-rate, latency, or model ranking is claimed.

Layout: oracles live outside the checkout

Keep the clone the agent can mutate separate from the oracle directory. The split is the control. Convenience copies are how the control disappears.

/trusted/oracles/          # not writable inside the agent sandbox
  score_contract.toml
  fixtures.lock
  properties.py
  flake_freeze.json
  rescore.py
/trusted/work/repo/        # patch applied here; oracles mounted read-only

The contract file stays small. Example only:

# /trusted/oracles/score_contract.toml
[score]
property_trials = 80
property_timeout_s = 45
max_changed_files = 12

[fixtures]
lock_path = "fixtures.lock"
allow_human_keys = ["docs/screenshots/"]

[flake]
freeze_path = "flake_freeze.json"
forbid_new_skip_markers = [
  "pytest.mark.skip",
  "pytest.mark.xfail",
  "@unittest.skip",
]

Do not commit score_contract.toml into the branch the agent can push. If the contract is in the write set, the score is circular.

Step 1 — Apply the remote diff on a clean tree

Fetch the patch as a file. Do not import a remote CI artifact that already reports passed.

git fetch origin main
git checkout --detach origin/main
git apply --check /tmp/agent.patch
git apply /tmp/agent.patch
git diff --name-only origin/main > /tmp/touched.txt
wc -l /tmp/touched.txt

--check first. A patch that does not apply is an unscored reject, not a retry signal. Cap the path count with max_changed_files. Wide diffs are a review problem. Refuse them before properties run.

Step 2 — Hash fixtures the patch can see

Fixtures are data. Data that changes under an agent diff is a behavior change, even when the in-repo suite stays green.

# /trusted/oracles/rescore.py — example verifier, unexecuted in this article
from __future__ import annotations

import hashlib, json, os, re, subprocess, sys, time
from pathlib import Path

REPO = Path(os.environ["REPO_ROOT"])
ORACLE = Path(os.environ["ORACLE_ROOT"])

def sha256_file(p: Path) -> str:
    h = hashlib.sha256()
    with p.open("rb") as f:
        for chunk in iter(lambda: f.read(1 << 16), b""):
            h.update(chunk)
    return h.hexdigest()

def fixture_verdict(allow_prefixes: list[str]) -> list[str]:
    lock = json.loads((ORACLE / "fixtures.lock").read_text())
    problems = []
    for rel, expected in lock.items():
        p = REPO / rel
        if not p.is_file():
            problems.append(f"fixture missing: {rel}")
            continue
        got = sha256_file(p)
        if got == expected:
            continue
        if any(rel.startswith(prefix) for prefix in allow_prefixes):
            continue
        problems.append(
            f"fixture drift: {rel} expected={expected[:12]} got={got[:12]}"
        )
    return problems

fixtures.lock is produced on main by a human-owned job. The agent never regenerates it. Missing files are drift. Drift is reject.

A typical fixtures.lock row is a repo-relative path plus a hex digest. Example schema, not live inventory:

{
  "tests/goldens/window.json": "ab12cd34ef56",
  "tests/goldens/empty_body.bin": "98fe76dc54ba"
}

Step 3 — Run properties with a wall clock the agent does not set

Properties belong in the oracle directory, not in tests/ inside the patch. In-repo tests are editable. Oracle properties are not.

def too_wide(max_files: int) -> list[str]:
    files = subprocess.check_output(
        ["git", "diff", "--name-only", "origin/main"],
        cwd=REPO,
        text=True,
    ).splitlines()
    files = [f for f in files if f.strip()]
    if len(files) > max_files:
        return [f"diff too wide: {len(files)} files > {max_files}"]
    return []

def run_properties(trials: int, timeout_s: int) -> list[str]:
    env = os.environ.copy()
    env["PROPERTY_TRIALS"] = str(trials)
    start = time.monotonic()
    try:
        proc = subprocess.run(
            [sys.executable, str(ORACLE / "properties.py")],
            cwd=REPO,
            env=env,
            capture_output=True,
            text=True,
            timeout=timeout_s,
        )
    except subprocess.TimeoutExpired:
        return [f"property timeout after {timeout_s}s"]
    elapsed = time.monotonic() - start
    if proc.returncode != 0:
        tail = (proc.stdout + proc.stderr)[-2000:]
        return [f"property fail rc={proc.returncode} elapsed={elapsed:.1f}s\n{tail}"]
    return []

Keep assertions in ORACLE and imports pointed at REPO. Sketch only:

# /trusted/oracles/properties.py — sketch, replace the module the diff touches
import os, sys
from pathlib import Path

trials = int(os.environ["PROPERTY_TRIALS"])
sys.path.insert(0, os.environ["REPO_ROOT"])

from parse_limits import clamp_window  # example symbol

def check_clamp_window() -> None:
    for width in range(trials):
        out = clamp_window(0, width, width)
        assert 0 <= out[0] <= out[1] <= width

if __name__ == "__main__":
    check_clamp_window()

Eighty trials is a budget, not a coverage claim. Raise it only on the trusted host. If the patch deletes clamp_window, the import fails, and the score is reject. That failure is the signal.

Step 4 — Compare skip markers to the freeze imported from main

Flaky tests exist. Freezing them is an operations decision. It is not a patch-author decision.

SKIP_RE = re.compile(
    r"pytest\.mark\.(skip|xfail)|@unittest\.skip|addSkip\(|self\.skipTest\("
)

def added_skip_markers(base: str = "origin/main") -> list[str]:
    diff = subprocess.check_output(
        ["git", "diff", "-U0", base, "--", "*.py"],
        cwd=REPO,
        text=True,
    )
    hits = []
    current = None
    for line in diff.splitlines():
        if line.startswith("+++ b/"):
            current = line[6:]
            continue
        if line.startswith("+") and not line.startswith("+++"):
            if SKIP_RE.search(line) and current:
                hits.append(f"{current}: {line[1:].strip()}")
    return hits

def flake_verdict() -> list[str]:
    freeze = json.loads((ORACLE / "flake_freeze.json").read_text())
    allowed = set(freeze.get("nodeids", []))
    problems = []
    for hit in added_skip_markers():
        if not any(key in hit for key in allowed):
            problems.append(f"unfrozen skip introduced: {hit}")
    return problems

flake_freeze.json is copied from main CI annotations. Example schema:

{
  "source": "main-ci",
  "nodeids": ["tests/test_net.py::test_retry_on_idle"]
}

The re-score does not expire, grow, or shrink that list. Expiry is a human job on main. The agent patch is not that job.

Step 5 — Emit a score the write set cannot see

def main() -> int:
    allow_prefixes = ["docs/screenshots/"]
    problems = []
    problems.extend(too_wide(max_files=12))
    problems.extend(fixture_verdict(allow_prefixes))
    problems.extend(run_properties(trials=80, timeout_s=45))
    problems.extend(flake_verdict())
    report = ORACLE / "last_score.json"
    report.write_text(json.dumps({"reject": problems}, indent=2) + "\n")
    if problems:
        print("REJECT")
        for item in problems:
            print(item)
        return 2
    print("ACCEPT")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

Wire it with two environment variables and no network:

export REPO_ROOT=/trusted/work/repo
export ORACLE_ROOT=/trusted/oracles
python /trusted/oracles/rescore.py
echo $?

Exit 2 is reject. Exit 0 is accept. Do not map remote pytest green onto those codes.

Failure analysis

Read the reject list before widening budgets. The first line is usually the real cause.

• Fixture drift plus green in-repo tests: goldens were rewritten to match new behavior. Reject. Restoring the lock is not optional.
• Property timeout: unbounded work, a hang, or accidental quadratic cost in the patch. Reject. Do not raise property_timeout_s from inside the diff.
• Import error in properties.py: a public symbol moved or vanished. Reject. The oracle still names the old contract.
• Unfrozen skip or xfail: flake laundering. Reject. Only main may add freeze entries.
• Diff too wide: scoring is the wrong tool. Reject and split the patch.

Two independent reds in one report still count as one reject. Do not average them.

Decision matrix

Signal	Where it ran	If red	If green
Agent unit tests	proposal host (free server or laptop sandbox)	cheap reject	hint only
Patch apply	trusted host	reject (unscored)	continue
File-count cap	trusted host	reject	continue
Fixture lock	trusted host	reject	continue
Oracle properties	trusted host	reject	continue
New skip/xfail vs freeze	trusted host	reject	continue
Human review	trusted humans	reject	merge candidate

The matrix is the reusable artifact. The scripts are examples. Substitute the language. Keep the split.

What this does not prove

Local re-score does not prove the patch is useful. Properties encode invariants the oracle author already believes. They will not invent domain rules that were never written down.

It does not prove the proposal host is safe. A free server can still see whatever checkout, token, or .env you copied onto it. Do not copy those.

It does not replace review of public API changes. A property can hold while a return value becomes less useful. That is a product question, not a hash question.

Limitations and who should not use this

Do not use this loop if generation and tests already share one trusted runner that the agent cannot write. You already have the isolation. A second score file would be ceremony.

Do not use it for documentation-only diffs. Screenshot fixture locks may still apply. Properties will not.

Do not use it as a capacity plan. Free model access and a free server option are availability claims, not throughput, hardware, or uptime guarantees. Queueing and preemption are unspecified here because they were not measured for this article.

Do not copy properties.py into the repo for convenience. Convenience is how the write set swallows the oracle.

Teams without a second machine can approximate the split with a second user and a read-only mount. Same laptop is acceptable only if the agent process cannot open ORACLE_ROOT for write. If that constraint is theater, skip the workflow.

Close

Proposal is cheap. Proof is local. If a remote agent loop already produces diffs, copy the oracle layout and the decision matrix first. The proposal half can stay on a free model and a free server; the vote on merge should not.