Zero Weight on Agent Tests: A Three-Input Merge Predicate

Agent-written tests should not vote on merge. A patch that authors its own suite can paint the bar green without preserving a single invariant. The merge predicate proposed here takes three inputs only: property-check results stored outside the diff, content-addressed fixture hashes, and time-boxed freeze leases. Agent tests still run. Their weight is zero.

This is a proposed harness, not a report of production metrics. Treat the code as a labeled example you can run locally and then re-run on a machine the laptop did not seed.

Why a suite is the wrong type

A test suite returns a bag of pass/fail bits. Merge needs a decision with precedence. Property violations and fixture drift are not the same event as a flaky clock test. Collapsing them into one green bar hides the only signal that should block the patch.

Agent patches make the type error worse. The model can add assertions that restate the implementation. Coverage rises. The invariant does not. If those assertions sit in the same tree the agent can write, they are not oracles. They are part of the diff.

The fix is not a stricter linter on assert text. The fix is to remove agent tests from the vote and keep three voters the agent cannot elect.

Decision table

Encode precedence before you write a runner. The table is the artifact. The code only implements it.

properties (oracles/)	fixture lock	freeze lease	merge
any fail	*	*	block
pass	hash mismatch	*	block
pass	match	none needed	allow
pass	match	valid lease, captured corpus, not expired	allow
pass	match	freeze requested, no corpus	block
pass	match	expired lease	block
pass	match	lease on a property or fixture test	block
error / timeout in oracles	*	*	block

Read the table top-down. A property failure never becomes a freeze. A fixture mismatch never becomes a freeze. Freeze is a lease over a classified non-deterministic test that already has a replayable corpus. Everything else is a block.

Agent-authored files under tests/agent/ do not appear in the table. They may be logged. They do not change the cell.

Repo layout the agent cannot vote from

Keep write permission and merge weight on different paths.

repo/
  src/                     # agent may patch
  tests/agent/             # agent may add tests; weight 0
  oracles/                 # humans/CI only; weight 1
    test_properties.py
    conftest.py
  fixtures/
    corpus/                # input blobs, not expected outputs
    fixtures.lock.json     # sha256 of corpus + schema
  qa/
    freeze_leases.json     # time-boxed, corpus-bound
    merge_predicate.py
    classify.yaml          # which tests may ever freeze

oracles/ is not a second copy of unit tests. It holds properties over public behavior: round-trip, monotonicity, idempotence, bounds, and “unknown keys stay unknown.” Fixtures are inputs. They are not golden stdout the agent can regenerate. Freeze leases name a test id, a corpus hash, an expiry, and a reason code. Missing fields mean the lease is invalid.

If your agent can open a PR that touches oracles/, fixtures.lock.json, or qa/, the predicate is theater. Enforce that with CODEOWNERS or a path filter in CI, not with a prompt.

Implement the three voters

The following is a self-contained example. It does not claim to be a published library.

1. Properties live outside the diff

# oracles/test_properties.py
from datetime import date, timedelta
from src.billing import accrue, invert

def test_accrue_never_negative():
    for cents in (0, 1, 99, 10_000):
        for days in (0, 1, 30, 365):
            start = date(2026, 1, 1)
            end = start + timedelta(days=days)
            assert accrue(cents, start, end) >= 0

def test_accrue_invert_round_trip():
    start = date(2026, 9, 1)
    end = date(2026, 9, 18)
    for cents in (0, 50, 2_499):
        forward = accrue(cents, start, end)
        assert invert(forward, start, end) == cents

def test_unknown_rate_code_is_rejected():
    start = date(2026, 9, 1)
    try:
        accrue(100, start, start, rate_code="not-a-rate")
    except ValueError as exc:
        assert "rate" in str(exc).lower()
    else:
        raise AssertionError("unknown rate_code must raise")

These checks do not import anything from tests/agent/. They do not read files the patch created. If the implementation changes a formula, the inverse or the bound fails here even when every agent test still passes.

2. Fixtures are content-addressed inputs

# qa/merge_predicate.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path
from datetime import datetime, timezone

ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "fixtures" / "corpus"
LOCK = ROOT / "fixtures" / "fixtures.lock.json"
LEASES = ROOT / "qa" / "freeze_leases.json"
CLASSIFY = ROOT / "qa" / "classify.yaml"


def sha256_file(path: Path) -> str:
    h = hashlib.sha256()
    h.update(path.read_bytes())
    return h.hexdigest()


def fixture_status() -> str:
    lock = json.loads(LOCK.read_text())
    expected = lock["files"]
    actual = {
        p.relative_to(CORPUS).as_posix(): sha256_file(p)
        for p in sorted(CORPUS.rglob("*"))
        if p.is_file()
    }
    if actual != expected:
        return "mismatch"
    return "match"

Lock files record hashes of inputs, plus a schema version. They do not record expected return values. If an agent “fixes” a fixture by rewriting the blob to match a new bug, the lock mismatches and the table blocks. That is the point.

Regenerate the lock only from a human-owned command:

python - <<'PY'
from pathlib import Path
import hashlib, json
corpus = Path("fixtures/corpus")
files = {
    p.relative_to(corpus).as_posix(): hashlib.sha256(p.read_bytes()).hexdigest()
    for p in sorted(corpus.rglob("*")) if p.is_file()
}
Path("fixtures/fixtures.lock.json").write_text(json.dumps(
    {"schema": 1, "files": files}, indent=2, sort_keys=True
) + "\n")
PY

3. Freeze is a lease, not a skip

def load_leases(now: datetime) -> dict[str, dict]:
    raw = json.loads(LEASES.read_text()) if LEASES.exists() else {"leases": []}
    valid = {}
    for lease in raw.get("leases", []):
        required = ("test_id", "corpus_sha256", "expires_at", "reason")
        if any(k not in lease for k in required):
            continue
        if lease.get("on_property") or lease.get("on_fixture"):
            continue
        expires = datetime.fromisoformat(lease["expires_at"].replace("Z", "+00:00"))
        if expires <= now:
            continue
        valid[lease["test_id"]] = lease
    return valid


def freeze_status(test_id: str, corpus_sha: str, now: datetime) -> str:
    leases = load_leases(now)
    if test_id not in leases:
        return "none"
    lease = leases[test_id]
    if lease["corpus_sha256"] != corpus_sha:
        return "no_corpus"
    return "valid"

A skip marker in pytest is not a lease. @pytest.mark.skip has no expiry and no corpus. The predicate must refuse a freeze that does not name a captured input hash. If you cannot replay the input, you do not have a flake. You have an untested branch.

Example lease file:

{
  "leases": [
    {
      "test_id": "tests/nondet/test_clock.py::test_deadline_window",
      "corpus_sha256": "6b3a0c1e9f0d4a77c2b1e8a4d5f60718293a4b5c6d7e8f90123456789abcdef0",
      "expires_at": "2026-09-25T00:00:00Z",
      "reason": "ntp step on shared runner; corpus is the observed wall-clock pair"
    }
  ]
}

Seven days is a policy choice, not a measured optimum. Put the expiry in the file so CI can fail closed when the date passes. Do not extend a lease from the same patch that needs it.

4. The predicate is a pure function

def decide(property_result: str, fixtures: str, freeze: str) -> str:
    if property_result != "pass":
        return "block"
    if fixtures != "match":
        return "block"
    if freeze in {"none", "valid"}:
        return "allow"
    return "block"


def main() -> int:
    # Wire these to your runner. Example assumes files written by CI steps.
    property_result = Path("/tmp/oracle_result.txt").read_text().strip()
    fixtures = fixture_status()
    now = datetime.now(timezone.utc)
    freeze = freeze_status(
        test_id=Path("/tmp/flake_test_id.txt").read_text().strip() or "",
        corpus_sha=Path("/tmp/flake_corpus.sha256").read_text().strip() or "",
        now=now,
    ) if Path("/tmp/flake_test_id.txt").exists() else "none"
    decision = decide(property_result, fixtures, freeze)
    print(
        json.dumps(
            {
                "properties": property_result,
                "fixtures": fixtures,
                "freeze": freeze,
                "agent_tests": "weight=0",
                "merge": decision,
            },
            indent=2,
        )
    )
    return 0 if decision == "allow" else 1


if __name__ == "__main__":
    raise SystemExit(main())

Run order in CI is fixed:

python -m pytest oracles/ -q --maxfail=1
echo pass > /tmp/oracle_result.txt   # only if pytest exited 0
python qa/merge_predicate.py

If oracles/ fails, do not run freeze logic. Short-circuiting is part of the table, not an optimization.

Numbered workflow

1. Freeze the oracle tree. Open oracles/ only through human review. Reject any agent diff that touches it, the lock file, or qa/.
2. Classify tests once. qa/classify.yaml lists ids that may request a freeze. Properties and fixture checks are never on that list.
3. Capture corpora as inputs. Store the bytes that produced the flake, hash them, and refuse a lease without that hash.
4. Run three jobs. Job A: pytest oracles/. Job B: lock verification. Job C: lease validation for the classified set. Merge is the function of A, B, and C. Agent tests are job D, informational.
5. Re-run off the laptop. Same commits, same lock, same leases. If the predicate flips when the machine changes, you are measuring the environment, not the patch.
6. Expire. A scheduler or the predicate itself blocks when expires_at is in the past. Renewal is a separate human commit.

Step 5 is where a second machine earns its keep. Laptop clocks, DNS caches, and GPU drivers leak into flakes. A free remote run does not make the predicate smarter. It makes the environment an independent variable.

Where free model access and a free server fit

Drafting candidate properties is slow if every invariant has to be invented in review. A coding agent can propose properties from public function signatures. A reviewer still promotes them into oracles/. The agent does not commit them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access is enough for that draft step: generate candidate properties, dump them into a review queue, keep them out of the merge vote until a human moves the file. MonkeyCode's free server option is enough for step 5: evaluate qa/merge_predicate.py on a host that did not seed the laptop run. Neither claim is a quota, a model name, or a benchmark. If the product is unavailable in your environment, the predicate still stands. Run job A/B/C on any other isolated runner.

The product is a convenience for the draft-and-rerun loop. It is not a voter.

What this does not solve

Properties that restate the implementation are still possible. An oracle that asserts accrue(...) == accrue(...) is a tautology with better housing. Review the property, not just the path. Fixture hashes do not detect missing cases. A lock can be perfect and the corpus still omit the leap-day input. Freeze leases do not convert flakiness into correctness. They bound how long you will tolerate a captured non-determinism.

The predicate also fails closed on oracle timeouts. That is deliberate. An agent that livelocks the property job should not merge because job D was green.

Do not use this approach if your codebase has no stable public surface to hang properties on, if agents are allowed to edit CODEOWNERS, or if “flake” in your team means “we never captured the input.” In those shops the table will block almost every patch, or reviewers will start stamping leases without corpora. Both outcomes are worse than a normal suite.

Use it when you already distrust tests that arrived in the same diff as the fix, and you can name three things the patch must not break: an invariant, an input corpus, and a clock you refuse to skip forever.

Agent tests can still be useful as diagnostics. Log them. Quote them in the PR. Give them zero weight.