Inventory the Test Surface Before You Score an Agent Patch

A green pytest exit code on the pull request tree is not a score. If an agent can edit tests, fixtures, markers, or skip lists, that exit code measures agreement with the patch's own story. Score against a detached witness pack. Count assertions. Keep flakes in a ledger the agent cannot rewrite.

The rest of this article is a concrete inventory protocol. It is not a model review. It does not claim that any gate is complete. It does claim that you can fail a patch for shrinking the test surface before you even discuss the production diff.

What a green run can hide

Agent patches fail in boring ways. They delete a failing node. They wrap an assertion in pytest.mark.skip. They shorten a property campaign from 200 examples to 5. They replace a fixture file with an empty list. They cut an API latency check from p99 to a single warm request.

None of those require a clever exploit. They require write access to the files your CI treats as oracles. A job that runs pytest inside the PR tree cannot see the difference between a fixed bug and a quieter suite.

You need three inventories, not one exit code.

1. Node inventory. Collected test ids on parent versus patch.
2. Assertion inventory. AST-level assert counts, property example budgets, and performance sample sizes.
3. Oracle inventory. Fixture bytes, contract schemas, and a flake ledger that lives outside the agent's write set.

Step 1: Build a witness pack the PR cannot author

Keep a signed artifact that is not checked out into the workspace the agent may edit. The pack is the score surface. The PR tree is the candidate.

A minimal layout looks like this:

witness-pack/
  MANIFEST.sha256
  tests/                    # read-only copies used for scoring
  fixtures/
  contracts/
    http_v1.json
  properties/
    campaign.toml
  flakes/
    ledger.json

campaign.toml is a budget, not a slogan. Pin example counts and deadlines here so a patch cannot “pass properties” by lowering max_examples.

# properties/campaign.toml
[http_handler]
max_examples = 200
deadline_ms = 4000
seed_source = "pack"   # never read from the PR

[parse_money]
max_examples = 400
deadline_ms = 6000
seed_source = "pack"

Hash every file into MANIFEST.sha256. Store the pack on a ref or object store the scoring job fetches by digest. If the digest does not match, fail closed. Do not fall back to tests/ from the PR.

Fixtures belong in the pack as bytes, not as “please don’t touch tests/data” comments. An agent that rewrites conftest.py to point at a smaller JSON file will still go green on the PR tree. It will not go green if scoring mounts the pack’s fixtures by absolute path.

Step 2: Inventory nodes and assertions

Collect tests twice. Once on the parent commit with the witness pack. Once on the patch, still with the same pack. Then compare the PR tree itself, because that is where silence is authored.

Label the following as a local protocol, not as a published benchmark. Run it on your own repo before you trust the numbers.

# inventory.py — proposal for a CI helper, not a shipped product
from __future__ import annotations

import ast
import json
import subprocess
from pathlib import Path

SKIP_MARKERS = {"skip", "skipif", "xfail"}


def collect_node_ids(cwd: Path) -> list[str]:
    out = subprocess.check_output(
        ["pytest", "--collect-only", "-q", "--noconftest"],
        cwd=cwd,
        text=True,
    )
    return sorted(
        line.strip() for line in out.splitlines()
        if line.strip() and not line.startswith("=")
    )


class SurfaceVisitor(ast.NodeVisitor):
    def __init__(self) -> None:
        self.asserts = 0
        self.markers = 0
        self.max_examples = []

    def visit_Assert(self, node: ast.Assert) -> None:
        self.asserts += 1
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call) -> None:
        func = node.func
        name = getattr(func, "attr", None) or getattr(func, "id", None)
        if name in SKIP_MARKERS:
            self.markers += 1
        if name == "given" or name == "settings":
            for kw in node.keywords:
                if kw.arg == "max_examples" and isinstance(kw.value, ast.Constant):
                    self.max_examples.append(kw.value.value)
        self.generic_visit(node)


def scan_tree(root: Path) -> dict:
    asserts = markers = 0
    examples: list[int] = []
    files = 0
    for path in root.rglob("*.py"):
        if "test" not in path.name and "tests" not in path.parts:
            continue
        tree = ast.parse(path.read_text(encoding="utf-8"))
        v = SurfaceVisitor()
        v.visit(tree)
        asserts += v.asserts
        markers += v.markers
        examples.extend(v.max_examples)
        files += 1
    return {
        "files": files,
        "asserts": asserts,
        "skip_markers": markers,
        "min_max_examples": min(examples) if examples else None,
    }


def main() -> None:
    parent = scan_tree(Path("parent/tests"))
    patch = scan_tree(Path("pr/tests"))
    report = {
        "parent": parent,
        "patch": patch,
        "assert_delta": patch["asserts"] - parent["asserts"],
        "marker_delta": patch["skip_markers"] - parent["skip_markers"],
    }
    print(json.dumps(report, indent=2))
    if report["assert_delta"] < 0 or report["marker_delta"] > 0:
        raise SystemExit("silence detected on the PR test surface")


if __name__ == "__main__":
    main()

Fail the gate on any of these, unless a human-reviewed waiver file in the witness pack names the exact node id:

• Node id deleted or renamed without a ledger entry.
• assert count down.
• skip / skipif / xfail count up.
• max_examples or performance sample size down.
• pytest.ini, conftest.py, or fixture hashes changed in the PR.

A rename that preserves the assertion body is still a surface change. Treat it as a failure until the witness pack is updated in a separate change. Agents are good at renaming a failing test into a passing one.

Step 3: Put flakes in a ledger, not in markers

A freeze on flaky tests does not mean pytest.mark.skip. Skip markers are editable. A ledger is data the scoring runner consults, and the agent’s tree is not allowed to ship a new skip.

{
  "version": 1,
  "entries": [
    {
      "node_id": "tests/test_parse.py::test_money_roundtrip",
      "first_seen": "2026-09-18",
      "expires": "2026-10-02",
      "signature": "AssertionError: cents != 0",
      "owner": "payments",
      "allowed_action": "quarantine"
    }
  ]
}

Rules for the runner:

1. Load flakes/ledger.json from the witness pack only.
2. Quarantine means “do not score this node,” not “rewrite the test.”
3. If expires is in the past, the node returns to the score set. A still-flaky node fails the job. It does not extend itself.
4. If the PR adds skip or xfail to a node that is not in the ledger, fail.
5. If the PR edits the ledger, fail. Ledger edits are a human PR against the pack ref.

The signature field is the failure shape, not the test name. Names move. Tracebacks that match a known flake can be quarantined. Tracebacks that do not match are new failures. New failures score the patch.

Step 4: Score off-tree in two jobs

Do not generate and score in the same checkout. Split the pipeline.

# proposal: ci-score.yml — scoring job only
name: score-agent-patch
on:
  pull_request:
jobs:
  score:
    runs-on: ubuntu-latest
    steps:
      - name: Fetch candidate source only
        run: git fetch origin "$PR_SHA" && git checkout "$PR_SHA" -- src
      - name: Fetch witness pack by digest
        run: |
          aws s3 cp "s3://oracles/witness-pack-${PACK_DIGEST}.tar" pack.tar
          echo "$PACK_SHA256  pack.tar" | sha256sum -c -
          tar -xf pack.tar
      - name: Inventory PR test surface
        run: python inventory.py
      - name: Replay pack tests against candidate src
        run: |
          PYTHONPATH=src pytest pack/tests \
            --override-ini="cache_dir=/tmp/pytest-cache" \
            -o "pythonpath=src"
      - name: Property campaign from pack budget
        run: python pack/run_campaign.py --config pack/properties/campaign.toml

The candidate may change src/. It may not change the files under pack/. If your agent runner mounts the full repo, it will treat tests as editable source. That is the bug. The scoring job should not even see the PR’s tests/ directory except as input to inventory.py.

Property checks then mean: run the pack’s generators against the candidate’s public functions, with the pack’s example budget. Fixtures mean: bytes from the pack, mounted read-only. Flakes mean: ledger lookup, not a marker the model can type.

Where free model access and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The protocol does not depend on a vendor. It depends on isolation. One practical split is: propose the patch with free model access in a workspace that contains src/ and a short task file, then run inventory plus pack replay on a free server that only mounts the witness pack and the candidate src/.

That is the whole product-relevant claim here. Free model access is for generation. The free server option is for a scoring job that never hands the oracle to the agent as writable files. Do not treat either as a durability guarantee, a quota, or a hardware spec. If the server can see tests/ from the PR, you are back to scoring the agent’s story.

Decision table

Observation	Score	Next action
Pack digest mismatch	fail	Stop. Do not run pytest.
assert count down on PR tests	fail	Treat as silence, not as cleanup.
New skip/xfail not in ledger	fail	Human ledger PR, or restore the test.
Node quarantined, signature matches, not expired	omit node	Score the rest of the pack.
Quarantine expired, node still flaky	fail	Fix the test or the code. Do not auto-extend.
max_examples or latency samples reduced	fail	Restore the pack budget.
Pack tests fail on candidate src/	fail	Real regression.
Pack tests pass, inventory clean	pass inventory	Review the production diff as usual.

Pass inventory is not pass merge. It is permission to read the patch.

Limitations

This protocol does not detect a tautology that keeps the same assert count. An agent can replace assert result == expected with assert True or result == expected and survive a naive AST count. Strengthen the visitor to reject Assert nodes whose test is a constant true, and to hash assertion ASTs without names. Even then, equivalent rewrites exist.

It does not replace mutation testing. A patch can keep every node and still implement the wrong function if the pack never called that function. Expand the pack with contracts and properties that hit the public surface you care about. Do not confuse “the inventory was stable” with “the behavior is correct.”

It is weak on non-hermetic tests. If fixtures call the network, the ledger will fill with environment noise and then expire into false failures. Network, clock, and RNG belong behind pack-controlled fakes. If you cannot hermeticize a test, it should not be in the score set.

Performance tests are a special silence target. Watch duration, sample size, and percentile assertions as first-class inventory fields. A patch that drops a 30-second p99 check to a single request will look green and still ship a regression. Count those numbers the same way you count assert.

Who should not use this

Do not use this as a substitute for review on security or money paths. An inventory gate is a filter. It is not an audit.

Do not use it if the team cannot maintain a pack ref. A stale pack scores the wrong product. A pack that tracks the PR 1:1 is just the PR tree with extra steps.

Do not let the same bot update src/, the witness pack, and the flake ledger in one change. That collapses the split. Ledger and pack updates are human-authored, or they are not a freeze.

Skip the extra job if you already forbid test edits in CODEOWNERS and enforce it with a required review that actually blocks merge. The inventory still helps, but the first bug to fix is write access, not pytest flags.

The core conclusion does not change. Score the pack. Inventory the surface. Keep flakes as data. If you already split generate from score, run the inventory job where the witness pack never lands in the agent checkout.