Green Tests Miss Sockets. Gate Agent Patches on a Capability Ledger

Agent patches can pass every unit test and still open a socket, write a cache file, or burn a CI minute in a retry loop. A green suite is not a capability contract. Gate the patch on a recorded ledger: denied network, a hard CPU budget, and a hashed write set.

That gate is independent of assertion style. Properties, examples, and snapshots can all stay green while the process does extra work. The ledger watches the process, not the return value.

The suite does not see side effects

A typical agent patch edits src/ and tests/ in one diff. Coverage rises. CI goes green. None of those signals answer three questions that matter at merge time.

1. Did the process connect anywhere?
2. Did it finish inside a CPU budget?
3. Which files did it create or truncate?

Machine-written fixes often add I/O. Download a wheel. Write a golden file. Retry until the clock moves. The test name is stable. The process is not.

Three columns, one committed file

Store the contract as tests/capability.ledger.json. Keep it in review, next to the patch. Do not keep it in a prompt.

Column	Recorded value	Merge rule
net	host:port connect attempts	empty, or each pair listed in allow_net
cpu_sec	user CPU seconds (RUSAGE_SELF)	<= cpu_budget
writes	SHA-256 of repo-relative write paths	must equal the committed digest

The file is the oracle. A new write path changes the digest. A human has to accept the new hash. That is the review surface.

Proposed harness (unexecuted)

The code below is a proposal. It uses Python audit hooks from PEP 578, which fire in-process. It is not a sandbox. C extensions and child processes can bypass it. Treat it as a merge filter for ordinary Python agent patches.

# capability_harness.py
"""Record connects, CPU budget, and a repo-relative write-set digest."""
from __future__ import annotations

import hashlib
import json
import os
import resource
import sys
from pathlib import Path

LEDGER = Path("tests/capability.ledger.json")
ACTUAL = Path("tests/capability.actual.json")
WRITE_FLAGS = os.O_WRONLY | os.O_RDWR | os.O_CREAT | os.O_TRUNC | os.O_APPEND


def repo_rel(path: str) -> str:
    root = Path.cwd().resolve()
    try:
        return str(Path(path).resolve().relative_to(root)).replace("\\", "/")
    except ValueError:
        return os.path.abspath(path)


def is_write(mode: object, flags: object) -> bool:
    if isinstance(mode, str):
        return any(ch in mode for ch in "wax")
    if isinstance(flags, int):
        return bool(flags & WRITE_FLAGS)
    return False


class Ledger:
    def __init__(self, budget: int) -> None:
        self.budget = budget
        self.connects: list[str] = []
        self.writes: set[str] = set()

    def hook(self, event: str, args: tuple) -> None:
        if event == "socket.connect":
            address = args[1]
            host, port = address[0], address[1]
            self.connects.append(f"{host}:{port}")
        elif event == "open":
            path, mode, flags = args[0], args[1], args[2]
            if is_write(mode, flags):
                self.writes.add(repo_rel(str(path)))

    def snapshot(self) -> dict:
        used = resource.getrusage(resource.RUSAGE_SELF).ru_utime
        blob = "\n".join(sorted(self.writes)).encode()
        return {
            "net": sorted(set(self.connects)),
            "cpu_sec": int(used),
            "cpu_budget": self.budget,
            "writes": hashlib.sha256(blob).hexdigest(),
            "write_paths": sorted(self.writes),
        }


def enforce(actual: dict, expected: dict) -> None:
    errors: list[str] = []
    allow = set(expected.get("allow_net", []))
    extra = [c for c in actual["net"] if c not in allow]
    if extra:
        errors.append(f"unexpected connects: {extra}")
    if actual["cpu_sec"] > expected["cpu_budget"]:
        errors.append(
            f"cpu {actual['cpu_sec']}s > budget {expected['cpu_budget']}s"
        )
    if actual["writes"] != expected["writes"]:
        errors.append(
            f"write digest {actual['writes']} != {expected['writes']}"
        )
        errors.append("paths: " + ", ".join(actual["write_paths"]) or "(none)")
    if errors:
        raise SystemExit("capability ledger mismatch:\n- " + "\n- ".join(errors))


def main(argv: list[str]) -> None:
    expected = json.loads(LEDGER.read_text())
    budget = int(expected["cpu_budget"])
    resource.setrlimit(resource.RLIMIT_CPU, (budget + 5, budget + 5))
    ledger = Ledger(budget)
    sys.addaudithook(ledger.hook)
    import pytest

    code = pytest.main(argv)
    actual = ledger.snapshot()
    ACTUAL.write_text(json.dumps(actual, indent=2) + "\n")
    if code != 0:
        raise SystemExit(code)
    enforce(actual, expected)


if __name__ == "__main__":
    main(sys.argv[1:])

Wrap pytest. Do not register this as a plugin the agent can drop from pytest.ini.

python capability_harness.py -q tests/

Seed a ledger on a known-good revision. The empty digest is SHA-256 of an empty path list.

python - <<'PY'
import hashlib, json
from pathlib import Path
empty = hashlib.sha256(b"").hexdigest()
Path("tests/capability.ledger.json").write_text(json.dumps({
    "allow_net": [],
    "cpu_budget": 30,
    "writes": empty,
}, indent=2) + "\n")
PY

After a legitimate golden-file update, copy writes from tests/capability.actual.json into the ledger. Commit both files in the same review. Do not let the job rewrite the ledger.

Numbered merge workflow

1. Protect capability_harness.py and tests/capability.ledger.json with CODEOWNERS. The agent may edit product code. It may not own the ledger.
2. Let the agent produce a patch. A free model endpoint is enough when the change is local and the oracle is the ledger, not the model text.
3. Apply the patch on an isolated runner. A free server option is enough when the suite is small and carries no production secrets.
4. Cut network at the job layer as a second belt. Audit hooks do not see a child curl.
5. Run python capability_harness.py -q tests/. If tests pass and the ledger mismatches, fail the job.
6. If the write set is intentional, a reviewer copies the digest and lists the new paths in the PR body.

Linux job-level deny:

unshare -n python capability_harness.py -q tests/

unshare -n is Linux-only. On a container runner, set the job network to none. The ledger still catches in-process urllib. The namespace catches helper binaries.

Protect the files:

# CODEOWNERS
/capability_harness.py              @maintainers
/tests/capability.ledger.json       @maintainers

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option map to two different columns of this workflow. The model proposes the diff. The server runs the harness. Mixing those roles is the usual failure: the same process that writes tests/ also decides whether the write was allowed.

Keep the model away from capability.ledger.json. Keep secrets off the free server. The ledger is a small JSON file. It does not need a GPU. It needs a process that cannot reach the public internet while pytest runs.

The split stays useful if product names change. The contract is the ledger file, not the vendor.

Classify the job before retrying

Do not retry first. Retries turn a capability leak into noise.

Observation	Likely cause	Action
tests fail, ledger unused	functional regression	reject the patch
tests pass, extra host:port	hidden download or telemetry	reject; do not add a host under time pressure
tests pass, CPU over budget	retry loop or accidental quadratic work	reject; shrink inputs only if the test still kills the bug
tests pass, write digest drift	new cache, golden, or .pyc	inspect write_paths; accept only durable fixtures
process dies on SIGXCPU	unbounded loop	treat as fail, not as an unstable job
tests pass, ledger matches	merge candidate	review the diff as usual

RLIMIT_CPU is coarse. It counts seconds, not assertions. Set cpu_budget high enough to include pytest startup, then stop raising it when a patch gets slower. A timeout is a failed contract.

Polyglot repos can add an outer trace. This command is a sketch, not a parser.

strace -f -e trace=connect,openat -o /tmp/cap.trace -- \
  python capability_harness.py -q tests/
rg -n "connect\(" /tmp/cap.trace | head

Use strace as evidence when a C extension bypasses the audit hook. Do not replace the ledger with a raw trace file. Traces are not stable across kernel versions.

Limitations

Audit hooks are in-process and Python-specific. They will not see a compiled extension that calls connect(2) without going through socket. They will not see a helper binary. RLIMIT_CPU sends SIGXCPU and behaves differently on macOS and inside some containers. Pytest itself consumes CPU, so the budget is not a per-test number.

The write-set digest is path-sensitive. Hashing absolute paths will break when CI uses a different checkout directory. Normalize to repo-relative paths before hashing, or the ledger becomes an unstable job. That bug sits in the harness, not in the agent.

Opening a file with + for read/update can land in the write set if flags include O_RDWR. The table above is conservative. Over-reporting writes is safer than missing a truncate. Under-reporting is a silent merge.

Do not use this approach when:

• the suite must talk to a real database, queue, or license server
• the patch is not Python and no equivalent trace exists
• no human owns allow_net and the digest
• the goal is malware isolation; this is a merge filter, not a sandbox

A capability ledger also does not replace property tests. It answers a different question: what the process was allowed to touch while those tests ran.

What to commit

Commit four files for the gate. Nothing else is required.

• capability_harness.py
• tests/capability.ledger.json
• CODEOWNERS entries for both
• the CI step that wraps pytest and disables network

Leave model prompts out of the repository. If the write digest changes without a reviewed path list, the job stays red.

If the runner is already isolated, commit the ledger file before asking the model for another patch. The cheaper control is a smaller allowlist, not a longer prompt.