Agent Patches Don't Need the Whole Suite: A Skip-Witness Gate With Patch-Keyed Shadow Sampling

A three-file agent patch should not cost a full-suite run. But a skipped test is a test you can no longer see, and "run only what's affected" turns an invisible regression into a green check. The compromise that has survived contact with real agent patches in my workflow: skip a test only if it carries a complete witness, and audit the skip set with a sample that is a deterministic function of the patch. Same patch, same audited tests, every time.

This post is the skip-side counterpart to the fixture and freeze gates I've written about before. It assumes you already keep test inputs pinned; it does not re-litigate that.

Why import-graph test selection breaks specifically on agent patches

Classic test impact analysis was built for humans who touch source files. Agents touch the things TIA doesn't model:

1. Configuration and fixtures. A patch that edits conftest.py, a YAML fixture, or a JSON snapshot changes the meaning of tests it never imports.
2. Ambient inputs. Environment variables, TZ, locale, hash seed, and freeze dates can be set by a patch and silently change outcomes across the whole suite.
3. Import closure drift. Agents add imports lazily, inside functions, or behind importlib. Your recorded graph from last week is wrong, and it's wrong in the direction that skips more tests.

So the witness has to record more than "which modules does this file import." It has to record the state the test was observed under, and it has to be allowed to say "I don't know" — which becomes an escalation, not a skip.

Step 1 — Record a witness per test during a normal full run

The witness is a byproduct of the full suite you already run on main. A small pytest plugin writes it out. The code below is a sketch, not a released package — treat the heuristics as the point, not the implementation.

# witness_plugin.py — sketch, pytest 7/8, Python 3.11+
import hashlib, json, os, pathlib, sys
import pytest

OUT = pathlib.Path(os.environ.get("WITNESS_OUT", ".witness/witness.json"))
RECORDS = {}
SUSPICIOUS = {"subprocess", "socket", "requests", "urllib3", "http"}
AMBIENT_KEYS = ("TZ", "LANG", "LC_ALL", "PYTHONHASHSEED", "SOURCE_DATE_EPOCH")

def file_fp(paths):
    h = hashlib.sha256()
    for p in sorted(set(paths)):
        try:
            h.update(pathlib.Path(p).read_bytes())
        except OSError:
            h.update(b"<missing>")
    return h.hexdigest()[:16]

def ambient_fp():
    payload = json.dumps({k: os.environ.get(k) for k in AMBIENT_KEYS}, sort_keys=True)
    return hashlib.sha256(payload.encode()).hexdigest()[:16]

def fixture_files(item):
    roots = [pathlib.Path(d) for d in os.environ.get("FIXTURE_DATA_DIRS", "").split(":") if d]
    out = []
    for parent in item.path.parents:
        conf = parent / "conftest.py"
        if conf.exists():
            out.append(str(conf))
    for root in roots:
        out += [str(p) for p in root.rglob("*") if p.is_file()]
    return out

def mod_files():
    return {getattr(m, "__file__", None) for m in list(sys.modules.values())}

@pytest.hookimpl(tryfirst=True)
def pytest_runtest_setup(item):
    item._witness_before = mod_files()

@pytest.hookimpl(trylast=True)
def pytest_runtest_teardown(item, nextitem):
    after = mod_files()
    touched = sorted(p for p in (after - item._witness_before) if p and "site-packages" not in p)
    suspicious = sorted(m for m in sys.modules if m.split(".")[0] in SUSPICIOUS)
    RECORDS[item.nodeid] = {
        "test_file": str(item.path),
        "imports": touched,
        "fixtures": list(item.fixturenames),
        "fixture_fp": file_fp(fixture_files(item)),
        "ambient_fp": ambient_fp(),
        "complete": not suspicious,
        "suspicious": suspicious,
    }

def pytest_sessionfinish(session, exitstatus):
    OUT.parent.mkdir(parents=True, exist_ok=True)
    OUT.write_text(json.dumps({"base": os.environ.get("GIT_SHA", ""), "tests": RECORDS}, indent=2))

Run it once on the branch tip:

GIT_SHA=$(git rev-parse HEAD) FIXTURE_DATA_DIRS=tests/data pytest -p witness_plugin -q

The complete flag is deliberately conservative. Any test that imported a subprocess, socket, or HTTP client is marked incomplete and can never be skipped later. That single rule costs you a handful of always-run tests and removes an entire class of "the agent added a network call and the skip hid it" bugs.

Step 2 — Classify each test three ways, not two

The gate reads the witness, the patch diff, and recomputes the fixture and ambient fingerprints at the patched commit. Comparing recorded against recomputed is what catches fixture drift; comparing file paths alone never will.

Condition at patched commit	Verdict	Why
Witness complete: false	ESCALATE (run)	Unknown dependency surface
Test file or any witnessed import in the diff	RUN	Direct impact
fixture_fp differs from witness	RUN	Test inputs moved
ambient_fp differs from witness	RUN (whole suite)	Patch changed the environment, scope unknown
Test in the freeze set	RUN, not shadow-sampled	Its failure is not attributable
Witness fresh, nothing above fires	SKIP-WITNESSED	Auditable skip

# gate.py — sketch
import hashlib, json, subprocess, sys, pathlib

WITNESS = json.loads(pathlib.Path(".witness/witness.json").read_text())["tests"]

def diff(base):
    out = subprocess.run(["git", "diff", "--name-only", f"{base}...HEAD"],
                         capture_output=True, text=True, check=True).stdout
    return {l for l in out.splitlines() if l}

def classify(changed, recompute_fp, freeze):
    plan, escalate = {}, False
    for nodeid, rec in WITNESS.items():
        if not rec["complete"]:
            plan[nodeid] = "escalate:incomplete-witness"
        elif nodeid in freeze:
            plan[nodeid] = "run:frozen-flaky"
        elif rec["test_file"] in changed or set(rec["imports"]) & changed:
            plan[nodeid] = "run:impacted"
        elif rec["fixture_fp"] != recompute_fp(rec):
            plan[nodeid] = "run:fixture-drift"
        elif rec["ambient_fp"] != recompute_fp("ambient"):
            plan[nodeid] = "run:ambient-drift"
            escalate = True
        else:
            plan[nodeid] = "skip:witnessed"
    return plan, escalate

Ambient drift is the one condition that promotes the whole suite rather than a subset. If a patch changed TZ or PYTHONHASHSEED, every witness in the file was recorded under a different world, and scoping the rerun to "impacted" tests is guesswork.

Step 3 — Audit the skips with a patch-keyed shadow sample

Random sampling of the skip set is close to useless here, because two runs of the same patch audit different tests and you can't separate "this patch is bad" from "we sampled differently." Key the sample to the patch hash instead:

def patch_hash(base="origin/main"):
    d = subprocess.run(["git", "diff", f"{base}...HEAD"],
                       capture_output=True, check=True).stdout
    return hashlib.sha256(d).hexdigest()

def shadow(skipped, phash, pct=10):
    picked = []
    for nodeid in sorted(skipped):
        n = int.from_bytes(hashlib.sha256(f"{phash}:{nodeid}".encode()).digest()[:4], "big")
        if n % 100 < pct:
            picked.append(nodeid)
    return picked

Add one hard rule: frozen flaky tests are excluded from the shadow sample. A flake in the sample produces a false "the skip was hiding a regression" signal, and after two of those the team disables the sample and the gate quietly becomes "skip whatever the graph says." Frozen tests still run in the normal path — they just don't get to testify about anyone else's skip.

Output a single JSON summary per patch (skipped, shadowed, escalated, ambient_drift) and attach it to the PR. The audit trail is the product; the CI time saved is the side effect.

Step 4 — Give freezes a thaw drill instead of a renewal habit

A freeze that auto-renews is a permanent skip with extra steps. Each frozen test gets three fields: reason, frozen_at_sha, and drill (the exact command that would prove it stable, with ambient inputs pinned). The drill is a repeated run under your own chosen iteration count — pick K from observed flake rate, not from a blog post, and record K in the freeze file so the bar can't drift.

Outcome handling:

• K/K passes with pinned ambient inputs → downgrade to an observed-stable lease that still re-runs weekly. It is not unfrozen; it is on probation with a shorter leash.
• Any failure → freeze renews, and the renewal must name the causing ambient input if known. "Still flaky" is not a reason.
• No drill run before expiry → the freeze expires and the test rejoins the shadow sample. Silence should not buy stability.

Where MonkeyCode fits in this workflow

Two parts of this are compute- or judgment-heavy, and both are optional:

• The ESCALATE bucket is a list of tests whose dependency surface couldn't be computed. Triage means reading each test and proposing the missing witness annotations (which data files, which env vars, which markers) as a reviewable diff. I use MonkeyCode's free model access for that first pass and accept nothing without reading it — the model proposes annotations, the human owns them.
• The thaw drill and the shadow sample are repeated runs that have no business competing with mainline CI minutes. Running them on MonkeyCode's free server option keeps the experiment off the shared runner queue while the drill is still being tuned.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Neither is load-bearing. If you replace both with a cron job on your own infra, the method is unchanged.

Limitations, and who should not do this

• Small suites. If the full run is under a couple of minutes, the witness plugin, the gate, and the freeze file cost more attention than they save. Just run everything.
• Heavy dynamic dependency surfaces. Tests that shell out, load plugins by entry point, or read zipped resources will mostly land in ESCALATE. If that bucket is most of your suite, the gate is reporting your architecture, not your patch.
• Test-order coupling. The witness is recorded on an order that may not match the skip-path order. Suites with shared mutable global state can pass witnessed and fail skipped for reasons the witness never recorded.
• No freeze owner. Without one person accountable for expiry, the freeze file becomes a graveyard and the shadow sample becomes noise.

What to measure before you trust it

Run the gate in shadow mode for ten patches: execute the full suite as usual, and also record what the gate would have skipped, escalated, and sampled. Two numbers tell you whether it's working — the escalate ratio (if it's high, fix the witness before widening the skips) and the shadow-sample failure count (any nonzero value is a witness bug, not a patch bug). Publish both in the PR summary so the gate's own reliability is visible, not assumed.

If you run the drill, the number I'd most like to see is the one your own repo produces.