Run the Agent's Tests Against the Parent Tree: A 2 2 Verdict Matrix for Patch Evidence

A green suite on the patched tree is one data point, not a conclusion. Run the same tests a second time against the parent source — with the agent's test files kept in place — and each check stops being a state and becomes a change in verdict. That change is the only part of the run that is about the patch.

Most teams produce the second run by accident: suite on main before merge, suite on the branch after. Here both runs are deliberately the same test set, and their verdicts are joined per test id.

Rule 1 — Build two trees from one patch commit

The unit of work is a commit, not a working directory. Two git worktrees share the object database, so the second tree costs disk, not clone time.

#!/usr/bin/env bash
# .gate/two-trees.sh — reference harness; not executed here. Adapt paths first.
set -euo pipefail

PATCH_REF="${1:?usage: two-trees.sh <patch-commit>}"
PARENT_TREE="${PARENT_TREE:-../gate-parent}"
CHILD_TREE="${CHILD_TREE:-../gate-child}"
TEST_GLOBS="${TEST_GLOBS:-.gate/test-globs}"   # e.g. ^tests/ | _test\.py$ | ^testdata/

rm -rf "$PARENT_TREE" "$CHILD_TREE"
git worktree add -f "$PARENT_TREE" "$PATCH_REF^" >/dev/null
git worktree add -f "$CHILD_TREE"  "$PATCH_REF"  >/dev/null

# Overlay ONLY the test-surface files from the patch onto the parent tree.
git diff --name-only "$PATCH_REF^" "$PATCH_REF" | grep -E -f "$TEST_GLOBS" > /tmp/gate-tests.txt || true
while read -r f; do
  [ -n "$f" ] && git -C "$PARENT_TREE" checkout "$PATCH_REF" -- "$f"
done < /tmp/gate-tests.txt

The overlay is the load-bearing step, and it is easy to get backwards. Without it, a test the agent added exists only on the child tree, never runs on the parent, and can never be falsified. With it, new tests run against old source — which is exactly where they should fail.

Rule 2 — Classify by verdict pair, not by pass

Run the identical command in both trees, with the same lockfile, the same seeds, and the same environment digest. Emit JUnit XML, then join.

# .gate/matrix.py
import xml.etree.ElementTree as ET

def verdicts(path):
    out = {}
    for tc in ET.parse(path).getroot().iter("testcase"):
        tid = f"{tc.get('classname')}::{tc.get('name')}"
        kind = "pass"
        for child in tc:
            if child.tag == "failure": kind = "fail"
            elif child.tag == "error": kind = "error"
        out[tid] = kind
    return out

parent, child = verdicts("gate-parent.xml"), verdicts("gate-child.xml")

parent verdict	patched verdict	class	usable as evidence?
fail	pass	load-bearing	yes, for that behaviour only
pass	pass	vacuous for this patch	no
fail	fail	unfixed	no — block or narrow the claim
pass	fail	regression	no — block
error	pass	coupling-only	no — rewrite as an assertion
not run	pass	invisible	no — the overlay or globs are broken

The row that changes reviews is pass / pass. Those tests are not wrong; they are simply silent about this diff. A suite where every test lands in that row has produced a green light with zero information content, and that is a finding about the change, not a reason to relax the gate.

This is a different question from reachability. Reachability asks which tests touched the changed lines; this matrix asks whether the test's verdict moved. A test can reach new code and still be pass / pass, and that combination is common when the agent narrows an assertion or wraps a call in a helper.

Rule 3 — Errors are not failures

A new test that raises ImportError or AttributeError on the parent tree is not proving a behaviour change; it is proving the patch added a symbol the test imports. Treat that as coupling-only and send it back for an actual assertion. Otherwise the weakest possible test — one that merely calls something new — occupies the strongest evidence row in the table.

Rule 4 — Property checks must falsify on the parent, then shrink

Invariants are where agent patches hide: the code changes shape while the example-based tests keep passing. A property check earns admission to the evidence set on three conditions, in order.

1. It fails on the parent tree for at least one seed in the declared seed set.
2. It passes on the child tree for the entire seed set — not just the failing seed.
3. The counterexample shrinks below a size bound and replays deterministically twice on the child tree with the same digest.

import hashlib, subprocess

def probe(target_dir, seed, invariant_cmd, timeout=120):
    p = subprocess.run([*invariant_cmd, "--seed", str(seed)],
                       cwd=target_dir, capture_output=True, text=True, timeout=timeout)
    payload = p.stdout.strip()
    return p.returncode, payload, hashlib.sha256(payload.encode()).hexdigest()[:16]

Condition 3 is the one people skip, and it is the one that matters. An unshrunk counterexample is a story; a shrunk, replayable one is a regression test you can commit.

Rule 5 — Pin the inputs the patch cannot write

If the agent can edit the corpus, the fixture proves nothing. Keep a manifest under a reviewer-owned path and verify it before either tree runs.

find fixtures/corpus -type f -print0 | sort -z | xargs -0 sha256sum > .gate/inputs.sha256
sha256sum -c .gate/inputs.sha256 || { echo "input drift"; exit 1; }
git diff --name-only "$PATCH_REF^" "$PATCH_REF" | grep -q '^\.gate/' \
  && { echo "patch touched the gate's own inputs"; exit 1; }

The second command is not paranoia. .gate/test-globs decides which files count as test surface; a patch that edits that file can move its own weak assertions out of the evidence set while leaving every verdict green.

Rule 6 — Freezes expire on a wall clock

Flaky tests corrupt the matrix because a flip between trees is indistinguishable from a change in behaviour. Quarantine them explicitly, with an expiry date CI can enforce.

# .gate/freezes.yaml
- test: "tests/http/test_retry.py::test_backoff_jitter"
  reason: "upstream clock skew"
  quarantined_at: 2026-10-16
  expires_at: 2026-11-15
  owner: "@finley"

A freeze only removes a test from the evidence set; it never converts a missing verdict into evidence. An expired freeze fails the gate on its own, before any test runs, so the freeze cannot outlive the investigation that justified it.

Where the extra runs go

The method doubles test minutes, and the parent-tree leg produces no product value on its own — so it belongs on disposable capacity rather than on the merge-blocking runner. MonkeyCode's free server option is one place to host that second tree, and its free model access is usable for drafting property candidates and proposing smaller counterexamples — both of which are generated then verified by deterministic replay, never accepted as evidence on their own.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I am not going to quote quotas, model names, hardware, or retention for the free tier, because I cannot verify any of them from the operator's brief. Treat availability as a variable: the gate above must remain runnable if the free capacity disappears tomorrow, at the cost of running the parent leg on your normal CI.

When the layers disagree

signal	property check	fixture manifest	verdict matrix	action
clean	load-bearing	verified	one or more rows 1	proceed to review
silent change	no counterexample	verified	all pass / pass	require a stated non-behavioural purpose, route to manual review
drifted inputs	load-bearing	mismatch	mixed	block — the evidence came from mutated inputs
masked regression	any	verified	any pass / fail	block — freezes never cover a regression
broken harness	any	verified	any not run	block — fix globs/overlay before reading results

Limitations and who should not use this

• Short suites. If the whole suite runs in under a minute, run both trees unconditionally and skip the selection logic. The matrix is for suites where 2× is a real budget decision.
• No per-test output. Without JUnit XML or equivalent, you cannot join verdicts by test id, and the matrix collapses into a single pass/fail bit.
• Agents that can edit the gate. If .gate/test-globs is not protected by reviewers, the test/source split is untrustworthy and every number below it is decorative.
• Long-lived freezes. A freeze count that only grows means your evidence set is shrinking while your confidence is rising. That inversion is the failure mode this whole pipeline exists to prevent.

Run the second tree once and the pass / pass row stops being invisible. It is the cheapest piece of missing evidence in most agent-patch reviews — and it does not require buying anything.