Shadow-Compare the Agent Patch. Merge Only Classified Divergences.

A green test run is not a behavior spec. An agent patch can keep every existing assertion passing and still change encodings, error types, empty-input handling, or the bytes written to stdout. Shadow-compare the candidate against a frozen baseline on the same corpus. Merge only after every divergence is classified in an accepted-delta ledger.

This article is a testing workflow, not a model bake-off. The harness below is labeled as a proposed, runnable pattern. It does not claim production timings, model names, or pass rates.

Why green CI misses the patch

Agent patches optimize for the tests they can see. Hidden behavior lives in branches the suite never names: trailing newlines, NaN keys, timezone-naive stamps, None versus []. Those are cheap to alter. They are expensive to notice after merge.

A dual-run gate treats the old artifact as the oracle for unspecified behavior. Specified behavior still belongs in ordinary tests. The ledger exists for the remainder: diffs you accept on purpose, and diffs you refuse.

Do not use this as a substitute for code review. Use it as a filter that review should not have to do by hand.

Artifact: baseline, candidate, ledger

Three files define the contract.

1. baseline/ — a pinned checkout, wheel, or container digest. Not main at HEAD.
2. candidate/ — the agent patch, applied on top of the same pin.
3. delta_ledger.yaml — every previously classified output divergence, keyed by fixture id.

Proposed layout:

shadow/
  corpus/                 # deterministic fixtures only
    001_empty.json
    002_unicode.json
    003_nested_null.json
  delta_ledger.yaml
  canonicalize.py
  shadow_compare.py

The corpus must be I/O-free. No clocks. No DNS. No home-directory probes. If a fixture needs time, inject it. If it needs a filesystem, pass a temp root the harness owns.

Step 1 — Freeze the baseline as an artifact

Record the exact bytes you will rerun. A git SHA is enough when the tree is hermetic. Prefer a built artifact when native extensions or generated code are in play.

git rev-parse HEAD > shadow/BASELINE_SHA
python -m pip wheel . -w shadow/baseline_wheel
sha256sum shadow/baseline_wheel/*.whl | tee shadow/BASELINE_WHEEL.sha256

Label the next command as a local check, not a measured benchmark:

python -c "import yourpkg,inspect; print(inspect.getfile(yourpkg))"

If that path can drift between baseline and candidate, the comparison is invalid. Pin PYTHONPATH. Pin the interpreter.

Step 2 — Make outputs comparable

Raw stdout lies. Timestamps, map iteration order, and exception messages that include object ids will manufacture diffs. Canonicalize before you compare.

Proposed canonicalize.py:

# Proposed helper. Unexecuted against your tree until you wire imports.
from __future__ import annotations

import json
import re
from typing import Any

_HEX_ID = re.compile(r"0x[0-9a-fA-F]+")
_PATH = re.compile(r"(/tmp|/var/folders)[^\s'"]+")


def canon_text(s: str) -> str:
    s = s.replace("\r\n", "\n").rstrip() + "\n"
    s = _HEX_ID.sub("0xID", s)
    s = _PATH.sub("<tmp>", s)
    return s


def canon_json(obj: Any) -> str:
    return json.dumps(obj, sort_keys=True, separators=(",", ":"), ensure_ascii=False)


def canon_exc(exc: BaseException) -> dict[str, str]:
    return {
        "type": type(exc).__qualname__,
        "module": type(exc).__module__,
        "msg": canon_text(str(exc)),
    }

Keep message canonicalization conservative. If you strip too much, a changed error class can hide behind a shared <tmp> token. Prefer comparing exception type first, message second.

Step 3 — Dual-run one fixture

Load both implementations as separate modules, not as two imports of the same name. importlib with distinct package roots is enough for pure Python. For extensions, use two venvs and subprocesses.

Proposed core of shadow_compare.py:

# Proposed dual-run. Treat as a template, not a published benchmark.
from __future__ import annotations

import importlib.util
import json
import traceback
from dataclasses import asdict, dataclass
from pathlib import Path
from typing import Any, Callable

from canonicalize import canon_exc, canon_json, canon_text


@dataclass(frozen=True)
class Run:
    ok: bool
    value: str
    exc: dict[str, str] | None
    trace: str


def load_fn(root: Path, mod: str, name: str) -> Callable[..., Any]:
    path = root / f"{mod.replace('.', '/')}.py"
    spec = importlib.util.spec_from_file_location(f"{root.name}_{mod}", path)
    if spec is None or spec.loader is None:
        raise RuntimeError(f"cannot load {path}")
    module = importlib.util.module_from_spec(spec)
    spec.loader.exec_module(module)
    return getattr(module, name)


def invoke(fn: Callable[..., Any], payload: Any) -> Run:
    try:
        out = fn(payload)
        if isinstance(out, (dict, list)):
            value = canon_json(out)
        else:
            value = canon_text(str(out))
        return Run(ok=True, value=value, exc=None, trace="")
    except BaseException as exc:  # noqa: BLE001 — we want the type, not a reraise
        return Run(
            ok=False,
            value="",
            exc=canon_exc(exc),
            trace=canon_text(traceback.format_exc()),
        )

Subprocess isolation is stricter. Use it when the patch might mutate process state:

python -I shadow/shadow_compare.py \
  --baseline shadow/baseline_src \
  --candidate shadow/candidate_src \
  --corpus shadow/corpus \
  --ledger shadow/delta_ledger.yaml \
  --report shadow/report.json

-I ignores user site-packages. That is the point.

Step 4 — Classify every divergence

A mismatch is not a failure until it is unclassified. The ledger is the only place a behavior change becomes intentional.

Proposed ledger shape:

# delta_ledger.yaml — human-owned. Do not let a model append blindly.
version: 1
entries:
  - id: 003_nested_null.json
    field: value
    baseline: '{"a":null}'
    candidate: '{"a":[]}'
    decision: reject
    reason: empty array is not JSON null; downstream SQL bind changes
  - id: 002_unicode.json
    field: value
    baseline: '{"s":"café"}'
    candidate: '{"s":"cafe\u0301"}'
    decision: accept
    reason: NFC vs NFD; we normalize at the HTTP edge, not here

Gate rule, stated as a predicate:

1. Same ok, same value, same exc.type → pass, no ledger row required.
2. Diff matches an accept row for that fixture id and field → pass.
3. Diff matches a reject row → fail with the stored reason.
4. Diff matches nothing → fail as unclassified. Do not auto-write the file.

def verdict(fid: str, field: str, b: str, c: str, ledger: dict) -> str:
    if b == c:
        return "match"
    for row in ledger.get("entries", []):
        if row["id"] == fid and row["field"] == field:
            if row["baseline"] == b and row["candidate"] == c:
                return row["decision"]  # accept | reject
    return "unclassified"

The important failure is unclassified. A silent accept list that grows without review is how you reintroduce the original problem under a new name.

Decision table

Observation	Likely cause	Merge action
Exception type changes, message similar	Agent swapped ValueError for AssertionError or a helper wrapper	Reject unless callers already catch the new type
JSON null becomes [] or {}	Agent “fixed” empty handling	Reject; pin a fixture named for the empty case
Key order changes after canonicalize	Canonicalizer is wrong, or the type is not JSON	Fix canonicalize; do not ledger it
Unicode combining forms differ	Normalization moved across a layer	Accept only if an outer layer still NFC/NFD-stabilizes
Extra trailing newline on stdout	Print vs return	Reject for libraries; accept for CLIs with a documented formatter
Candidate reads os.environ the baseline did not	Hidden defaulting	Reject; inject config instead
Float 0.1 + 0.2 text changes	Print precision, not math	Canon with repr or decimal text; do not accept blindly

Read the table top to bottom. If two rows could apply, take the stricter one.

Step 5 — Keep the corpus hostile and small

Do not generate a million random blobs on the first pass. Start with a hand-built set that names the assumptions agents invent most often.

Minimum fixture set for a JSON-in, JSON-out transform:

1. Empty object, empty array, JSON null, missing key.
2. Duplicate keys if your parser is not RFC-strict.
3. Mixed NFC/NFD identifiers.
4. Integers near 2**53 if the payload later touches JavaScript.
5. Nested depth 20, then depth 2.
6. A string that looks like a date and a string that does not.

Property generators can come later. They belong behind the same ledger. An unclassified random fail is still a fail. Do not freeze it as flaky. If the generator is non-repeatable, it is not a fixture.

Seeded generators are allowed when the seed is in the report:

python shadow/gen_corpus.py --seed 20260905 --out shadow/corpus

Store the seed next to BASELINE_SHA. A corpus you cannot rebuild is a ledger you cannot audit.

Where a free model and a free server fit

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access and free server option can sit beside this gate, not inside the predicate. The server is a place to run shadow_compare.py against a pinned corpus without competing with interactive work. The model is an optional pre-labeler: given baseline text, candidate text, and the decision table, it may suggest accept or reject with a one-line reason.

The suggestion is not a ledger row. A human still writes the YAML. If the model labels accept on a null→[] change, the harness must still fail until a person records that choice. That split is the whole method.

Do not send production payloads to any model. Redact. Truncate. Prefer synthetic fixtures that already live in shadow/corpus.

Step 6 — Wire the gate in CI without trusting the working tree

Proposed job sketch:

# Proposed CI fragment. Adjust runners to your org.
name: shadow-compare
on: [pull_request]
jobs:
  dual-run:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0
      - name: materialize baseline
        run: |
          git checkout "$(cat shadow/BASELINE_SHA)" -- .
          mkdir -p /tmp/baseline && cp -a . /tmp/baseline
          git checkout -
      - name: run dual compare
        run: |
          python -I shadow/shadow_compare.py \
            --baseline /tmp/baseline \
            --candidate . \
            --corpus shadow/corpus \
            --ledger shadow/delta_ledger.yaml \
            --report shadow/report.json
      - name: fail on unclassified
        run: python -c "import json,sys; r=json.load(open('shadow/report.json')); sys.exit(r['unclassified']!=0)"

Keep the ledger in the same PR as the patch. A patch that needs twelve new accept rows is a patch that changed the product. That is visible in review. A patch that needs zero rows and still compiles is the cheap case. Both should be possible to see in one screen.

Limitations

Shadow-compare only works when the function under test is a function. Hidden threads, live sockets, and clocks break the equality relation. Stub them or exclude those entry points.

Canonicalizers can over-smooth. If you replace every hex id and every path, you may accept a patch that started logging secrets into a new filename. Review the canonicalizer when you review the ledger.

The ledger can rot. An accept row that cites a layer you later removed is a landmine. Re-run the corpus after refactors that move normalization. Delete rows whose baseline no longer reproduces.

Model pre-labels drift with prompt and temperature. They are comments. They are not evidence.

This workflow does not prove correctness. It proves you saw the behavior change.

Who should not use this

Do not adopt the harness if you have no frozen baseline. Comparing main to itself after a rebase is theater.

Do not adopt it for first-version greenfield code with no users and no specified edge behavior. Write ordinary tests first. Dual-run needs a left-hand side.

Do not adopt it as a security boundary. An agent that adds an outbound call can still pass if your fixtures never exercise that branch. Pair with an allowlist of opened files and hosts if that is your threat model; that is a different gate.

Do not let the ledger become a flaky-test freezer. Unstable diffs mean the corpus is impure. Fix the impurity.

Close

Pin a baseline. Run the same corpus twice. Canonicalize. Fail on unclassified diffs. Write accepted deltas in YAML that review can read. That sequence is the test strategy. The model, if you use one, only drafts the sentence that a person still has to sign.