If the Patch Authored the Test, Score the Overlap

Same-session tests are not evidence. If an agent writes a production hunk and the assertions in one pass, a green build often means the two files agreed on a private story, not that the public contract holds. Score the overlap first. Then rewrite the oracle in a session that never sees the patch.

This is not a flake policy and not a fixture lock. Those gates ask whether a test is stable. This gate asks whether the test is independent. A stable, overlapping oracle is worse than a noisy one. It fails closed on the wrong question.

The failure mode

Agent patches fail in a repeatable way. The model emits src/fee.py and tests/test_fee.py together. Shared literals appear in both: the same timeout, the same error substring, the same sentinel UUID. The test does not probe the API. It echoes the implementation.

A second pattern is quieter. The production function grows an extra branch that exists only to satisfy a comment in the test. Coverage rises. Behavior for real callers does not. Token overlap catches the first pattern. Literal mutation plus an isolated oracle session catches the second.

Treat both as merge smells. Do not treat a passing pytest run as a property.

Artifact: overlap score plus literal mutation

The artifact below is a proposed local gate. It is not a published benchmark and it has not been run against a private corpus here. Label it as a method, not a result.

# overlap_gate.py — proposed pre-merge check, not a scored study
from __future__ import annotations

import ast
import json
import re
import sys
from pathlib import Path

IDENT = re.compile(r"[A-Za-z_][A-Za-z0-9_]{2,}")
STRING = re.compile(r"['\"]([^'\"]{3,})['\"]")
NUMBER = re.compile(r"\b\d{2,}\b")

STOP = {
    "self", "true", "false", "none", "return", "assert", "test",
    "def", "class", "import", "from", "with", "for", "not",
}


def tokens(text: str) -> set[str]:
    found = set()
    for rx in (IDENT, STRING, NUMBER):
        for m in rx.findall(text):
            t = m.lower() if isinstance(m, str) else m
            if t not in STOP:
                found.add(t)
    return found


def literals_from_ast(src: str) -> set[str]:
    out: set[str] = set()
    try:
        tree = ast.parse(src)
    except SyntaxError:
        return out
    for node in ast.walk(tree):
        if isinstance(node, ast.Constant) and isinstance(node.value, (str, int, float)):
            out.add(repr(node.value))
    return out


def overlap_report(prod: str, test: str) -> dict:
    pt, tt = tokens(prod), tokens(test)
    pl, tl = literals_from_ast(prod), literals_from_ast(test)
    token_inter = pt & tt
    lit_inter = pl & tl
    token_score = len(token_inter) / max(1, len(tt))
    lit_score = len(lit_inter) / max(1, len(tl))
    return {
        "token_overlap": round(token_score, 3),
        "literal_overlap": round(lit_score, 3),
        "shared_tokens": sorted(token_inter)[:40],
        "shared_literals": sorted(lit_inter)[:40],
        "contaminated": token_score >= 0.35 or lit_score >= 0.25,
    }


def main() -> int:
    prod = Path(sys.argv[1]).read_text(encoding="utf-8")
    test = Path(sys.argv[2]).read_text(encoding="utf-8")
    report = overlap_report(prod, test)
    print(json.dumps(report, indent=2))
    return 1 if report["contaminated"] else 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it on the two sides of a proposed diff, not on the whole repository. Whole-repo token sets are dominated by project vocabulary. That inflates the score and hides the leak.

git diff --unified=0 origin/main...HEAD -- src/fee.py > /tmp/prod.diff
git diff --unified=0 origin/main...HEAD -- tests/test_fee.py > /tmp/test.diff
python overlap_gate.py /tmp/prod.diff /tmp/test.diff
echo $?

A non-zero exit is a review flag, not an automatic revert. Shared public names (compute_fee, FeeRequest) are expected. Shared private sentinels are not.

Mutate the constants the test already knows

Overlap is a static smell. Mutation is the dynamic check. Take every non-trivial constant in the new test file and perturb it. If the suite still passes, the assertion was never bound to that value.

# mutate_literals.py — proposed, unexecuted helper
import ast
from pathlib import Path

class NudgeConstants(ast.NodeTransformer):
    def visit_Constant(self, node: ast.Constant):
        if isinstance(node.value, str) and len(node.value) >= 4:
            return ast.copy_location(ast.Constant(node.value + "_x"), node)
        if isinstance(node.value, int) and abs(node.value) >= 10:
            return ast.copy_location(ast.Constant(node.value + 1), node)
        return node

src = Path("tests/test_fee.py").read_text(encoding="utf-8")
tree = ast.parse(src)
Path("/tmp/test_fee.mut.py").write_text(
    ast.unparse(NudgeConstants().visit(tree)), encoding="utf-8"
)

cp tests/test_fee.py /tmp/test_fee.orig.py
cp /tmp/test_fee.mut.py tests/test_fee.py
pytest -q tests/test_fee.py; echo "mutated_exit:$?"
mv /tmp/test_fee.orig.py tests/test_fee.py

Read the two exit codes together. Original pass plus mutated pass means the test is insensitive. Original pass plus mutated fail is the minimum bar. It is still not independence. Independence requires a second author that never saw the patch source.

Split the oracle session

Keep the patch session and the test session apart. The patch session may read failing tests, logs, and the production tree. The oracle session may read the public types, a short property list, and example payloads. It must not receive the patch diff, the patch chat, or the first test file.

A second machine is useful here because it removes accidental context bleed: open editor tabs, reused chats, copied stack traces. MonkeyCode’s free model access and free server option can host that second session when the operator wants the oracle writer off the patch workstation. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims are the only product facts used here. No model names, quotas, hardware, or runtimes are asserted.

Do not paste secrets into that session. Do not upload customer fixtures. Hand the oracle writer a redacted API sketch.

# oracle_brief.md — what the second session is allowed to see
Module: billing.fee
Public function: compute_fee(req: FeeRequest) -> FeeResult
Invariants:
  1. fee >= 0 for every well-formed request
  2. currency on the result equals currency on the request
  3. unknown product_code raises ValueError, never a zero fee
Forbidden inputs to this session:
  - the patch diff
  - any test file from the patch session
  - internal helper names

Numbered merge workflow

1. Capture the patch diff and the test diff as separate files. Do not score the entire tree.
2. Run overlap_gate.py. If contaminated is true, drop the same-session tests. Keep the production hunk.
3. Run literal mutation on any remaining same-session tests. Insensitive tests are discarded, not skipped.
4. Open an isolated oracle session with only the public brief. Generate or hand-write properties against the unpatched tree first. Those properties must fail or skip, not pass.
5. Apply the patch. The same properties must pass without reading the patch source into the oracle session.
6. Record three artifacts in CI: the overlap JSON, the mutation exit pair, and the oracle brief hash.
7. Reject the merge if any of those artifacts is missing. A missing score is a failed gate, not a warning.

# proposed CI fragment
python overlap_gate.py /tmp/prod.diff /tmp/test.diff | tee overlap.json
test "$(python -c 'import json;print(int(json.load(open("overlap.json"))["contaminated"]))')" -eq 0
pytest -q tests/properties/

Decision table

Signal	Meaning	Merge action
Token overlap high, mutation kills tests	Tests know private names but still bind values	Rewrite tests in a split session; keep patch
Token overlap high, mutation still passes	Echoed implementation, no oracle	Discard tests; do not merge on them
Token overlap low, unpatched properties already pass	Oracle is tautological or too weak	Tighten properties; do not credit the patch
Token overlap low, unpatched properties fail, patched properties pass	Independent check	Allow, subject to review
Oracle session received the diff	Process contamination	Invalidate the oracle; start a new session

The table is a policy, not a metric dashboard. Do not average the scores across a week and call the mean a quality index. Each merge is a separate decision.

What this does not prove

Overlap thresholds are heuristics. Public APIs with long identifier names will look “contaminated” if you score the wrong files. Always diff-scope the input.

Literal mutation misses structural cheating: a helper that returns a canned object the test imported from the same generated module. Split-session oracles miss specification errors if the brief is wrong. Neither gate replaces a human reading the patch.

Who should not use this approach: teams whose tests are the specification and must share every domain token with production; air-gapped repos that cannot send even a redacted brief to any off-workstation runner; security-sensitive trees where payloads cannot leave the build network; and anyone hoping a second model session will substitute for review. If the public contract is unstated, isolation only duplicates the confusion.

A free off-workstation session is optional. The required split is contextual, not commercial. Two local directories with two chats and a denied file list already implement the rule. Use a separate runner only when the patch workstation is the contamination source.

Independent oracles cost a second pass. That cost is the point. A cheaper green test that was authored with the patch is not cheaper. It is unread.