DEV Community

Finley Zhou
Finley Zhou

Posted on

Score Test Hunks Before You Trust a Green Agent Patch

#testing #python #agents #ci

A green suite after an agent patch is not a product result if the patch also edited tests. It is a mixed result. Split the diff. Score the test hunks for assertion erosion before you interpret any product-test outcome.

Agents change more than production files. They insert skip markers. They widen pytest.approx. They drop pytest.raises. They inflate timeouts. Each of those edits can turn CI green without proving the behavior the patch claims to implement. Treat every test hunk as a second hypothesis. Give it its own verdict.

Erosion is a change in observability

Erosion is any test-corpus edit that makes a failure harder to observe. Strengthening is a new falsifiable check. Neutral edits do not change observability: comments, import order, renamed locals that remain inside an assert. Privileged files are not ordinary tests. conftest.py, pytest.ini, and pytest keys in pyproject.toml reconfigure the runner. Score them as infrastructure.

Do not net the two counts. One deleted raises is not offset by two new equality checks on a helper. Report erosion and strengthening separately. Fail closed on erosion.

This policy is about what the suite can still catch. It is not a style grade. A patch may add careful tests and still fail the gate. That is intended.

Why product tests cannot see this

Product tests answer a different question. They ask whether the current corpus, after the patch, is green. They do not ask whether the corpus lost a contract.

Line coverage can rise when a hard test is skipped and a shallow happy-path test is added. Mutation testing of production code will not resurrect a deleted raises block. Timeout inflation does not appear as a failure if the test still finishes. A skip marker is invisible to pass/fail once the test is no longer collected as an active check. Those meters can all look healthy. The test-hunk scan is the check that looks at the corpus itself.

Decision table for test hunks

Edit in the test corpus Class Gate
Removed assert, pytest.raises, or self.assert* erosion reject
New skip, xfail, or skipif without an allowlisted ticket token erosion reject
Timeout increased, or timeout= introduced on an existing test erosion reject
pytest.approx rel/abs increased, or almostEqual places decreased erosion reject
Bare except / except Exception: pass added erosion reject
assert True or an empty test body added erosion reject
New test function with at least one product assertion strengthening allow
Comment-only or whitespace-only test change neutral allow
Fixture extract that preserves assertion lines refactor human review
Any hunk in conftest.py, pytest.ini, tox.ini, or pytest config keys privileged reject unless path allowlisted

The table is a policy artifact. Copy it into the repo next to the scanner. Change it in review, not in the same patch that the agent produced.

Workflow

Run the test-hunk gate before the product suite. Product tests are expensive. They are also meaningless if the corpus was weakened to make them pass.

  1. Export two views of the same patch. Keep one SHA. From that SHA, derive a full unified diff. Do not let the agent choose which files enter the scan.
  2. Classify paths with audited rules. Map each file to product, test, or privileged. Filename rules belong in the scanner, not in a prompt. If a path matches both test and privileged, privileged wins.
  3. Scan test and privileged hunks only. Count erosion signals, strengthening signals, and privileged hits. Write one JSON verdict. Leave product hunks out of this score.
  4. Fail closed on erosion or privileged hits. Do not start the product suite. Do not merge. The product hypothesis is not yet testable against a trusted corpus.
  5. Run product tests only after a clean test-hunk verdict. Accept strengthen, neutral, or product_only. Record both verdicts in CI. Do not fold them into a single pass.

The order is the method. A product-test pass that follows an eroded corpus is not evidence about the product. It is evidence that the corpus no longer objects.

Constructed patch

The unified diff below is a constructed example. It is not taken from a production repository. It exists so the scanner output can be checked by hand.

--- a/billing/tax.py
+++ b/billing/tax.py
@@ -12 +12 @@
-    return amount * Decimal("0.20")
+    return amount * Decimal("0.18")

--- a/tests/test_tax.py
+++ b/tests/test_tax.py
@@ -8 +8 @@
-    assert compute_tax(Decimal("100")) == Decimal("20.00")
+    assert compute_tax(Decimal("100")) == pytest.approx(Decimal("18.00"), rel=1e-1)
@@ -14,3 +14,3 @@
-def test_rejects_negative():
-    with pytest.raises(ValueError):
-        compute_tax(Decimal("-1"))
+@pytest.mark.skip(reason="flaky on agent runner")
+def test_rejects_negative():
+    compute_tax(Decimal("-1"))
Enter fullscreen mode Exit fullscreen mode

Three corpus edits happened besides the rate change. The equality check became a wide approx. A raises contract disappeared. A skip appeared without a ticket token. If you only run tests, the file can go green. The test-hunk verdict must still be erode.

Scanner

The script uses the standard library only. Point it at a unified diff on stdin. It prints one JSON object and exits 1 on erosion or privileged hits. Treat it as a starting gate, not a full AST proof.

#!/usr/bin/env python3
"""Score test hunks in a unified diff for assertion erosion."""
from __future__ import annotations

import json
import re
import sys
from dataclasses import dataclass, field

TEST_PATH = re.compile(
    r"(?:^|/)(tests?/|test_[^/]+\.py$|[^/]+_test\.py$)", re.I
)
PRIVILEGED_PATH = re.compile(
    r"(?:^|/)(conftest\.py$|pytest\.ini$|tox\.ini$|setup\.cfg$|pyproject\.toml$)",
    re.I,
)
REMOVED_CONTRACT = [
    re.compile(r"\bassert\b"),
    re.compile(r"pytest\.raises"),
    re.compile(r"self\.assert"),
]
ADDED_EROSION = [
    re.compile(r"pytest\.mark\.(skip|skipif|xfail)\b"),
    re.compile(r"@unittest\.skip"),
    re.compile(r"\bassert\s+True\b"),
    re.compile(r"except\s+(Exception|BaseException)?\s*:\s*(pass|\.\.\.)"),
]
TICKET = re.compile(r"\b(TICKET|ISSUE|JIRA|GH)-\d+\b", re.I)
TIMEOUT = re.compile(r"timeout\s*=\s*(\d+(?:\.\d+)?)")
APPROX_REL = re.compile(r"pytest\.approx\([^)]*\brel\s*=\s*([0-9eE.+-]+)")
STRENGTHEN = [
    re.compile(r"\bassert\b"),
    re.compile(r"pytest\.raises"),
    re.compile(r"self\.assert"),
]
HUNK_FILE = re.compile(r"^\+\+\+\s+(?:b/)?(.+)$")


def classify_path(path: str) -> str:
    if path == "/dev/null":
        return "product"
    if PRIVILEGED_PATH.search(path):
        return "privileged"
    if TEST_PATH.search(path):
        return "test"
    return "product"


@dataclass
class Report:
    erosion: list[str] = field(default_factory=list)
    strengthen: list[str] = field(default_factory=list)
    privileged_files: list[str] = field(default_factory=list)
    test_files: list[str] = field(default_factory=list)
    product_files: list[str] = field(default_factory=list)

    def klass(self) -> str:
        if self.privileged_files:
            return "privileged"
        if self.erosion:
            return "erode"
        if self.strengthen:
            return "strengthen"
        if self.test_files:
            return "neutral"
        return "product_only"


def parse(lines: list[str]) -> Report:
    report = Report()
    path = ""
    kind = "product"
    removed_timeouts: list[float] = []
    added_timeouts: list[float] = []
    removed_rel: list[float] = []
    added_rel: list[float] = []

    def flush_numeric(current: str, current_kind: str) -> None:
        if current_kind != "test" or not current:
            return
        if added_timeouts and (
            not removed_timeouts or max(added_timeouts) > max(removed_timeouts)
        ):
            report.erosion.append(f"{current}: timeout widened or introduced")
        if added_rel and (not removed_rel or max(added_rel) > max(removed_rel)):
            report.erosion.append(f"{current}: approx rel widened or introduced")

    for raw in lines:
        if raw.startswith("+++ "):
            flush_numeric(path, kind)
            path = HUNK_FILE.match(raw).group(1).strip() if HUNK_FILE.match(raw) else ""
            kind = classify_path(path)
            removed_timeouts, added_timeouts = [], []
            removed_rel, added_rel = [], []
            if kind == "privileged":
                report.privileged_files.append(path)
            elif kind == "test":
                report.test_files.append(path)
            elif path and path != "/dev/null":
                report.product_files.append(path)
            continue
        if not path or raw[:1] not in {"+", "-"} or raw.startswith(("+++", "---")):
            continue
        body = raw[1:]
        if kind == "product":
            continue
        if kind == "privileged" and body.strip():
            continue
        if raw.startswith("-"):
            if any(p.search(body) for p in REMOVED_CONTRACT):
                report.erosion.append(f"{path}: removed contract: {body.strip()}")
            if TIMEOUT.search(body):
                removed_timeouts.append(float(TIMEOUT.search(body).group(1)))
            if APPROX_REL.search(body):
                removed_rel.append(float(APPROX_REL.search(body).group(1)))
            continue
        if any(p.search(body) for p in ADDED_EROSION):
            if TICKET.search(body):
                continue
            report.erosion.append(f"{path}: added silence: {body.strip()}")
        if any(p.search(body) for p in STRENGTHEN):
            report.strengthen.append(f"{path}: {body.strip()}")
        if TIMEOUT.search(body):
            added_timeouts.append(float(TIMEOUT.search(body).group(1)))
        if APPROX_REL.search(body):
            added_rel.append(float(APPROX_REL.search(body).group(1)))
    flush_numeric(path, kind)
    report.privileged_files = sorted(set(report.privileged_files))
    report.test_files = sorted(set(report.test_files))
    report.product_files = sorted(set(report.product_files))
    return report


def main() -> int:
    report = parse(sys.stdin.read().splitlines())
    payload = {
        "class": report.klass(),
        "erosion": report.erosion,
        "strengthen": report.strengthen,
        "privileged_files": report.privileged_files,
        "test_files": report.test_files,
        "product_files": report.product_files,
    }
    json.dump(payload, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 1 if payload["class"] in {"erode", "privileged"} else 0


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

Run it against the current branch:

git diff --unified=0 origin/main...HEAD | python3 score_test_hunks.py
Enter fullscreen mode Exit fullscreen mode

On the constructed diff, class must be erode and the process must exit non-zero. The product rate change is ignored at this stage on purpose. That line belongs to the product hypothesis. It is scored later, against a corpus that has not been silently weakened.

Save the JSON. The later product-test job should refuse to start unless class is strengthen, neutral, or product_only.

Wiring in CI

Keep the job tiny. Do not install the app. Do not boot fixtures. The gate only needs git and Python 3.

# constructed CI sketch — unexecuted example
test-hunk-gate:
  stage: gate
  script:
    - git diff --unified=0 origin/main...HEAD > patch.diff
    - python3 score_test_hunks.py < patch.diff > test_hunk_verdict.json
  artifacts:
    paths: [test_hunk_verdict.json]
Enter fullscreen mode Exit fullscreen mode

Parse the verdict in later jobs. If class is erode or privileged, skip product tests and fail the pipeline. If a human override is required, demand a dated allowlist entry that names the file, the erosion kind, and a ticket. Do not accept a blanket flag that says the agent may edit tests.

A second command is useful when you want to inspect only test paths before the scanner runs:

git diff --unified=0 --name-only origin/main...HEAD | python3 -c "
import re, sys
priv = re.compile(r'(conftest\\.py|pytest\\.ini|pyproject\\.toml)$')
test = re.compile(r'(^|/)(tests?/|test_[^/]+\\.py$|[^/]+_test\\.py$)')
for line in sys.stdin:
    p = line.strip()
    bucket = 'privileged' if priv.search(p) else 'test' if test.search(p) else 'product'
    print(f'{bucket}\t{p}')
"
Enter fullscreen mode Exit fullscreen mode

Use that listing in review. The scanner still makes the merge decision. A human listing is not a substitute for the fail-closed exit code.

Cheap isolated compute

Some teams generate the product patch with a model and still need a place to run this gate without attaching it to the main paid runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option can host that split: the model proposes the patch, the server runs score_test_hunks.py as its own job. The scanner does not depend on that product. A local pipe into the same script is enough.

If you already evaluate patches on an isolated server, add this scan as a sibling job. Do not replace product tests with it.

Limitations

The scanner is line-oriented. It will miss an assertion rewritten across lines. It will false-positive when a refactor replaces assert x == y with a stricter helper whose name is not in the pattern list. It is Python-centric. Go, JavaScript, and Java suites need their own pattern tables.

It does not prove that a new assert is about product behavior rather than about a mock. It does not detect property-test vacuity of the form "for all x, True". It does not replace review of newly added tests. Unified diffs with renamed files can look like mass deletion plus mass addition. Pair this gate with git diff --detect-renames, or mechanical moves will be rejected. Large golden blobs that are not assertions will not be scored well. Keep those behind the path classifier as product or as a third artifact class.

Numeric comparisons on timeout and rel are per-file, not per-test. Two tests in one file can confuse the max-value check. Tighten that with an AST pass if the false-positive rate matters in your repo.

Who should not use this

Do not use this gate if your policy is already "agents may never edit tests." A path denylist is simpler and stricter. Do not use it as the only review on security or privacy patches. Do not use the default regex table on a mixed-language monorepo and trust the zeros.

Do not let the same agent that wrote the patch update the allowlist. Do not treat strengthen as permission to skip human review of new tests that assert on mocks, clocks, or network stubs. Do not net erosion against strengthening in a later "improvement" of the script. That change would reintroduce the failure mode this gate exists to catch.

Record two verdicts

Ship a product verdict only after the test-hunk verdict is strengthen or neutral. A mixed green is not green. Keep the JSON artifact next to the patch SHA. When a later incident shows a silenced contract, the erosion class should already be in the log. A single passed job will not tell you which hypothesis actually survived.

Top comments (0)