The Agent Patch Loosened the Assertion and the Suite Stayed Green: An Oracle-Latitude Gate

A patch can keep every test green and still weaken what the tests mean. Count the latitude of the oracle, not the colour of the run, and gate on that count.

Last week I spent an hour reviewing a diff where the only test change was one line:

-    assert invoice.total == Decimal("42.00")
+    assert invoice.total > 0

The suite passed before and after. The seeded fixture still produced 42.00, so nothing broke. But the test no longer claims anything precise — a total of 0.01 would also satisfy it. Fixing fixtures, pinning inputs, or mutating bugs would not have caught this, because the fixture was fine and the bug was never introduced. The oracle itself shrank.

Why pass/fail cannot see this

Pass/fail is a single bit evaluated against one fixture. Latitude is a property of the assertion, independent of any fixture.

An agent editing a failing test has an obvious gradient: make the assertion cheaper to satisfy. Replace equality with a range, ValueError with Exception, 1e-9 with 0.5, or delete the second assert. Each move is locally reasonable and each one reduces the amount of behaviour the test constrains.

Move	Example	What is lost
Comparison weakening	== 42 → > 0	The expected value
Exception widening	raises(ValueError) → raises(Exception)	The failure mode
Tolerance inflation	abs(a-b) < 1e-9 → < 0.5	Numerical precision
Assertion removal	3 asserts → 1	Whole claims
Skip / xfail insertion	@pytest.mark.skip	The test itself

Step 1 — Extract assertion facts, not assertion text

String-diffing tests gives you noise. Parse them and record facts per test function: comparison operator ranks, exception types, numeric tolerances, assertion counts, skip markers.

# latitude.py  (Python 3.11+, stdlib only)
from __future__ import annotations

import argparse
import ast
import json
from pathlib import Path

CMP_RANK = {
    ast.Eq: 0, ast.NotEq: 0, ast.Is: 0, ast.IsNot: 0,
    ast.Lt: 1, ast.LtE: 1, ast.Gt: 1, ast.GtE: 1,
    ast.In: 2, ast.NotIn: 2,
}
UNIT_MATCHERS = {
    "assertEqual": 0, "assertIs": 0, "assertIsNone": 0,
    "assertIn": 2, "assertAlmostEqual": 2, "assertTrue": 3,
}
EXC_RANK = {"ValueError": 0, "KeyError": 0, "TypeError": 0,
            "Exception": 3, "BaseException": 3}


def test_functions(tree: ast.AST):
    for node in ast.walk(tree):
        if isinstance(node, (ast.FunctionDef, ast.AsyncFunctionDef)) \
                and node.name.startswith("test"):
            yield node


def tolerance(node: ast.AST) -> float | None:
    """Recognise abs(a - b) < X and pytest.approx(..., abs=X|rel=X)."""
    if isinstance(node, ast.Compare) and len(node.ops) == 1:
        left = node.left
        right = node.comparators[0]
        if (isinstance(left, ast.Call) and getattr(left.func, "id", None) == "abs"
                and isinstance(right, ast.Constant)
                and isinstance(right.value, (int, float))):
            return float(right.value)
    if isinstance(node, ast.Call) and getattr(node.func, "attr", None) == "approx":
        for kw in node.keywords:
            if kw.arg in ("abs", "rel") and isinstance(kw.value, ast.Constant):
                return float(kw.value.value)
    return None


def facts(fn: ast.AST) -> dict:
    out = {"compare": [], "tolerance": [], "raises": [],
           "asserts": 0, "skip": False}
    for node in ast.walk(fn):
        if isinstance(node, ast.Assert):
            out["asserts"] += 1
            if isinstance(node.test, ast.Compare):
                out["compare"].append(
                    sorted(CMP_RANK.get(type(op), 2) for op in node.test.ops))
            else:
                out["compare"].append([3])  # bare truthiness
        if isinstance(node, ast.Call):
            name = getattr(node.func, "attr", None)
            if name in UNIT_MATCHERS:
                out["asserts"] += 1
                out["compare"].append([UNIT_MATCHERS[name]])
            if name == "raises" and node.args and isinstance(node.args[0], ast.Name):
                out["raises"].append(EXC_RANK.get(node.args[0].id, 1))
        tol = tolerance(node)
        if tol is not None:
            out["tolerance"].append(tol)
        if isinstance(node, ast.Attribute) and node.attr in ("skip", "xfail"):
            out["skip"] = True
    return out


def scan(root: Path) -> dict[str, dict]:
    found = {}
    for path in sorted(root.rglob("test_*.py")):
        tree = ast.parse(path.read_text(encoding="utf-8"))
        for fn in test_functions(tree):
            found[f"{path.relative_to(root)}::{fn.name}"] = facts(fn)
    return found

Step 2 — Diff the facts across base and patch

Every fact pair maps to a weight. Positive means the oracle got cheaper. Negative means it got stricter and needs no gate.

SIGNAL_WEIGHTS = {"compare": "rank delta", "raises": "rank delta",
                  "tolerance": 2, "dropped_assert": 1, "new_skip": 4}


def score(base: dict, patch: dict) -> tuple[int, list[str]]:
    total, why = 0, []
    for i, ranks in enumerate(patch["compare"]):
        before = base["compare"][i] if i < len(base["compare"]) else [0]
        for old, new in zip(before, ranks):
            if new > old:
                total += new - old
                why.append(f"compare rank {old}->{new}")
    for old, new in zip(base["raises"], patch["raises"]):
        if new > old:
            total += new - old
            why.append(f"exception rank {old}->{new}")
    for old, new in zip(base["tolerance"], patch["tolerance"]):
        if new > old:
            total += SIGNAL_WEIGHTS["tolerance"]
            why.append(f"tolerance {old}->{new}")
    dropped = base["asserts"] - patch["asserts"]
    if dropped > 0:
        total += dropped * SIGNAL_WEIGHTS["dropped_assert"]
        why.append(f"{dropped} assertion(s) removed")
    if patch["skip"] and not base["skip"]:
        total += SIGNAL_WEIGHTS["new_skip"]
        why.append("skip/xfail added")
    return total, why


def main(argv: list[str] | None = None) -> int:
    ap = argparse.ArgumentParser()
    ap.add_argument("base")
    ap.add_argument("patched")
    ap.add_argument("--max-score", type=int, default=4)
    ap.add_argument("--json")
    args = ap.parse_args(argv)

    base, patched = scan(Path(args.base)), scan(Path(args.patched))
    report, total_score = {}, 0
    for name in sorted(set(base) & set(patched)):
        value, why = score(base[name], patched[name])
        if value:
            total_score += value
            report[name] = {"score": value, "signals": why}

    verdict = "FAIL" if total_score >= args.max_score else "PASS"
    print(json.dumps({"verdict": verdict, "score": total_score,
                      "tests": report}, indent=2))
    if args.json:
        Path(args.json).write_text(json.dumps(report, indent=2), encoding="utf-8")
    return 0 if verdict == "PASS" else 1


if __name__ == "__main__":
    raise SystemExit(main())

Run it on two worktrees:

git worktree add /tmp/base origin/main
python latitude.py /tmp/base/src/tests src/tests --max-score 4 --json latitude.json

Step 3 — Turn the score into a policy

A number without a policy gets ignored. Post a first-line comment on the patch with the failing test names and the signals, so the reviewer sees the claim that changed rather than the line that changed.

Score	Meaning	Action
0	Oracle unchanged or stricter	Merge on tests alone
1–3	Small widening	Require reviewer acknowledgement
≥ 4	New skip, or multiple widenings	Block; ask for a replacement assertion

The --max-score default of 4 is a starting point, not a standard. Tune it per repository, then freeze it in CI so the threshold cannot drift silently.

Step 4 — Prove the gate with a self-test corpus

A gate nobody tested is a vibe. Keep six hand-written base/patched pairs under tests/fixtures/latitude/ and assert the expected score.

Pair	Change	Expected	Verdict at 4
eq_to_range	== 42 → > 0	1	pass
eq_to_neq	== 42 → != 0	0	pass
widen_exception	raises(ValueError) → raises(Exception)	3	pass
inflate_tolerance	< 1e-9 → < 0.5	2	pass
drop_second_assert	2 asserts → 1	1	pass
add_skip	@pytest.mark.skip added	4	fail

Two of these are negative controls (eq_to_neq must not fire). If your gate fires on those, the ranks are wrong. Keep the corpus in the same repo as the gate and run it in the same job.

Step 5 — Run the loop with free model access and a free server

This gate is most useful against many candidate patches, not one. That means two costs: generating candidates, and running the harness often enough that a green run is not luck.

That is the slot where I used MonkeyCode's free model access to produce candidate patches against a seeded bug, and its free server option to run the harness repeatedly on a machine that is not my laptop. The loop is boring on purpose: generate a patch, run the suite, run latitude.py, keep only patches that are green and score 0, then review those by hand. Availability of the free model access and the free server option is operator-supplied; treat quotas, hardware, and any permanence as things you verify yourself before depending on them.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The step the product genuinely helps with is volume. The step it cannot do for you is deciding whether a widened assertion was the right fix — some of them are, and no AST can tell.

Limitations, and who should skip this

• It is syntactic. Assertions hidden inside helpers, custom matchers, or factories are invisible to the scanner.
• List matching uses zip, so insertion of a new assertion earlier in a function can misalign the comparison. Sort or key your facts if that happens in your codebase.
• It flags widening, not incorrectness. Replacing exact float equality with approx is usually an improvement, and the gate will still score it.
• Python-only as written. The fact model ports to other ASTs, the code does not.
• If your suite is small enough that a human reads every test diff, you do not need this. The gate pays off when test diffs arrive faster than reviewers can read them.

The one thing to keep

Fixtures tell you what the test sees. Latitude tells you what the test claims. If the claim can shrink while the run stays green, your definition of "the tests passed" is doing less work than you think — the corpus of six pairs above is a cheap place to start checking.