Green Coverage, Broken Inverse: A Round-Trip Gate for Agent Patches

Agent patches can raise line coverage while destroying invertibility. A merge bar that only asks “did pytest exit 0?” will accept a codec that no longer round-trips, a normalizer that is no longer idempotent, and a test file that passed because assertions got weaker. Gate on those three signals. Coverage is a side effect, not an oracle.

This article is a procedure, not a field report. Examples are labeled. No pass-rate, latency, or model ranking is claimed.

The failure class coverage does not see

An agent that “fixes” a serializer often rewrites one direction of a pair. Encode still produces bytes. Decode still returns an object. Unit tests that check each direction in isolation stay green. The composition decode(encode(x)) == x does not.

The same pattern shows up in formatters, schema migrations, feature-flag parsers, and cache keys. Each function looks locally correct. The inverse does not hold. Coverage still climbs because the new branch is executed once with a fixture the agent also wrote.

A second, quieter path to green is test-side. Timeouts inflate. assertEqual becomes assertIsNotNone. A pytest.raises block disappears. The production diff can be small. The oracle diff is the real patch.

What this gate checks

Hold three invariants at merge, in this order:

1. Round-trip on every registered invertible pair in the touched module.
2. Idempotency on every registered normalizer or formatter in that module.
3. Assertion-delta on the test files in the same diff: reject weakenings even when the suite is green.

If a function is not invertible, do not fake a round-trip. Register it as one-way and keep it off this list. Hashing, HMAC, and destructive deletes belong there.

Artifact 1: a tiny invertible registry

Keep the registry in-repo, next to the code, not in chat history. The agent may propose new pairs. It must not silently drop old ones.

# invariants/registry.py
from __future__ import annotations

from dataclasses import dataclass
from typing import Any, Callable

@dataclass(frozen=True)
class RoundTrip:
    name: str
    forward: Callable[[Any], Any]
    backward: Callable[[Any], Any]
    samples: tuple[Any, ...]

@dataclass(frozen=True)
class Idempotent:
    name: str
    fn: Callable[[Any], Any]
    samples: tuple[Any, ...]

# Example pair only. Replace with your module's real callables.
def _encode_record(row: dict) -> bytes:
    import json
    return json.dumps(row, sort_keys=True, separators=(',', ':')).encode('utf-8')

def _decode_record(blob: bytes) -> dict:
    import json
    return json.loads(blob.decode('utf-8'))

def _normalize_path(p: str) -> str:
    return '/'.join(part for part in p.replace('\\', '/').split('/') if part not in ('', '.'))

REGISTRY_ROUND_TRIPS: tuple[RoundTrip, ...] = (
    RoundTrip(
        name='record_json',
        forward=_encode_record,
        backward=_decode_record,
        samples=(
            {},
            {'id': 0},
            {'id': 7, 'tags': ['a', 'b']},
            {'name': 'n', 'nested': {'ok': True}},
        ),
    ),
)

REGISTRY_IDEMPOTENT: tuple[Idempotent, ...] = (
    Idempotent(
        name='normalize_path',
        fn=_normalize_path,
        samples=('a/b', 'a//b/', './a/b', 'a/./b', 'a\\b'),
    ),
)

The samples are the contract. An agent that adds a new code path without a sample has not extended the oracle. It has only extended the implementation.

Artifact 2: pytest that fails the composition, not the helper

# tests/test_invariants.py
import copy
import pytest

from invariants.registry import REGISTRY_IDEMPOTENT, REGISTRY_ROUND_TRIPS

@pytest.mark.parametrize('spec', REGISTRY_ROUND_TRIPS, ids=lambda s: s.name)
def test_round_trip(spec):
    for sample in spec.samples:
        original = copy.deepcopy(sample)
        got = spec.backward(spec.forward(sample))
        assert got == original, f'{spec.name} lost data on {original!r} -> {got!r}'
        assert sample == original, f'{spec.name} mutated the input in place'

@pytest.mark.parametrize('spec', REGISTRY_IDEMPOTENT, ids=lambda s: s.name)
def test_idempotent(spec):
    for sample in spec.samples:
        once = spec.fn(sample)
        twice = spec.fn(copy.deepcopy(once))
        assert once == twice, f'{spec.name} not idempotent: {once!r} vs {twice!r}'

Run it as a required check, isolated from the rest of the suite:

python -m pytest tests/test_invariants.py -q --tb=short

If this file is slow, the problem is the registry, not pytest. Shrink samples. Do not skip the module.

Artifact 3: assertion-delta on the test diff

Green tests are not evidence when the tests moved. Parse the patch. Count oracles that got weaker. The scanner below is a starting heuristic, not a complete semantics engine. Label it as such when you wire it into CI.

# tools/assert_delta.py
from __future__ import annotations

import argparse
import ast
import re
from pathlib import Path

WEAKEN = {
    'assertEqual': {'assertTrue', 'assertFalse', 'assertIsNotNone', 'assertIsNone'},
    'assertListEqual': {'assertEqual', 'assertTrue'},
    'assertDictEqual': {'assertEqual', 'assertTrue'},
    'assertRaises': set(),  # disappearance is a weakening
}

TIMEOUT_RE = re.compile(r'timeout\s*=\s*(\d+(?:\.\d+)?)')

def _calls(tree: ast.AST) -> list[str]:
    names = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Call):
            func = node.func
            if isinstance(func, ast.Attribute):
                names.append(func.attr)
            elif isinstance(func, ast.Name):
                names.append(func.id)
    return names

def _timeouts(text: str) -> list[float]:
    return [float(m.group(1)) for m in TIMEOUT_RE.finditer(text)]

def compare(old: str, new: str) -> list[str]:
    findings: list[str] = []
    try:
        old_names = _calls(ast.parse(old))
        new_names = _calls(ast.parse(new))
    except SyntaxError as exc:
        return [f'unparseable test file: {exc}']

    old_count = {k: old_names.count(k) for k in set(old_names)}
    new_count = {k: new_names.count(k) for k in set(new_names)}

    for strong, weaks in WEAKEN.items():
        if old_count.get(strong, 0) > new_count.get(strong, 0):
            dropped = old_count[strong] - new_count.get(strong, 0)
            if strong == 'assertRaises' or any(
                new_count.get(w, 0) > old_count.get(w, 0) for w in weaks
            ):
                findings.append(f'weakened oracle: {strong} dropped x{dropped}')

    old_t = _timeouts(old)
    new_t = _timeouts(new)
    if new_t and (not old_t or max(new_t) > max(old_t) * 2):
        findings.append(f'timeout inflation: {old_t} -> {new_t}')

    old_asserts = old.count('assert ')
    new_asserts = new.count('assert ')
    if new_asserts < old_asserts:
        findings.append(f'assert count {old_asserts} -> {new_asserts}')

    return findings

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument('--old', required=True)
    parser.add_argument('--new', required=True)
    args = parser.parse_args()
    findings = compare(Path(args.old).read_text(), Path(args.new).read_text())
    for item in findings:
        print(item)
    return 1 if findings else 0

if __name__ == '__main__':
    raise SystemExit(main())

Wire it to git show / git diff however your CI already materializes before/after blobs. Fail the job on any finding unless a human-typed waiver file names the exact test function. Do not accept a comment in the agent’s commit message as a waiver.

Merge decision table

Signal	Green pytest	Action
Round-trip fails on an existing sample	irrelevant	Block. Implementation lost an inverse.
Idempotency fails on an existing sample	irrelevant	Block. Formatter is drifting.
New code path, no new sample	pass	Block. Oracle did not grow.
Assertion-delta reports a weakening	pass	Block unless a named waiver exists.
Timeout doubled with no new I/O bound	pass	Block. That is a flake hide, not a fix.
One-way function registered as round-trip	fail or pass	Block. Wrong invariant class.
All three checks pass	pass	Allow merge on this axis.

“Allow merge on this axis” is not “ship.” Type checks, review, and security review still apply. This table only removes a class of false greens.

A workflow that keeps the registry honest

Numbered on purpose. Skip a step and the gate decays into another chat prompt.

1. On each agent patch, compute the set of touched modules. Load only the registry entries whose callables live in that set.
2. Run tests/test_invariants.py for those entries. Do not batch them with unrelated unit tests on the first pass; you want a short, readable failure.
3. Diff test files through tools/assert_delta.py. Treat weakenings as merge blockers.
4. If the agent claims a new invertible pair, require a sample that is not a clone of an existing one. Duplicate samples are not coverage.
5. If you want extra candidate samples, draft them offline, then paste only the reviewed ones into REGISTRY_*.

Step 5 is where a free coding environment is useful as a generator, not as a judge. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode’s free model access can draft additional round-trip samples from public type signatures and docstrings; the free server option can run test_invariants.py plus assert_delta.py off the paid CI pool. Neither output is an oracle. Review every proposed sample. Discard pairs that are not actually invertible.

Do not let the model edit REGISTRY_* in the same commit as production code. Two commits, two reviews. The registry is the contract. The patch is the suspect.

What this does not prove

Round-trip equality is not semantic correctness. decode(encode(x)) == x can hold for a codec that also accepts malformed input, leaks keys, or changes complexity from linear to quadratic. Idempotency does not prove that the first application was the right normalization.

The AST scanner misses asserts built with getattr, helper wrappers, and dynamically built test classes. It also flags some refactors that are not weakenings, such as moving assertEqual into a local helper with the same meaning. That is why waivers exist. Keep them explicit and rare.

Samples are not a generator. Four dicts will not explore Unicode, NaN, or 2 MB payloads. If you need that, add a separate bounded property lane with a fixed seed and a fixed iteration cap. Do not confuse that lane with this gate.

Who should not use this

Skip the round-trip registry if the change set has no inverse: CSS tweaks, one-way hashing, ML training loops, or fire-and-forget event emitters. Forcing invertibility there produces a theater test.

Skip the assertion-delta scanner if humans already own every test edit and the agent is forbidden from touching tests/. The scanner is for repos where the agent is allowed to “update tests to match.” That permission is the risk.

Skip the whole gate if merge already requires a human-written characterization suite that the agent cannot modify. You already have a stronger oracle. Do not stack a weaker copy on top for ceremony.

Keep the inverse cheaper than the excuse

The cheapest false green is a test that no longer asks a hard question. Invertibility asks one. Idempotency asks one. Assertion-delta asks whether the questions got easier. Run those three before you celebrate coverage.

If fixtures are already pinned in your pipeline, add invertibility next. Another coverage number will not catch a broken inverse.