Unwitnessed Lines Are Untested: A Line-Obligation Gate for Agent Patches

An agent patch is not tested because its suite is green. It is tested when every changed executable line has a trusted witness. Agent-authored assertions do not count as that witness. Coverage that includes those assertions is a vanity metric, not a merge signal.

The gate below fails closed on unwitnessed lines. It accepts three discharges only: a human-owned check, a metamorphic relation over a frozen corpus, or an explicit waiver with an expiry date and a reason code. Everything else is residual risk.

The failure mode this gate is for

Agents write the implementation and the example tests in the same turn. The tests encode the code. The code satisfies the tests. A coverage report then marks the new lines as hit. Nothing in that loop asked whether the change is correct against a source the agent does not control.

Exact-output goldens are the wrong fix for this. So is blaming flaky tests. The missing object is an obligation: each executable line in the diff must be discharged by something the agent is not allowed to edit.

This article proposes a line-obligation map. It is a merge artifact, not a dashboard widget. Treat it as a patch-local proof obligation, closer to a type checker than to a coverage badge.

What counts as a witness

Classify every changed line that a compiler or interpreter can execute. Comments, blank lines, and pure import reshuffles are out of scope. The rest must land in exactly one bucket.

1. Human-owned check. A test file under a path the agent cannot modify, for example tests/trusted/. CODEOWNERS or a CI path filter must reject agent edits there. The check must execute the line on this patch.
2. Metamorphic relation. A relation R(x, T(x)) that does not assert a single golden y. Relations live in tests/relations/ and consume a hashed corpus. The agent may propose a relation. A reviewer must accept it before it discharges anything.
3. Waiver. A YAML record naming the file, the line hash, a reason code, an owner, and an expiry. Expired waivers fail the gate. “Will add tests later” is not a reason code.

Agent-authored tests under tests/generated/ may still run. They never discharge an obligation. That split is the whole method.

Artifact: a line-obligation map

The map is a JSON document produced from git diff plus a coverage export that was collected while running only trusted checks and relations. Generated-test coverage is dropped before scoring.

# Proposed local commands. Label: unexecuted example.
git diff --unified=0 origin/main...HEAD > /tmp/patch.diff
pytest tests/trusted tests/relations \
  --cov=src --cov-branch --cov-report=json:/tmp/cov_trusted.json
python tools/obligation_map.py \
  --diff /tmp/patch.diff \
  --cov /tmp/cov_trusted.json \
  --waivers tests/waivers.yaml \
  --out /tmp/obligation.json

A minimal mapper in Python is enough to make the rule reviewable. It does not need a platform. It needs a stable line identity: path, content hash of the new line, and whether trusted coverage hit it.

# tools/obligation_map.py — proposed harness, not a measured CI run
from __future__ import annotations

import argparse, ast, hashlib, json, re
from pathlib import Path

HUNK_RE = re.compile(r"^@@ -(\d+)(?:,(\d+))? \+(\d+)(?:,(\d+))? @@")
REASON_CODES = {"codegen", "vendor", "pure-refactor", "experiment"}

def sha1(s: str) -> str:
    return hashlib.sha1(s.encode()).hexdigest()[:12]

def parse_diff(text: str) -> list[dict]:
    lines = []
    path, new_ln = None, 0
    for raw in text.splitlines():
        if raw.startswith("+++ b/"):
            path = raw[6:]
            continue
        m = HUNK_RE.match(raw)
        if m:
            new_ln = int(m.group(3))
            continue
        if path is None or path.startswith("tests/generated/"):
            if raw.startswith("+") and not raw.startswith("+++"):
                new_ln += 1
            continue
        if raw.startswith("+") and not raw.startswith("+++"):
            content = raw[1:]
            lines.append({
                "path": path,
                "line": new_ln,
                "hash": sha1(content),
                "content": content,
                "executable": is_executable(path, content),
            })
            new_ln += 1
        elif not raw.startswith("-"):
            new_ln += 1
    return lines

def is_executable(path: str, content: str) -> bool:
    s = content.strip()
    if not s or s.startswith("#") or path.endswith((".md", ".json", ".yaml", ".yml")):
        return False
    if path.endswith(".py"):
        try:
            ast.parse(s)
        except SyntaxError:
            # Incomplete snippet still counts if it is not a comment.
            return not s.startswith(("\"\"\"", "'''"))
    return True

def trusted_hits(cov: dict) -> set[tuple[str, int]]:
    hits = set()
    for path, rec in cov.get("files", {}).items():
        executed = rec.get("executed_lines", [])
        for n in executed:
            hits.add((path.replace("\\", "/"), int(n)))
    return hits

def load_waivers(path: Path, today: str) -> dict[tuple[str, str], dict]:
    import yaml
    data = yaml.safe_load(path.read_text()) or {}
    out = {}
    for w in data.get("waivers", []):
        if w.get("reason") not in REASON_CODES:
            raise SystemExit(f"invalid reason: {w}")
        if w.get("expires", "0000-00-00") < today:
            raise SystemExit(f"expired waiver: {w}")
        out[(w["path"], w["line_hash"])] = w
    return out

def main() -> None:
    p = argparse.ArgumentParser()
    p.add_argument("--diff")
    p.add_argument("--cov")
    p.add_argument("--waivers")
    p.add_argument("--out")
    p.add_argument("--today", default="2026-09-16")
    args = p.parse_args()
    changed = parse_diff(Path(args.diff).read_text())
    hits = trusted_hits(json.loads(Path(args.cov).read_text()))
    waivers = load_waivers(Path(args.waivers), args.today)
    rows = []
    for item in changed:
        if not item["executable"]:
            continue
        key = (item["path"], item["hash"])
        if (item["path"], item["line"]) in hits:
            status = "trusted-hit"
        elif key in waivers:
            status = "waived"
        else:
            status = "unwitnessed"
        rows.append({**item, "status": status})
    summary = {
        "unwitnessed": sum(r["status"] == "unwitnessed" for r in rows),
        "trusted_hit": sum(r["status"] == "trusted-hit" for r in rows),
        "waived": sum(r["status"] == "waived" for r in rows),
        "rows": rows,
    }
    Path(args.out).write_text(json.dumps(summary, indent=2))
    if summary["unwitnessed"]:
        raise SystemExit(f"unwitnessed lines: {summary['unwitnessed']}")

if __name__ == "__main__":
    main()

The exit code is the policy. Non-zero means the patch still has executable lines with no trusted hit and no live waiver. Do not parse the JSON for a percentage and then “accept 90%”. Percentages reintroduce the vanity metric this gate removes.

Numbered workflow

Run this as a required check on every agent patch, after the agent has finished editing and before human review of behavior.

1. Freeze the trusted tree. tests/trusted/ and tests/relations/ must be unmodifiable by the agent job. Enforce that with a path denylist in the orchestrator, not with a prompt instruction.
2. Compute the executable diff. Use git diff --unified=0 against the merge base. Feed it to the mapper. Do not use file-level “touched or not” flags. A one-line change in a 400-line file is one obligation, not 400.
3. Collect trusted coverage only. Invoke pytest on trusted checks and relations. Pass --cov for the application package. Discard any .coverage data produced by tests/generated/.
4. Score the map. A line is discharged when its new-line number appears in the trusted coverage export, or when a waiver matches path + line_hash. Matching on line number alone is unstable across rebases. The content hash is the identity.
5. Propose, do not auto-accept, relations. If unwitnessed lines remain, a model may draft a relation and a corpus slice. A reviewer either lands that relation in tests/relations/ or files a waiver. Drafts in /tmp discharge nothing.
6. Fail closed. Unwitnessed count greater than zero blocks merge. No skip label except the waiver file itself.

Relations that actually discharge lines

A relation is useful when you cannot name the correct output, but you can name an invariant across inputs. That is common for parsers, batch APIs, codecs, and pure transforms. It is a poor fit for one-shot UI copy.

# tests/relations/test_batch_permutation.py — proposed example
import json
from pathlib import Path
from src.billing import price_order  # application under test

CORPUS = Path(__file__).parent / "corpus" / "orders.jsonl"

def test_item_permutation_preserves_total():
    for raw in CORPUS.read_text().splitlines():
        order = json.loads(raw)
        items = list(order["items"])
        if len(items) < 2:
            continue
        base = price_order(order)
        flipped = dict(order, items=list(reversed(items)))
        assert price_order(flipped)["total"] == base["total"]

Pin the corpus with a manifest. If the agent rewrites orders.jsonl to make the relation pass, the manifest hash changes and the gate fails for a different reason: corpus drift.

# tests/relations/corpus/MANIFEST.sha256
# sha256sum orders.jsonl > MANIFEST.sha256  (recompute only in a human commit)
e3b0c44298fc1c149afbf4c8996fb924 orders.jsonl

Replace the placeholder digest with a real sha256sum output in the human commit that lands the corpus. The check is sha256sum -c MANIFEST.sha256 in the same job that runs relations.

Waiver file, kept small on purpose

# tests/waivers.yaml
waivers:
  - path: src/codegen/emit.py
    line_hash: "a1b2c3d4e5f6"
    reason: codegen
    owner: platform-reviewers
    expires: "2026-10-16"
    note: "Line is emitted by the schema compiler; oracle is the .proto file."

Four reason codes are enough to start: codegen, vendor, pure-refactor, experiment. If a team needs a fifth, the fifth is usually “we do not want to write a trusted check.” Reject that. Waivers are for lines whose oracle lives outside the test runner, not for fatigue.

Decision table

Changed line kind	Trusted hit?	Allowed discharge	Gate result
Branch in application code	Yes	Human-owned check or relation	Pass
Branch in application code	No	Waiver with codegen/vendor only	Pass if waiver live
Branch in application code	No	Agent unit test only	Fail
Comment or blank	n/a	None required	Pass
Test under tests/generated/	n/a	Never a witness	Ignored as obligation
Relation corpus file	n/a	Manifest hash must match	Fail on drift
Trusted test edit by agent	n/a	Path denylist	Fail before scoring

The table is the reviewer’s cheat sheet. If a row is not in it, add a row before adding a skip.

Where a free model and a free server belong

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The obligation map should run on a machine that is not the developer’s laptop cache and not the production CI pool. Local .pyc files, leftover coverage data, and pre-seeded virtualenvs all inflate trusted hits. A clean runner that boots from the patch ref is the minimum isolation.

MonkeyCode’s free model access is relevant at step 5 only: draft candidate relations and waiver notes from the unwitnessed slice of /tmp/obligation.json. The drafts are prompts for a reviewer. They are not evidence. MonkeyCode’s free server option is relevant at steps 3–6: run trusted pytest, the mapper, and sha256sum -c on that runner so the gate is reproducible without attaching the job to a billed workflow.

Do not send the model the trusted tests and ask it to “make coverage go up.” That recreates the original failure. Send it the unwitnessed lines and the corpus schema, and require a human commit to land any relation it proposes.

Limitations

Line identity is brittle in files the agent reformats wholesale. A black/rustfmt-only commit should be split from a behavior commit, or every reformatted line becomes an obligation. That is correct, and it is also noisy. Split the commits.

Coverage instrumentation misses some executable lines: except paths, __repr__ debug arms, and code behind TYPE_CHECKING. Unwitnessed then over-reports. The fix is a targeted trusted check or a waiver with experiment and a short expiry, not a global ignore.

Metamorphic relations do not prove functional correctness. A permutation-invariant pricing bug still passes test_item_permutation_preserves_total. Relations shrink the unwitnessed set. They do not replace a human-owned example for the business rule that actually changed.

The mapper above understands Python at a shallow level. Other languages need their own is_executable predicate and a coverage format adapter. Copying the JSON schema is fine. Copying the AST heuristic is not.

This workflow also assumes you can prevent the agent from editing tests/trusted/, tests/relations/, and tests/waivers.yaml. If your orchestrator cannot enforce a write denylist, the map is theater. Prompt-level “please do not touch trusted tests” is not a control.

Who should not use this

Do not install this gate on patches that a reviewer already wrote tests for, line by line, in tests/trusted/. The map will pass, and you will have paid for a report nobody needed.

Do not use it as the only check on security-sensitive diffs: authz, crypto, and secret handling need dedicated oracles, threat review, and often a second human. An unwitnessed-line count of zero is not a security audit.

Do not use it on generated trees where the generator commit is the oracle and the output is not executed in production. Waiver reason codegen exists for the thin wrapper around that output, not for vendoring the entire generated tree into the obligation set.

Teams with no human-owned corpus should not start here. Start by moving three existing tests into tests/trusted/ and locking the path. The map has nothing to score against if every check is agent-owned.

Close

Green agent tests measure self-consistency. A line-obligation map measures whether the diff was observed by a check the agent could not rewrite. Keep the generated suite if it is useful for smoke. Keep it out of the witness set.

If you already run agent jobs on a throwaway runner, add the mapper next to trusted pytest and fail on unwitnessed > 0. MonkeyCode’s free server option is one place to keep that job off both laptops and production CI, without changing the rule the JSON encodes.