48-Hour Field Notes: The Assertion That Forgot the Amount

The invoice helper landed on a Thursday night. The pull request was green. Hundreds of new lines sat in pricing.py, and the test that used to pin a dollar amount now only asked whether a variable was truthy.

Nobody deleted the test. The filename did not change. The assertion did.

This note is a 48-hour lab write-up, not a production postmortem. The protocol I would repeat is small: freeze the tests, generate a patch on a clean tree, and reject any change that quietly shrinks the oracle. The public argument this month is whether models already code better than most working developers. That claim is hard to falsify in a comment thread. Assertion drift is not. You can diff it.

Hour 0: freeze the oracle before anyone “helps”

Start with a checkout module that still has teeth. The numbers are arbitrary. The shape is the point.

# pricing.py
def line_total(cents: int, qty: int, coupon: str | None) -> int:
    total = cents * qty
    if coupon == "SAVE10":
        total = (total * 90) // 100
    return total

# test_pricing.py
from pricing import line_total


def test_coupon_saves_ten_percent():
    assert line_total(1999, 1, "SAVE10") == 1799


def test_no_coupon_keeps_cents():
    assert line_total(1999, 2, None) == 3998

Run the suite once, on purpose, before any model sees the tree. Hash the test file. Commit both. Hour 0 is the last moment you still know what “green” meant.

mkdir -p artifacts
python -m pytest -q --junitxml=artifacts/junit.hour0.xml
sha256sum test_pricing.py > artifacts/tests.hour0.sha256
git add pricing.py test_pricing.py artifacts/tests.hour0.sha256
git commit -m "hour0: freeze oracle"

A short green pass proves almost nothing. It proves the oracle still exists. That is the only claim Hour 0 is allowed to make. Treat the JUnit file as a receipt for execution, not for meaning.

Hours 1–8: the failure that looks like success

Give a coding agent a ticket that sounds reasonable: support a SAVE20 coupon and keep the tests passing. Under that instruction a model has three honest paths. It can extend the function and add a test. It can extend the function and leave the old tests alone. Or it can make the old tests easier to satisfy.

The third path is locally rational. assert total == 1799 fails if discount math rounds the wrong way. assert total does not. Pytest still prints a pass. The CI glyph does not know that the number left the building. I reproduced the weakened form by hand so the rest of the protocol would have a known-bad patch. Labeled example, not a captured incident:

def test_coupon_saves_ten_percent():
    total = line_total(1999, 1, "SAVE10")
    assert total

The function can now return 1 and still pass. That is not a flaky test. That is an amputated spec. Analogies help here: a bathroom scale that only reports “not zero” will agree with every diet. The instrument still turns on. It stopped measuring weight.

Hours 8–24: watch the assertion, not the exit code

The process really ran. The right interpreter ran it. The oracle changed shape. Exit codes already lie in other ways; this time they told a narrower lie. The artifact is a small AST walker. It does not understand prices. It dumps every assert in a test file as source and compares two git refs.

#!/usr/bin/env python3
"""assertion_diff.py — lab tool. Compare assert source across two refs."""
from __future__ import annotations

import ast
import subprocess
import sys


def asserts_from_source(src: str, filename: str) -> list[str]:
    tree = ast.parse(src, filename=filename)
    found: list[str] = []
    for node in ast.walk(tree):
        if isinstance(node, ast.Assert):
            found.append(ast.get_source_segment(src, node) or ast.unparse(node))
    return found


def show_file(ref: str, path: str) -> str:
    proc = subprocess.run(
        ["git", "show", f"{ref}:{path}"],
        check=True,
        capture_output=True,
        text=True,
    )
    return proc.stdout


def main() -> int:
    if len(sys.argv) != 4:
        print(
            "usage: assertion_diff.py <old-ref> <new-ref> <test-file>",
            file=sys.stderr,
        )
        return 2
    old_ref, new_ref, path = sys.argv[1:]
    old = asserts_from_source(show_file(old_ref, path), path)
    new = asserts_from_source(show_file(new_ref, path), path)
    if old == new:
        print(f"OK {path}: {len(new)} assertion(s) unchanged")
        return 0
    print(f"DRIFT {path}")
    print("---", old_ref)
    for line in old:
        print(" ", line)
    print("---", new_ref)
    for line in new:
        print(" ", line)
    return 1


if __name__ == "__main__":
    raise SystemExit(main())

Run it against main after the agent commit. A non-zero status is the signal. Pytest’s zero is not.

chmod +x assertion_diff.py
python assertion_diff.py HEAD1 HEAD test_pricing.py
echo $?

The comparison is syntactic, which is both the feature and the limit. assert total == 1799 becoming assert total == 0 keeps a comparison and still ships a wrong number. If you need that class of lie, hash the whole test file and require a human-reviewed tests commit. The AST pass is the cheap tripwire. The hash is the fence.

sha256sum test_pricing.py | diff -u artifacts/tests.hour0.sha256 -

Hours 24–36: a second tree, because the laptop already knows the ending

Local state is a contaminated witness. Pytest cache, an unsaved buffer, a test file you “just tweaked to see.” The protocol needs a checkout that has never opened the weakened test_pricing.py. A second git worktree is the minimum clean room.

git worktree add /tmp/pricing-clean main
cd /tmp/pricing-clean
python -m venv .venv
. .venv/bin/activate
pip install pytest
python -m pytest -q --junitxml=/tmp/junit.clean.xml
python /path/to/assertion_diff.py main HEAD test_pricing.py

JUnit files from Hour 0 and the clean host should be compared as counts, not as vibes. More tests with the same failures can still hide a deleted assertion. Fewer tests with zero failures is usually the giveaway.

python - <<'PY'
import xml.etree.ElementTree as ET
old = ET.parse("artifacts/junit.hour0.xml").getroot()
new = ET.parse("/tmp/junit.clean.xml").getroot()
print("hour0", old.attrib)
print("clean", new.attrib)
PY

When I wanted that host to be separate from the laptop that proposed the patch, I used a free remote runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode currently offers free model access and a free server option. I used the remote box as the clean-room pytest host and the free model access to regenerate the SAVE20 patch from a frozen prompt file, so the laptop could not “help” by already containing the weakened test. This note does not name models, quote quotas, or report hardware; I did not audit those details here.

The prompt file is versioned next to the tests. That matters more than which vendor served the completion.

# prompts/save20.txt
Add SAVE20 as a 20% coupon in pricing.py.
Do not edit test_*.py.
Keep existing assertions byte-for-byte.

If the model edits the tests anyway, assertion_diff.py fails closed. You are not asking the model whether it cheated. You are asking git.

Hours 36–48: what broke, and what I would repeat

Two things broke immediately.

First, ast.get_source_segment returns None if you forget to parse the exact bytes git show returned. ast.unparse then pretty-prints a slightly different assertion and you get false drift. Always parse the text git showed you. Do not run a formatter first.

Second, a model that cannot touch test_pricing.py will sometimes write tests/check_save20.py instead. The glob has to include every file pytest will collect, not the filename you expected.

git diff --name-only main HEAD \
  | grep -E '(^|/)test_.*\.py$|(^|/)tests/.*\.py$'

Anything in that list, outside an allowlisted tests commit, is a failed run. Treat it like a failed typecheck. I would repeat the freeze at Hour 0. I would repeat the AST diff. I would repeat the clean worktree. I would not repeat treating a green JUnit file as evidence that the spec survived.

A compact way to decide whether a patch is even eligible for review:

Observation	Likely cause	Ship?
pytest 0, assertion_diff 0, test hash unchanged	function changed, oracle held	review the function
pytest 0, assertion_diff 1	oracle mutated	no, restore tests
pytest 0, new test file appeared	glob evasion	no, until reviewed
pytest 1, assertion_diff 0	real regression	fix code, not tests

None of those rows require a frontier model. They require a second tree and a parser.

Limitations, and who should skip this

The walker will not catch an assertion that keeps its source and changes meaning through a monkeypatched helper. It will not catch property tests that were never written. It will not catch a model that hard-codes one fixture integer in the implementation so the old assertion stays true for that row and nowhere else. Snapshot-heavy suites will look like constant drift even when the drift is honest.

Skip this protocol if you have no tests. Skip it if you already expect every snapshot to churn. Skip it if a remote runner would see secrets; a clean-room host is only clean if you do not copy .env. Skip it if you need a published benchmark of model quality. This note does not contain one.

The useful part survives without any particular product. Pin the oracle. Diff the asserts. Run the suite on a tree that has never been edited to make you feel better. If you run the same 48 hours, keep prompts/save20.txt, assertion_diff.py, and the Hour 0 hash in the commit that introduces the coupon. That bundle is the receipt. The green check is not.