I Scored Tests That Could Not Fail

Green is cheap when failure is unreachable. That is the whole finding. I stopped treating a generated "all tests passed" line as evidence until I could score whether a red result was even in the reachable set.

The timeline is loud again. Is AI already better at coding than most of us? Depends which benchmark someone is defending this week. I have a smaller question, and it actually ships. When a suite prints passed, was a real failure even possible?

Vibe coding is not the interesting insult. Calling a costume suite engineering is. A test that cannot go red is not a test. It is interior decorating for a pull request.

I wanted a number I could recompute on Monday. Not a vibe. A classifier over the test file plus a pytest collect, run on fixtures I can check into git. If the scorer and the labels disagree, the scorer is wrong. That is the contract. Cute narratives do not get a vote.

The failure I kept missing

I used to read the summary first. Passed: 8. Failed: 0. Skipped: 0. Then I would skim the test names like they were a changelog. That is how you get fooled. The model did not lie about pytest. Pytest told the truth about a suite that never asked a hard question.

Think of a smoke alarm that is not wired to the battery. It is green. You can screenshot it. You still do not have a fire system. Vacuous tests are that alarm. assert True is the obvious joke. The expensive ones are quieter. No assertion at all. A patch that replaces the function under test and then "proves" the return value. A function named check_add that pytest will never collect. assert add(2, 2) == add(2, 2), which stays green even when add is a dumpster fire, so long as it is a deterministic dumpster fire.

Would you accept that suite from a junior hire? Then why accept it from a model?

A tiny subject under test

I do not start with a real service. I start with something I can hold in one screen. If the scorer cannot be honest here, it will not be honest in your monorepo either.

# sut.py
def add(a: int, b: int) -> int:
    return a + b


def clamp(n: int, lo: int, hi: int) -> int:
    if lo > hi:
        raise ValueError("lo > hi")
    return max(lo, min(n, hi))

Eight fixtures live next to it. I labeled them before I ran anything. The labels are the experiment, not a memoir.

# fixtures/test_cases.py
import pytest
from unittest.mock import patch
from sut import add, clamp


def test_add_binding():
    assert add(2, 2) == 4
    assert clamp(11, 0, 10) == 10


def test_add_true():
    add(2, 2)
    assert True


def test_add_no_assert():
    add(2, 2)
    clamp(3, 0, 10)


def test_add_mocked():
    with patch("sut.add", return_value=4):
        from sut import add as mocked_add
        assert mocked_add(2, 2) == 4


def check_add_uncollected():
    assert add(1, 1) == 2


def test_always_skip():
    pytest.skip("not today")
    assert add(2, 2) == 4


def test_add_not_none():
    assert add(2, 2) is not None


def test_add_tautology():
    assert add(2, 2) == add(2, 2)

Binding means a wrong add or a wrong clamp can make pytest red. Everything else is a costume with a different tailor.

The scorer, not the speech

I score files, not vibes. AST first. Collection second. The model does not get to grade its own homework. If that sounds unkind, good. Kindness is how vacuous tests survive.

# vacuous_score.py
from __future__ import annotations

import ast
import json
import subprocess
import sys
from pathlib import Path

WEAK_OPS = {ast.Is, ast.IsNot, ast.NotEq}
SKIP_NAMES = {"skip", "xfail", "importorskip"}


class TestFn:
    def __init__(self, name: str, node: ast.FunctionDef):
        self.name = name
        self.node = node
        self.asserts = []
        self.skipped = False
        self.patches_sut = False
        self.calls_sut = False


class Visitor(ast.NodeVisitor):
    def __init__(self, sut_names: set[str]):
        self.sut_names = sut_names
        self.tests: list[TestFn] = []
        self._cur: TestFn | None = None

    def visit_FunctionDef(self, node: ast.FunctionDef):
        prev, self._cur = self._cur, TestFn(node.name, node)
        self.generic_visit(node)
        self.tests.append(self._cur)
        self._cur = prev

    def visit_Assert(self, node: ast.Assert):
        if self._cur:
            self._cur.asserts.append(node)
        self.generic_visit(node)

    def visit_Call(self, node: ast.Call):
        if not self._cur:
            return self.generic_visit(node)
        fn = node.func
        name = fn.attr if isinstance(fn, ast.Attribute) else getattr(fn, "id", "")
        if name in SKIP_NAMES:
            self._cur.skipped = True
        if name == "patch":
            if node.args and isinstance(node.args[0], ast.Constant):
                val = str(node.args[0].value)
                if any(s in val for s in self.sut_names):
                    self._cur.patches_sut = True
        if name in self.sut_names:
            self._cur.calls_sut = True
        self.generic_visit(node)


def _is_true_const(n: ast.AST) -> bool:
    return isinstance(n, ast.Constant) and n.value is True


def _is_tautology(n: ast.AST) -> bool:
    return (
        isinstance(n, ast.Compare)
        and len(n.comparators) == 1
        and ast.dump(n.left, annotate_fields=False)
        == ast.dump(n.comparators[0], annotate_fields=False)
    )


def _is_weak(n: ast.AST) -> bool:
    if not isinstance(n, ast.Compare):
        return False
    if any(isinstance(op, (ast.Is, ast.IsNot)) for op in n.ops):
        return True
    return False


def classify(t: TestFn) -> str:
    if not t.name.startswith("test_"):
        return "uncollected"
    if t.skipped:
        return "skipped"
    if t.patches_sut:
        return "mocked_sut"
    if not t.asserts:
        return "no_oracle"
    tests = [a.test for a in t.asserts]
    if tests and all(_is_true_const(x) for x in tests):
        return "vacuous"
    if tests and all(_is_tautology(x) for x in tests):
        return "tautology"
    if tests and all(_is_weak(x) for x in tests):
        return "weak_oracle"
    if t.calls_sut:
        return "binding"
    return "no_oracle"


def collect_names(path: Path) -> set[str]:
    r = subprocess.run(
        [sys.executable, "-m", "pytest", str(path), "--collect-only", "-q"],
        capture_output=True,
        text=True,
        check=False,
    )
    names = set()
    for line in r.stdout.splitlines():
        if "::" in line:
            names.add(line.split("::")[-1].strip())
    return names


def score(path: Path, sut_names: set[str]) -> dict:
    tree = ast.parse(path.read_text())
    v = Visitor(sut_names)
    v.visit(tree)
    collected = collect_names(path)
    rows = []
    for t in v.tests:
        label = classify(t)
        if t.name.startswith("test_") and t.name not in collected and label != "skipped":
            label = "uncollected"
        rows.append({"name": t.name, "label": label})
    return {"file": str(path), "rows": rows}


if __name__ == "__main__":
    target = Path(sys.argv[1])
    print(json.dumps(score(target, {"add", "clamp"}), indent=2))

Run it like a build step, not like a blog claim.

python vacuous_score.py fixtures/test_cases.py
pytest fixtures/test_cases.py -q --tb=no

On these fixtures the scorer is designed to emit this matrix. I am not reporting a hidden production run. I am reporting the labels the fixtures were built to force, which is the only honest number I have in this article.

test_add_binding          binding
test_add_true             vacuous
test_add_no_assert        no_oracle
test_add_mocked           mocked_sut
check_add_uncollected     uncollected
test_always_skip          skipped
test_add_not_none         weak_oracle
test_add_tautology        tautology

Pytest on the same file can still look fine. Binding tests pass. Vacuous tests pass. The skip is a skip. The uncollected function is invisible. That is the trick. The runner is not the oracle for quality. It is the oracle for "did the collected nodes raise."

So I keep a gate that does not care about charm.

FAIL_LABELS = {"vacuous", "no_oracle", "mocked_sut", "uncollected", "tautology"}
WEAK_OK = 0  # I do not let weak oracles count as coverage


def gate(rows: list[dict]) -> int:
    labels = [r["label"] for r in rows]
    binding = labels.count("binding")
    weak = labels.count("weak_oracle")
    bad = sum(1 for x in labels if x in FAIL_LABELS)
    if binding < 1:
        return 2
    if bad:
        return 2
    if weak > WEAK_OK:
        return 1
    return 0

Exit 2 means I do not ship the suite. Exit 1 is a warning I still refuse in CI. Zero is not "correct code." Zero is "these tests could fail." That bar is low. It is still higher than a screenshot of green.

Where a free model and a free server actually help

I still let a model propose the first draft of tests. I do not let it propose the verdict. When I want that loop on a scratch box instead of my laptop, I use MonkeyCode's free model access and the free server option to generate a candidate suite and run pytest there. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The scorer stays local. It is boring Python. If you peel the product out of this article, the gate still works. That is how I know the article is not a brochure.

What is the free path good at? Drafting volume. Getting a file to exist. Giving pytest something to collect so I can score it the same day. What does it break on? It cannot tell you the tests are binding. It cannot promise the server is a reproduction of prod. I am not going to invent model names, quotas, or latency numbers I did not measure in this writeup. If a vendor card will not fit in git next to vacuous_score.py, it is not part of the experiment.

I treat the server like a microwave. Useful. Not a laboratory. If the generated file trips mocked_sut or uncollected, I throw the file back. I do not "fix" it by weakening the gate. Have you watched someone delete an assertion to get green. Same energy. Different autocomplete.

What this does not prove

This scorer is a tripwire, not mutation testing. It will miss clever vacuity. A test can call add and still only check a log side effect you never cared about. Parametrize, dynamic exec, and unittest-style methods need more visitors. Collection depends on pytest's default naming. If you generate tests in a helper that pytest never imports, you can still look clean.

It also does not prove sut.py is right. Binding tests can encode the same bug twice. assert add(2, 2) == 5 is binding and wrong. Different gate. I want that to fail in pytest, not in the vacuity classifier.

Who should not use this as a primary control? Anyone in payments, safety, or an audit trail that needs fault injection. Anyone who thinks a free generation loop is a substitute for a maintainer reading the assertions. Anyone scoring models by how confident the summary paragraph sounds. If you need kill scores, use mutation testing. If you need this article to invent a leaderboard, you are in the wrong tab.

I keep the rule small enough to remember. A suite is evidence only if a wrong function can make it scream. Everything else is costume jewelry on a passing build. Score the jewelry. Then maybe read the summary.

If you want a scratch box to draft a suite and immediately run this gate, MonkeyCode's free model access and free server option are enough to host the loop. Do not confuse that with a verdict. The verdict is the JSON.