DEV Community

Cover image for 1,558 Tests Green and No Auth: The Tests That Never Actually Ran
Debashish Ghosal
Debashish Ghosal

Posted on AI-assisted

1,558 Tests Green and No Auth: The Tests That Never Actually Ran

#ai #programming #security #testing

A test named test_all_adapters_importable asserted nothing. It would pass forever, even if every adapter was broken.

57 of 65 assertion files were in the wrong format, and the harness returned 0 / 0 without a whisper.

If a green suite makes you relax, this post is going to un-relax you.

Three Times a Green Suite Hid a Real Failure

1. The test gutted to pass. In planner-critic-engine, a test literally had a pass body:

def test_all_adapters_importable():
    # if it imported, it's fine — except this proves nothing
    pass
Enter fullscreen mode Exit fullscreen mode

It was caught in code review, not by CI, and only before the LLM sweep because a human read it. Issue #236.

2. The harness that returned 0 / 0 and called it green. In the same repo, 57 of 65 assertion files were in the wrong format. The harness parsed them, found zero assertions to run, and returned 0 / 0 — which it treated as success. The suite was green because it had executed nothing.

3. The auth guard that never ran. In CauterRule v0.3.0 the suite reported 1,558 tests green. The MCP HTTP bearer-auth guard never ran, because an import was swallowed. Every unit test passed. The only reason we found the unauthenticated store read was a Docker field test running the real thing in a real container.

The Dangerous Thing Wasn't the Bug

In all three cases the bug mattered, but it wasn't the scariest part. The scariest part was the confidence.

We treat "tests pass" as evidence. Sometimes it's evidence that the harness silently skipped a module, or that an import was swallowed, or that a fixture file was parsed into an empty set. The suite stays green and the system ships with an unauthenticated read path.

The Fix Is a Meta-Test, Not More Tests

The fix is not to write more tests. It's to assert that your tests actually ran and actually asserted.

def test_suite_is_not_empty():
    results = run_assertion_files("tests/assertions/")
    assert results.executed > 0, "harness ran zero assertions"
    assert results.assertions > 0, "assertions parsed to empty set"
Enter fullscreen mode Exit fullscreen mode

Two rules fall out of this:

  • CI must fail when a module produces zero results. 0 / 0 is an error state, not a pass.
  • Distinguish "did not run" from "ran and passed." A swallowed import and a skipped module should be loud failures, not silence.

That single distinction is the difference between a flaky gate and a strict one.

The Honest Limitation

Meta-tests add process, and process can rot — a meta-test that stops checking is just another green checkmark. And no amount of test discipline catches the false negatives you never thought to test for. This reduces the class of "green but broken." It does not eliminate it.

But it does close the worst category: the test that never ran and told you everything was fine.

What's the last green build you caught lying to you? I now trust a green suite about as far as I can read its raw output.

Repos and receipts: CauterRule · CauterRule v0.3.0 field test report · PlannerCritic failure-mode register · agent-tooltrust design decisions — all MIT, all public.

Top comments (0)