A Skipped Test Is Not a Passed Test

TL;DR: Exit code zero does not prove required tests ran; compare structured outcomes against a frozen manifest and fail on missing or undeclared results. The parent slice log documents why.

Your test command exited zero. Which required tests actually ran? If your answer is “the job was green,” you have a hole in the gate. Exit-code satisfaction allowed a skipped test, or a vanished one, to read as success.

In this note

• Zero exited cleanly while the suite got smaller
• Judge outcomes against a frozen manifest
• Go hunt for disappearance in your pipeline
• What the proof covers and where it stops
• Make your next green run account for every test

The measured failure was not theoretical. Twenty-seven tests were destroyed while the remainder stayed green. The gate had no way to distinguish that partial absence from a healthy run.

A passing exit code is a promise about process completion. It is not a complete account of the test outcomes you needed.

Zero exited cleanly while the suite got smaller

Before SLICE-009, Ranex accepted exit_code == 0 and nothing else for its tests-executed claim. A test that asserted False, then received a skip marker, reported one skipped test, exited zero, and satisfied the claim.

Pytest refuses a suite with total absence through exit code 5. Partial absence is different. A suite can skip itself into silence or shrink without changing its exit code.

That exact shape had already happened: two agents in one worktree destroyed 27 tests, and the remainder stayed green. The problem was not that pytest lied. It returned the process status it was designed to return. The gate asked that status to prove more than it could prove.

An exit code is a promise about nothing your gate has defined. Until you define the required outcomes, zero only tells you that the command chose zero.

This is broader than agents. A changed test selection, a conditional skip, an environment difference, or a deleted file can all turn your familiar green badge into a smaller test run. A test that runs and sometimes fails is a different disease; that one is worth a read too. If the only evidence you retain is a status code, you cannot compare what ran with what should have run.

Judge outcomes against a frozen manifest

The fix is to bind structured test outcomes into the evidence and compare them against a manifest frozen from the suite. SLICE-009 binds junitxml into the digest-bound command argv, then signs structured outcomes as evidence v3.

The signed summary includes outcome counts, sorted non-passed IDs with their kinds, and a full-outcome digest. The artifact is read from the hermetic sample before teardown. An older evidence v2 row is refused loudly rather than treated as a skip.

The manifest is generated by a freeze ceremony from a hermetic run’s junitxml and committed beside the gate catalog. It is outcome-blind: it records the expected test IDs, not a convenient claim that every test passed. Expected skips are declared by ID with a reason.

Then the manifest diff becomes the rule.

• Every manifest ID must appear as passed.
• A declared expected-skip may skip or pass.
• An undeclared skip blocks.
• An xfail, xpass, error, or missing ID blocks.
• An absent, unparseable, oversized, or duplicate-ID artifact blocks.
• Extra IDs neither satisfy nor block the claim.

This is not a parser trick. The slice requires a real pytest run to produce the artifact, then freezes and judges that same artifact through the same path. A handwritten XML string can exercise a parser while proving nothing about the pipeline that produced it.

The manifest also comes from the dispatch-time base tree during delegated judging, never the candidate. A candidate that can edit the measuring stick cannot be expected to judge itself strictly.

Go hunt for disappearance in your pipeline

You do not need to build a new kernel to test this idea. Start by asking whether your gate can name every required test and say what happened to each one.

• Run a required test with a skip marker and see whether your release gate still approves the result.
• Delete a required test file in a disposable branch. Confirm the gate fails for the missing test, not only when the suite crashes.
• Add a selection rule that deselects a required test. Check whether the gate notices the missing ID.
• Collect a structured result artifact such as junitxml from the exact command your gate judges.
• Freeze the expected test IDs before the change being judged. Keep declared skips explicit and reasoned.
• Compare outcomes to that frozen list, including skips, xfails, xpasses, errors, and duplicate IDs.
• Keep the manifest outside the candidate’s authority when a delegated worker is being judged.

A test that cannot be found is not a test that passed. That sounds obvious until a zero exit code waves it through.

What the proof covers and where it stops

SLICE-009 proves that an undeclared skip blocks, declared expected skips are explicit permission rather than an obligation, non-passing outcome kinds block, and a missing ID blocks. It also proves this against a deleted test file, the same class of disappearance as the 27-test incident.

Ranex now gates its own repository using this rule. The README says the current gate judges signed structured outcomes against a manifest diff rather than exit code alone. It PASSes only against that manifest and flips to FAIL when a frozen test file is deleted.

The close-out record gives the implementation’s then-current measurement: a 736-ID manifest, 67 declared expected skips, and 669 passed in the sealed sample — the manifest has grown since; governance/suite_manifest.json currently carries 943 IDs and 113 expected skips. The source also states why the skipped tests exist: harness-fork, cold-start by design, dependency and provisioning conditions, one OpenRouter credential, and one mount namespace. Those skips were declared at freeze time with reasons. They were not silently accepted.

There is a boundary here. A hostile tree can fabricate an all-pass artifact. The slice has a passing test that states this forgery boundary rather than hiding it. Structured outcomes improve what the exit code could establish; they do not make an untrusted producer truthful.

That is why this belongs with how the kernel works: a verdict is only as strong as the evidence and authority around it. Ranex is pre-release, and its README lists a working verdict path alongside gaps that remain designed rather than built. The source records are available in the Ranex repository under docs/slices/done/.

Questions people actually ask

These questions help you inspect skipped and missing tests before a green status approves them.

Why is exit code zero not enough for a test gate?

A skipped or vanished test can leave pytest exiting zero, so exit-code satisfaction alone cannot show that required tests ran.

How can a test gate detect skipped or missing tests?

A gate can compare signed structured junitxml outcomes with an outcome-blind manifest frozen from the suite and block missing IDs.

What happens when a frozen test file is deleted?

Ranex’s own gate flips from PASS to FAIL and names the missing test ID.

Make your next green run account for every test

Take one required test file and delete it in a disposable branch. If your gate stays green, stop treating that gate as proof that the suite ran. Bind a structured outcome artifact to the command, freeze the required IDs, and compare the result against that list.

Try it. Break it. Tell me what broke. Star Ranex on GitHub if this gives you a sharper test gate, and send the honest critique. The missing test your pipeline catches next is the point.

Disclosure: this post was drafted with AI assistance. Every factual claim traces to the repository’s README or slice records, the same fact gate the product enforces on code. It ships only after Anthony’s own review.