Postmortem: Zero Tests Collected, Exit Code 0, and a Green Gate

Conclusion first. The patch gate approved a diff because the test step exited 0. The suite had collected zero tests.

That one line is the whole incident. The rest explains how a green result got manufactured, and how the gate now refuses it.

Scope. The failure reproduces from the fixture in this post. Timestamps are relative to merge time. No proprietary logs are needed.

The failure in one command

A selector rename emptied the suite. The runner still reported success.

pytest -q tests -k "patch_gate_contract"
# no tests ran in 0.02s

The exit code was 5. The pytest documentation defines that code as "no tests were collected". The gate wrapper flattened it to a pass.

# gate/run_suite.sh (before)
run_suite() {
  pytest -q "$@" >"$LOG" 2>&1
  rc=$?
  case "$rc" in
    0) echo pass ;;
    5) echo pass ;;   # "nothing to do" — the defect
    *) echo fail ;;
  esac
}

Line 5 is the bug. An empty selection became a positive verdict.

Timeline

• T−6d — The selector string is renamed in gate config. No test references the old name.
• T−0 — A model-generated patch touches one module. The gate starts.
• T+2m — Collection returns zero tests. The wrapper maps code 5 to pass.
• T+3m — The reviewer model reads tests: pass and emits approve.
• T+7m — The merge lands. CI on main runs the full suite and stays green.
• T+19m — A scheduled job on the merged commit fails. The touched module has no coverage.
• T+26m — Revert. The diff is small, so the revert is trivial. The gate stays wrong.
• T+58m — The wrapper is patched to treat code 5 as an error.

The revert took four minutes. Locating the cause took fifty.

Contributing factors

1. Exit codes were treated as a verdict. The wrapper collapsed three states into two.
2. No baseline test count existed. Nobody knew the suite should collect 412 tests.
3. The model was the only reader. It saw pass and had nothing to cross-check.
4. CI on main hid the defect. Full-suite runs never exercised the gate's selector path.

Factor 4 extended detection time by an order of magnitude.

The fix: numbers come from the runner, never the model

The gate now follows a short written contract.

• The runner emits counts. The model never emits counts.
• A verdict without collected and passed is invalid.
• A run with passed == 0 is a hard block.
• The expected test inventory is pinned in the repository.

Step 1: pin the inventory

pytest -q tests --collect-only > gate/test_inventory.txt
wc -l gate/test_inventory.txt
# 412 gate/test_inventory.txt

The file is committed. A selector rename now produces a visible diff in review.

Step 2: enforce it before the model runs

# gate/test_contract.py  (runnable; pytest >= 7)
import re, subprocess, sys
from pathlib import Path

EXPECTED = {
    line.strip()
    for line in Path("gate/test_inventory.txt").read_text().splitlines()
    if "::" in line
}

def sh(cmd):
    p = subprocess.run(cmd, capture_output=True, text=True)
    return p.returncode, p.stdout + p.stderr

def collect(cmd):
    rc, out = sh(cmd + ["--collect-only", "-q"])
    if rc != 0:
        sys.exit(f"BLOCK collection rc={rc}")
    return {ln.strip() for ln in out.splitlines() if "::" in ln}

def passed_count(out):
    m = re.search(r"(\d+) passed", out)
    return int(m.group(1)) if m else 0

def main():
    ids = collect(sys.argv[1:])
    missing = sorted(EXPECTED - ids)
    added = sorted(ids - EXPECTED)
    if missing:
        sys.exit(f"BLOCK {len(missing)} tests missing, e.g. {missing[:3]}")
    if not ids:
        sys.exit("BLOCK collection empty")
    rc, out = sh(sys.argv[1:] + ["-q"])
    if passed_count(out) == 0:
        sys.exit("BLOCK zero tests passed")
    if rc != 0:
        sys.exit(f"BLOCK suite rc={rc}")
    print({"rc": rc, "passed": passed_count(out),
           "collected": len(ids), "added": len(added)})

main()

Usage against the broken selector:

python gate/test_contract.py pytest tests -k "patch_gate_contract"
# BLOCK 412 tests missing, e.g. ['tests/test_gate.py::test_blocks_empty_diff']

The gate now fails before any model call. That saves tokens as well as time.

Step 3: make the verdict carry evidence

{
  "run_id": "run-2026-10-27-041",
  "diff_sha256": "9f2c41ab...",
  "command": "pytest -q tests",
  "collected": 412,
  "passed": 412,
  "failed": 0,
  "skipped": 0,
  "inventory_missing": 0
}

The policy fits in one sentence. A model approve with collected: 0 fails schema validation, not debate.

Verification plan

Run the fixture three ways. The gate must behave identically on every replay.

1. bash repro/zero_collection.sh → blocked, reason collection empty.
2. bash repro/good_patch.sh → allowed, collected=412.
3. bash repro/selector_typo.sh → blocked, reason 412 tests missing.

If step 3 ever prints pass, the contract is broken. The check is cheap enough to run on every commit.

Where a free tier fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The replay harness is the expensive part of this workflow. Every contract change needs re-running against past patches. MonkeyCode's operator reports free model access and a free server option, which makes that replay cheap. The same operator reports a free-tier token allowance in the millions; that figure is unverified here and should be read from the project's own documentation.

A second-opinion reviewer is the natural first use. The gate stays deterministic. The model only argues against a verdict.

# replay/second_opinion.py  (pseudocode, not executed in this post)
import json, os, urllib.request

BASE = os.environ["MONKEYCODE_BASE_URL"]  # value from project docs
KEY = os.environ["MONKEYCODE_API_KEY"]

def ask(payload):
    req = urllib.request.Request(
        f"{BASE}/chat/completions",
        data=json.dumps(payload).encode(),
        headers={"Authorization": f"Bearer {KEY}",
                 "Content-Type": "application/json"},
    )
    with urllib.request.urlopen(req, timeout=60) as r:
        return json.load(r)

Two rules keep this honest.

• The second opinion may subtract. It can block a patch, never approve one alone.
• Every response is logged with run_id and diff_sha256 for later replay.

A free server is fine for the harness itself. Keep it off the approval path until its failure behavior is measured.

Limitations and who should skip this

• Free tiers have rate limits and no uptime guarantee. Budget for retries and backoff.
• A pinned inventory rots. Regenerate it whenever tests are intentionally added or removed.
• The contract checks presence, not quality. Deleting an assertion preserves the test ID.
• Teams without a committed inventory should fix that before touching the model layer.
• Regulated environments needing dedicated capacity and SLAs should not route approvals through a shared free tier.
• Gates requiring sub-second latency should keep the model out of the blocking path entirely.

Checklist

1. Store the collected test inventory in the repository.
2. Block any run where passed == 0.
3. Reject exit code 5 explicitly, in code and in tests.
4. Validate verdict schema before reading prose.
5. Replay three fixtures after every contract change.
6. Log run_id and diff_sha256 with each verdict.

The original incident cost one uncovered merge and fifty minutes. The contract costs one file and twelve lines of Python.

If the replay harness needs a cheap place to run before dedicated capacity is justified, MonkeyCode's free tier is a reasonable starting point.