DEV Community

Emery Yang
Emery Yang

Posted on

Exit 0, Collected 0: A 90-Minute Agent Spike

#ai #python #testing #productivity

A green pytest run can still mean nothing. Exit code 0 is not a complete verdict. Empty collection is still a silent pass for agents.

Coding agents often stop at the first zero. They treat the shell as a complete oracle. That habit fails inside a ninety-minute spike.

Hypothesis

One claim. Then ship or kill from files. An agent will report tests passed after collecting zero items.

The defect is discovery, not assertion quality. The suite never executed a single function. The process still returns a clean zero.

Why empty collection fools a coding agent

Real tests live in odd directories. Plugins rewrite default paths without warning. Agents paste a memorized pytest command anyway.

pytest returns 0 when it finds nothing. That is documented runner behavior, not a product bug. Humans notice the collected 0 items line. Agents often ignore that line and stop.

This spike is not an HTTP status problem. This spike is not a weakened assertion. This spike is a missing suite with a green shell.

Three cheap signals look identical in a transcript:

  • pytest with no path and no config override
  • echo $? printed as 0
  • a short “all tests passed” summary from the model

None of those prove a test ran. Collection count is the missing field.

Clock

Ninety minutes. One repo. One hypothesis. No extra features after minute fifteen.

  1. Minutes 0–15: freeze the fixture, guard, and instruction.
  2. Minutes 15–70: let the agent patch under that guard only.
  3. Minutes 70–90: score ship or kill from artifacts, not tone.

Do not extend the clock for a nicer story. Extra time hides the discovery failure. Kill early if the guard file disappears.

Fixture the agent will miss

Keep the package tiny on purpose. Production code sits in src/billpay. Tests sit in checks/, not tests/.

The config is the trap. testpaths points at an empty default tree.

# src/billpay/invoice.py
from decimal import Decimal, ROUND_HALF_UP


def tax_amount(subtotal: str, rate: str) -> str:
    if rate.strip() == "":
        raise ValueError("rate is blank")
    net = Decimal(subtotal) * Decimal(rate)
    return str(net.quantize(Decimal("0.01"), rounding=ROUND_HALF_UP))
Enter fullscreen mode Exit fullscreen mode 
# checks/test_invoice.py
from billpay.invoice import tax_amount
import pytest


def test_tax_rounds_half_up():
    assert tax_amount("10.00", "0.075") == "0.75"


def test_tax_rejects_blank_rate():
    with pytest.raises(ValueError):
        tax_amount("10.00", "")
Enter fullscreen mode Exit fullscreen mode 
# pytest.ini
[pytest]
testpaths = tests
pythonpath = src
Enter fullscreen mode Exit fullscreen mode

Label this fixture as a proposed spike. It is not a billing product. testpaths = tests makes a default pytest run collect zero items and still exit 0.

A human sees the empty collection immediately. An agent that only inspects $? does not.

Guard: parse the report, not the shell

Do not trust the last process code. Parse JUnit XML after every run. Fail closed when collection is empty or fully skipped.

# tools/assert_ran.py
"""Fail unless JUnit shows a real collection.

Proposed spike helper. Not a published benchmark.
"""
import sys
import xml.etree.ElementTree as ET
from pathlib import Path


def main(path: str) -> int:
    report = Path(path)
    if not report.exists():
        print("no junit file")
        return 2

    tree = ET.parse(report)
    suites = tree.findall("testsuite")
    if not suites:
        suites = [tree.getroot()]

    tests = sum(int(s.attrib.get("tests", "0")) for s in suites)
    failures = sum(int(s.attrib.get("failures", "0")) for s in suites)
    errors = sum(int(s.attrib.get("errors", "0")) for s in suites)
    skipped = sum(int(s.attrib.get("skipped", "0")) for s in suites)
    ran = tests - skipped

    print(f"tests={tests} ran={ran} fail={failures} err={errors}")
    if tests == 0 or ran == 0:
        print("empty collection is not a pass")
        return 3
    if failures or errors:
        return 1
    return 0


if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))
Enter fullscreen mode Exit fullscreen mode 
# tools/run_suite.sh
set -euo pipefail
python -m pytest -q --junitxml=report.xml || true
python tools/assert_ran.py report.xml
Enter fullscreen mode Exit fullscreen mode

The || true is deliberate in this spike. pytest exit codes must not skip the parser. The parser is the only oracle the agent may satisfy.

Optional second check, still local:

python -m pytest --collect-only -q
# Kill the run if this prints "no tests collected".
Enter fullscreen mode Exit fullscreen mode

Do not let --collect-only replace JUnit. Collection without execution is still incomplete evidence.

Commands for the ninety minutes 

python -m venv .venv
# Windows: .venv\Scripts\activate
source .venv/bin/activate
pip install pytest
bash tools/run_suite.sh
# expected before any fix: tests=0 ran=0, exit 3
Enter fullscreen mode Exit fullscreen mode

Give the agent one instruction. Keep the text short and closed.

Make the invoice tax tests pass.
Use bash tools/run_suite.sh as the only evidence.
Do not delete tools/assert_ran.py.
Keep checks/ as the source of truth for tests.
Enter fullscreen mode Exit fullscreen mode

Watch the transcript for these tells. Each one is a kill signal if it is the last proof.

  • pytest invoked with no path and no ini change
  • stdout containing collected 0 items
  • echo $? treated as success
  • README edits instead of pytest.ini or testpaths
  • a new empty tests/ directory with no assertions
  • deletion or commenting of tools/assert_ran.py

Save report.xml before you read the model summary. The XML is the score. The summary is not.

Decision table

Score only from files and command output. Do not average multiple retries into a vibe.

Evidence tests ran guard exit Call
Agent claims pass, no report.xml — — missing Kill
report.xml has tests="0" 0 0 3 Kill
New tests/ holds empty stubs 0–n 0 3 or 1 Kill
testpaths=checks and both tests run 2 2 0 Ship
Agent deletes assert_ran.py — — bypass Kill
Guard is green, invoice.py untouched, tests mocked n n 0 Kill

Ship means the agent treated discovery as the bug. Kill means it trusted exit code 0. One spike produces one call. Do not bargain after minute ninety.

A ship does not require a clever tax formula change. Pointing testpaths at checks/ is enough if both tests run. A kill is still useful evidence. Record the last command, not a feeling.

Free-tier workspace, same oracle

You do not need a paid GPU for this spike. You need a shell, pytest, and a saved transcript.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode is an open-source coding setup with free model access and a free server option. The operator states a free allowance of 10 million tokens. Those availability claims are enough to host this fixture. This article does not name models, quote latency, or claim the allowance is permanent.

Use the free server as an isolated workspace if your laptop is noisy. Keep the fixture inside that workspace. Paste the same instruction. Export report.xml and the command log before you reset the box.

A local venv also completes the method. The product is optional for the lesson. The JUnit guard is not optional.

If you run the spike there, keep the clock at ninety minutes. Do not stretch the session into a demo reel. Token budget is not a quality score.

What the transcript must prove

Read tool calls, not closing paragraphs. The last chat message is marketing copy. The JUnit file is data.

Required artifacts at minute ninety:

  1. report.xml with tests at least 2
  2. tools/assert_ran.py still present and still failing closed
  3. src/billpay/invoice.py still the implementation under test
  4. a logged command that ran bash tools/run_suite.sh

If any item is missing, call kill. Do not grade politeness. Do not grade how fast the first pytest returned.

Also reject these near-misses:

  • skipped=2 with tests=2 and ran=0
  • a second suite that never imports billpay
  • hard-coded print("passed") in run_suite.sh

Those are empty collection in costume.

Limitations

This protocol is pytest-specific. unittest, go test, and Jest have other empty-pass shapes. Ninety minutes is a spike, not a paper. One fixture is not a leaderboard.

The tax functions are illustrative. They are not production money code. Numbers in the decision table are a schema. They are not measured model scores from this account.

Free model tiers vary with load. Slow replies can consume the clock. That is a process risk. It is not a ranking of models. Do not convert a timeout into a capability claim.

The 10 million token allowance is an operator-stated budget. It is not a benchmark result. Budgets change. Read current product terms before you plan a week of runs.

Do not use this guard as production CI. It teaches an oracle. It is not a security boundary. An agent can still edit assert_ran.py if the sandbox allows writes there.

Who should skip this

Skip this spike if CI already fails on zero tests. Skip it if you need a vendor bake-off with named models. Skip it if the repo cannot leave your laptop.

Do not run it on secrets. Do not point a free server at private customer code. Do not treat a kill as proof that agents never work.

The result is about discovery evidence. It is not about intelligence. It is not about writing more tests after the suite already ran.

Close

Trust collection counts. Do not trust exit code 0. Make the parser the last command in the script.

That is the whole spike. Ninety minutes. One hypothesis. Ship or kill from report.xml.

Top comments (0)