DEV Community

Emery Yang
Emery Yang

Posted on

Empty Stdout Is a Kill: A 90-Minute Agent Spike

#ai #testing #agents #python

An agent green with empty stdout is false evidence. Kill the run and refuse the merge. A ninety-minute spike can encode that rule.

The artifact is a fail-closed log gate. Empty logs never count as execution proof here.

Hypothesis

One hypothesis drives this whole timed spike. Remote success requires three strict machine-checked result fields. Missing any field kills the whole agent run.

The three required fields stay small and strict.

  • The exit_code field must equal integer zero.
  • The stdout field must hold non-whitespace text.
  • The log_sha256 field must match stdout bytes.

No narrative summary can replace those fields. A model sentence is not a test log.

Why empty green appears

Agents often emit compact success-shaped JSON result objects. Wrappers often swallow the real runner output. Several common paths still produce a fake pass.

  • The model describes tests it never launched.
  • The runner exits zero without executing files.
  • Stdout is trimmed down to an empty string.
  • A skipped step still returns an ok flag.
  • Remote checkout hash drifts from local HEAD.

Each path looks green in a chat transcript. None of those paths prove actual execution.

Spike rules

Time-box all spike work at ninety minutes. Keep one hypothesis and no extra feature work.

Ship criteria for this spike stay narrow.

  • Four fixtures run without later hand-edit passes.
  • The decision table lists kill versus ship.
  • The gate fails closed on empty stdout.

Kill criteria for this spike are also narrow.

  • These fixtures need more than ninety minutes total.
  • The hash check depends on hidden local state.
  • The gate accepts whitespace-only fake logs here.

Clock

Work the clock in four hard blocks.

  1. Minutes 0-15 freeze the hypothesis and kill list.
  2. Minutes 15-40 implement gate.py and the fixtures.
  3. Minutes 40-70 execute four cases and record outputs.
  4. Minutes 70-90 freeze the decision table, then stop.

Do not extend the clock after ninety minutes. Overtime means this spike has already failed.

Artifact: gate.py

Label the next file as proposed local code. It is not a production control plane yet. Save it as gate.py in the spike repo. Run it against captured runner JSON files.

#!/usr/bin/env python3
"""Fail closed when a test runner returns empty stdout."""
from __future__ import annotations

import hashlib
import json
import sys
from pathlib import Path

MIN_STDOUT_CHARS = 24


def sha256_text(text: str) -> str:
    return hashlib.sha256(text.encode("utf-8")).hexdigest()


def evaluate(payload: dict) -> dict:
    exit_code = payload.get("exit_code")
    stdout = payload.get("stdout")
    claimed = payload.get("log_sha256")
    reasons: list[str] = []

    if not isinstance(exit_code, int):
        reasons.append("exit_code_missing")
    elif exit_code != 0:
        reasons.append(f"exit_code_{exit_code}")

    if not isinstance(stdout, str):
        reasons.append("stdout_missing")
        stdout = ""

    stripped = stdout.strip()
    if len(stripped) < MIN_STDOUT_CHARS:
        reasons.append("stdout_too_short")

    digest = sha256_text(stdout) if isinstance(stdout, str) else ""
    if not isinstance(claimed, str) or claimed != digest:
        reasons.append("log_hash_mismatch")

    status = "kill" if reasons else "ship"
    return {
        "status": status,
        "reasons": reasons,
        "log_sha256": digest,
        "stdout_chars": len(stripped),
    }


def main() -> int:
    if len(sys.argv) != 2:
        print("usage: gate.py result.json", file=sys.stderr)
        return 2
    payload = json.loads(Path(sys.argv[1]).read_text())
    verdict = evaluate(payload)
    print(json.dumps(verdict, indent=2))
    return 0 if verdict["status"] == "ship" else 1


if __name__ == "__main__":
    raise SystemExit(main())
Enter fullscreen mode Exit fullscreen mode

The script never trusts a success boolean. It ignores chat text from the coding agent. Only exit code, stdout, and hash ever matter.

Add a tiny unit check if minutes remain. Label it as unexecuted until you run it. Save the file as gate_test.py beside gate.py.

from gate import evaluate

EMPTY_SHA = "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"


def test_empty_stdout_kills():
    verdict = evaluate(
        {
            "exit_code": 0,
            "stdout": "",
            "log_sha256": EMPTY_SHA,
        }
    )
    assert verdict["status"] == "kill"
    assert "stdout_too_short" in verdict["reasons"]
Enter fullscreen mode Exit fullscreen mode 
python3 -m pytest -q gate_test.py
Enter fullscreen mode Exit fullscreen mode

Fixtures

Create a fixtures directory right beside gate.py now. Keep each captured case under twenty lines total.

empty_ok.json

This case is the core kill path.

{
  "exit_code": 0,
  "stdout": "",
  "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855"
}
Enter fullscreen mode Exit fullscreen mode

That hash is SHA-256 of an empty string. The gate must still kill this case. Empty content is not real execution evidence here.

nonzero.json

A real failure must still stay a kill.

{
  "exit_code": 1,
  "stdout": "FAILED tests/test_health.py::test_ready - assert 503 == 200\n",
  "log_sha256": "replace-me"
}
Enter fullscreen mode Exit fullscreen mode

Compute the hash before you run the gate.

python3 - <<'PY'
import hashlib, json
from pathlib import Path
p = Path("fixtures/nonzero.json")
data = json.loads(p.read_text())
data["log_sha256"] = hashlib.sha256(data["stdout"].encode()).hexdigest()
p.write_text(json.dumps(data, indent=2) + "\n")
print(data["log_sha256"])
PY
Enter fullscreen mode Exit fullscreen mode

hash_mismatch.json

A pretty log with a bad hash is kill.

{
  "exit_code": 0,
  "stdout": "======================= 3 passed in 0.12s =======================\n",
  "log_sha256": "deadbeef"
}
Enter fullscreen mode Exit fullscreen mode

real_pass.json

Only this last case may return ship.

{
  "exit_code": 0,
  "stdout": "======================= 3 passed in 0.12s =======================\n",
  "log_sha256": "replace-me"
}
Enter fullscreen mode Exit fullscreen mode

Hash real_pass.json with the same snippet first. Then run the four commands in order.

python3 gate.py fixtures/empty_ok.json; echo $?
python3 gate.py fixtures/nonzero.json; echo $?
python3 gate.py fixtures/hash_mismatch.json; echo $?
python3 gate.py fixtures/real_pass.json; echo $?
Enter fullscreen mode Exit fullscreen mode

Expected process exits are 1, 1, 1, and 0. Record the JSON verdicts in the spike note.

How to read a verdict

A kill verdict always includes a reasons list. Treat any listed reason as sufficient for reject.

{
  "status": "kill",
  "reasons": ["stdout_too_short"],
  "log_sha256": "e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855",
  "stdout_chars": 0
}
Enter fullscreen mode Exit fullscreen mode

The stdout_too_short reason beats a matching empty hash. Do not special-case the empty-string digest here.

Decision table

Read this table as ship-or-kill spike evidence. Do not argue with a failing row.

Case exit_code stdout hash Verdict
Empty ok envelope 0 empty hash of empty kill
Failing tests 1 failure log matching kill
Green log, wrong hash 0 pytest summary deadbeef kill
Green log, matching hash 0 pytest summary matching ship

Ship the gate only if all four hold. One wrong row kills the whole spike.

Wiring a remote runner

The gate does not care who ran pytest. It only cares that bytes came back.

A free remote server is enough here. You need a checkout and a test command. You also need captured stdout and exit_code.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode offers free model access and a free server option. Those two pieces can host the runner. Do not treat them as signed CI fabric. This article does not claim quotas or uptime. Use the free server option only as the runner host.

A minimal remote shape can look like this.

# Label: proposed commands, not a vendor SLA.
git archive HEAD | ssh runner 'mkdir -p /tmp/spike && cd /tmp/spike && tar xf - && python -m pytest -q'
Enter fullscreen mode Exit fullscreen mode

Capture stdout and the process exit code. Hash the stdout bytes, then feed gate.py. Blank output with exit zero is still kill.

The agent may draft the patch under test. The runner must still speak in logs. The model does not veto a log kill.

Failure analysis

Walk empty_ok before you trust real_pass. The envelope looks successful at first glance.

The exit_code value is zero and looks healthy. The log_sha256 value matches the empty-string digest. The stdout field has zero characters after strip.

The gate should still emit stdout_too_short now. That single reason is the whole point. Chat transcripts will hide this exact case.

Walk hash_mismatch next if time remains. The log text can look like pytest output. A deadbeef digest means the bytes were swapped. Kill that case even when tests sound green.

What this spike does not prove

This gate does not prove test quality. It does not prove hermetic or reproducible builds. It does not prove the runner's machine identity.

Whitespace padding can still cheat the length check. Raise MIN_STDOUT_CHARS if your suite is noisy. Prefer a required token such as passed.

The log hash is not a cryptographic signature. Anyone who can write JSON can hash stdout. Pair this later with an allowlisted host. That pairing is out of scope today.

Who should not use this

Skip this spike in each situation below.

  • Your team already runs attested CI today.
  • Your release process needs signed build provenance.
  • Your test suite legitimately prints no stdout.
  • Secrets cannot touch a shared remote server.
  • Jobs routinely run longer than ninety minutes.

If tests are silent by design, stop. The hypothesis is wrong for that suite. Pick another artifact such as JUnit XML files.

Spike note template

Write this block after minute seventy ends. Keep the written spike note short and boring.

hypothesis: empty stdout => kill
clock: 90 minutes
fixtures: empty_ok, nonzero, hash_mismatch, real_pass
gate_exits: 1,1,1,0
decision: ship | kill
reason:
Enter fullscreen mode Exit fullscreen mode

Fill the decision field with one word. If fixtures drifted, write kill instead here.

Close

False green is cheaper than a red log. That cheap green is the failure mode. Bound it with exit code, bytes, and hash.

Keep the gate local and the rules public. Ninety minutes is enough to ship or kill.

Top comments (0)