Agent stdout Is Not Your Test Plan

Did your agent print All tests passed again today? Did you merge because the log felt complete? That habit is how fake green sneaks in.

I keep hearing four myths about agent logs. They sound like engineering. They are not a test plan.

This is a myth-busting FAQ. You can delete every product name. The checklist still works.

Why logs became a fake CI

Agents write confident sentences. Humans treat sentences like gates. That is a category error, right?

A log line is a claim. A claim is not a process. Who launched the process you trust?

I want a boring gate. I want it copyable. I want it on a scratch machine.

Myth 1: A printed pass means the suite ran

Have you grepped agent.log for the word passed? I have. It lies in three ways.

The model can narrate a run it skipped. It can paste an old snippet. It can praise intent as if it were output.

What evidence remains then? Only a process you started. Only bytes you captured. Only an exit code you stored.

Corrected mental model: stdout is untrusted speech. Speech is not a run.

You need four fields or you have nothing:

• the argv you actually invoked
• the working directory
• the numeric exit code
• hashes of stdout and stderr

No fourth field? You are reading a story.

Myth 2: Exit code 0 means the change is finished

Zero is one integer. Which contract did it sign?

A formatter can return 0. A skipped pytest session can return 0. A --dry-run flag can return 0. Did you name the contract first?

Ask a sharper question. Did this exact argv fail? That is all zero answers.

Corrected mental model: exit code is narrow. Completeness lives elsewhere. Completeness needs file lists and git votes.

Myth 3: Complete-looking logs mean a clean tree

Logs sit in a buffer. The repo sits on disk. They drift fast.

Can the agent print a diff it never wrote? Yes. Can it write files it never mentioned? Also yes. Which artifact did you review?

Corrected mental model: git status is a vote. git diff --stat is a vote. The chat transcript is not a vote.

If HEAD is unknown, you are not reviewing a change. You are reviewing a vibe.

Myth 4: One green scratch run is portable

You got a clean VM. The command passed once. Will CI share PATH, lockfiles, and OS packages?

A scratch host is a lab. A lab is not memory. Do not ship the lab as proof.

Corrected mental model: a green run is one sample. Samples need a recipe. Recipes need pins you can paste.

The artifact: refuse speech, record votes

I do not want another dashboard. I want a script that ignores English.

Label this as a runnable example. I am not attaching fake timings or fake pass rates.

Save accept_run.py at the repo root. Make it the only gate after an agent session.

#!/usr/bin/env python3
"""Acceptance harness: treat agent speech as untrusted."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
from pathlib import Path

ROOT = Path(os.environ.get("ACCEPT_ROOT", ".")).resolve()
CMD = os.environ.get("ACCEPT_CMD", "python -m pytest -q")
REPORT = Path(os.environ.get("ACCEPT_REPORT", "accept_report.json"))


def run(cmd: str, cwd: Path) -> dict:
    proc = subprocess.run(
        cmd,
        cwd=cwd,
        shell=True,
        text=True,
        capture_output=True,
    )
    return {
        "cmd": cmd,
        "cwd": str(cwd),
        "exit_code": proc.returncode,
        "stdout_sha256": hashlib.sha256(proc.stdout.encode()).hexdigest(),
        "stderr_sha256": hashlib.sha256(proc.stderr.encode()).hexdigest(),
        "stdout_bytes": len(proc.stdout.encode()),
        "stderr_bytes": len(proc.stderr.encode()),
    }


def git(args: list[str]) -> str:
    proc = subprocess.run(
        ["git", *args],
        cwd=ROOT,
        text=True,
        capture_output=True,
        check=False,
    )
    return proc.stdout.strip()


def tree_votes() -> dict:
    diff = git(["diff", "HEAD"])
    return {
        "head": git(["rev-parse", "HEAD"]),
        "status_porcelain": git(["status", "--porcelain"]),
        "diff_stat": git(["diff", "--stat"]),
        "diff_sha256": hashlib.sha256(diff.encode()).hexdigest(),
    }


def decide(run_info: dict, tree: dict) -> list[str]:
    failures: list[str] = []
    if run_info["exit_code"] != 0:
        failures.append("command_failed")
    if not tree["head"]:
        failures.append("not_a_git_repo")
    if tree["status_porcelain"]:
        failures.append("working_tree_dirty")
    if run_info["stdout_bytes"] == 0 and run_info["stderr_bytes"] == 0:
        failures.append("empty_stdio")
    return failures


def main() -> int:
    run_info = run(CMD, ROOT)
    tree = tree_votes()
    failures = decide(run_info, tree)
    report = {
        "run": run_info,
        "tree": tree,
        "failures": failures,
        "accepted": failures == [],
    }
    REPORT.write_text(json.dumps(report, indent=2) + "\n")
    print(json.dumps(report, indent=2))
    return 0 if report["accepted"] else 1


if __name__ == "__main__":
    sys.exit(main())

Wrap it if you want a human summary:

#!/usr/bin/env bash
set -euo pipefail
export ACCEPT_CMD="${ACCEPT_CMD:-python -m pytest -q}"
python3 accept_run.py
python3 - <<'PY'
import json
from pathlib import Path
rep = json.loads(Path("accept_report.json").read_text())
print("accepted:", rep["accepted"])
print("failures:", ", ".join(rep["failures"]) or "(none)")
print("head:", (rep["tree"]["head"] or "n/a")[:12])
print("dirty:", bool(rep["tree"]["status_porcelain"]))
PY

Want dirt to fail closed? Keep working_tree_dirty. Want intentional edits? Drop that flag. Make the policy explicit. Do not hide it in chat.

Wrong gate, for contrast:

# Do not do this.
if grep -qi 'pass' agent.log; then
  echo ship_it
fi

That grep is how myth 1 becomes a merge.

A labeled false-green test also helps teach the team:

# labeled example: pytest can exit 0 while proving nothing
def test_placeholder():
    print("All tests passed")
    assert True

Run pytest. Watch exit code 0. Then run the harness. Notice how little you proved besides zero. That is the lesson.

Decision table: what each signal proves

Signal	Proves	Does not prove
Agent said "passed"	The model emitted text	Any process ran
Exit code 0	That argv did not fail	The suite was complete
stdout hash	Bytes were captured	Bytes are correct
git status --porcelain	Disk disagrees with HEAD	The diff is good
git diff --stat	Size of the delta	Intent of the delta
Empty stdio	You captured nothing	Silence is success
Green on a scratch host	One sample existed	CI will match

Argue with the table. Stop arguing with the chatbot.

A workflow that keeps the model off the grade

1. Start from a known HEAD.
2. Let the agent work on a branch.
3. Ignore victory speech in the log.
4. Run ACCEPT_CMD='python -m pytest -q' python3 accept_run.py.
5. Read failures in accept_report.json.
6. Only then inspect git diff.

Need a cheap lab for that loop? Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are the only product facts I rely on here. That pair is a scratch lab for rehearsing the harness. The model may draft commands. The harness still grades the run. Do not let the speaker mark its own exam.

How to read the JSON without lying to yourself

Open accept_report.json. Read failures first. Then read exit_code. Then read status_porcelain.

If accepted is true and you never looked at the diff, you skipped a vote. The script cannot review intent. You still can.

If empty_stdio is present, your command printed nothing. Silence is not success. Fix the command. Do not praise the agent.

If working_tree_dirty fires on every good session, change the policy. Record dirt as a flag. Review it. Do not hide policy in a prompt.

Compare two reports only after you pin ACCEPT_CMD. Different argv means different contracts. Hash mismatches then prove nothing useful.

What this will not catch

This harness is small on purpose. It is not a security scanner. It is not a flake oracle.

It will miss:

• tests skipped by a bad glob
• mocks that hide production failures
• secrets echoed into captured logs
• hashes that churn because of timestamps
• tools installed outside the working tree

Nondeterministic stdout will fight hash comparison. Pin seeds. Pin locale. Or drop cross-host hash equality. Keep exit code plus git votes.

Who should not use this

Skip this if you need a signed audit trail. Skip this if tests cannot run headless. Skip this if a shared host would see real secrets.

Do not copy .env onto a free server. Do not call a lab gate production verification. If CI already stores these four votes, do not duplicate the ritual.

FAQ

Q: Can I parse the agent's English instead?

No. English is not a schema. Emit JSON from your process.

Q: Is a screenshot of the run enough?

Did it show cwd, argv, and exit code? Usually it did not.

Q: May the agent write the harness?

It can draft. You freeze the policy. Then you run it.

Q: What if a dirty tree is expected?

Then dirty is a flag, not a failure. Review the diff as its own vote.

Q: Does a scratch host make the result official?

No. It makes the sample cheap. Official still means CI plus pins.

Limitations, said plainly

I did not benchmark hosts. I did not name models. I did not promise quotas, hardware, or uptime.

A free server can vanish. A free model can be wrong. Your harness must remain copyable.

Remove every product mention and the method should still stand. If it does not, I wrote a brochure. I tried not to.

Try the harness on a throwaway branch. Paste a redacted failures list if you want a second pair of eyes. Bring votes. Leave the victory speech behind.