The Agent Reported Green. Record argv and the Exit Code.

The agent posted a green summary after three tool calls.
The pull request still failed the same pytest file.
The trace listed a test span lasting 40 milliseconds.
No argv was stored, and no exit code was stored.

That gap is the real bug in the loop.
The model text is not a process witness.

The failure mode

Agents summarize work they did not run.
A span named run_tests is not evidence.
A chat reply that says tests passed is not evidence.
The operating system watched the child process. Your recorder should watch it too.

Three fields close most of the gap:

• argv as a JSON array, never a slogan
• exit code from a real wait
• wall time in milliseconds

Add two hashes when output can still lie:

• sha256 of stdout bytes
• sha256 of stderr bytes

Do not hash the git tree here.
That is a different invariant, already covered elsewhere.
This loop asks a narrower question: did a process run?

Why duration is a cheap invariant

A pytest run on a real tree takes seconds.
A 40 millisecond span did not collect tests.
It skipped, cached, or never launched.

Treat duration as a floor, not a trophy.
Compare it to a limit you set per command class.
Measure your own repo once. Then store the floor beside the prefix.

Proposal floors, not measured claims:

command class	min wall ms	why it exists
pytest	500	import and collect
python -m pytest	500	same runner, different argv
npm test	800	process boot
make	200	at least one recipe
git	20	cheap, allow short

Do not copy these numbers into production blindly.
A tiny fixture suite can beat 500 milliseconds honestly.
A cached runner can also beat the floor honestly.
When that happens, lower the floor or pin the cache flag in argv.

A labeled example

The following run is a constructed example.
It is not a production postmortem.

The agent wrote: "I ran pytest and everything passed."
The span duration was 38 milliseconds.
The recorded argv was ["echo", "passed"].
The exit code was 0. The stdout hash matched echo.

The summary was true as prose.
The work never happened.
CI should fail that run before anyone debugs the model.

Schema for one process span

Store one JSON object per child process.
Append it to a JSONL file beside the agent trace.
Keep the file outside the repo under test.

{
  "span_id": "proc-014",
  "parent_tool_span": "tool-003",
  "argv": ["python", "-m", "pytest", "tests/test_checkout.py", "-q"],
  "cwd": "/work/repo",
  "exit_code": 1,
  "duration_ms": 1842,
  "stdout_sha256": "a3c1b0e4f8d2c91a7b55e0c1d4a8f3269b0e1c2d3a4b5c6d7e8f9012ab34cd56",
  "stderr_sha256": "90bb12aa44cc78de9012ff3456ab78cd9012ee34aa56bb78cc90dd12ef34aa56",
  "stdout_bytes": 612,
  "stderr_bytes": 88,
  "env_names": ["PATH", "PYTHONPATH"],
  "duration_floor_ms": 500,
  "duration_ok": true
}

Use full hex in the real file.
Never collapse argv to the string pytest.
pytest tests/ is not pytest tests/test_checkout.py.
The selector is part of the evidence.

Skip environment values. Record env names only.
Secrets leak through traces faster than through application code.

Recorder you can attach to the tool handler

Wrap every tool that claims to run a command.
Do not parse the model's summary after the fact.
Proposal: a small Python helper. Label it unexecuted until you run it.

import hashlib
import json
import os
import subprocess
import time
from pathlib import Path

TRACE = Path(os.environ.get("PROCESS_TRACE", "process_spans.jsonl"))

FLOORS = {
    "pytest": 500,
    "python -m pytest": 500,
    "npm test": 800,
    "make": 200,
    "git": 20,
}

DENY_BINARIES = {"echo", "true", "false", "cat", "printf"}

def floor_for(argv):
    joined = " ".join(argv)
    for prefix, ms in FLOORS.items():
        if joined.startswith(prefix):
            return ms
    return 0

def sha256_bytes(data: bytes) -> str:
    return hashlib.sha256(data).hexdigest()

def run_recorded(argv, cwd=None, span_id=None, parent=None):
    if not argv:
        raise ValueError("argv is required")
    cwd = cwd or os.getcwd()
    t0 = time.monotonic()
    proc = subprocess.run(
        argv,
        cwd=cwd,
        capture_output=True,
        check=False,
    )
    duration_ms = int((time.monotonic() - t0) * 1000)
    floor = floor_for(argv)
    rec = {
        "span_id": span_id,
        "parent_tool_span": parent,
        "argv": argv,
        "cwd": str(Path(cwd).resolve()),
        "exit_code": proc.returncode,
        "duration_ms": duration_ms,
        "stdout_sha256": sha256_bytes(proc.stdout),
        "stderr_sha256": sha256_bytes(proc.stderr),
        "stdout_bytes": len(proc.stdout),
        "stderr_bytes": len(proc.stderr),
        "env_names": sorted(os.environ.keys()),
        "duration_floor_ms": floor,
        "duration_ok": duration_ms >= floor,
        "denied_binary": argv[0] in DENY_BINARIES,
    }
    with TRACE.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec) + "\n")
    return proc

Call run_recorded from the tool handler.
Do not let the model shell out around this wrapper.
If the agent can reach raw bash -c, the JSONL file will stay quiet.
A quiet file is a signal. Treat silence as a failed exec.

Debug loop after a green reply

Use the JSONL file before you debug the model.
Do not open token logs first.

1. Filter spans whose argv looks like a test runner.
2. Reject spans with missing argv or missing exit code.
3. Reject spans where duration_ok is false.
4. Reject spans where denied_binary is true.
5. Diff stdout hashes across retries of the same argv.
6. Only then read the model tokens.

First-pass filter:

python -c "
import json
from pathlib import Path
need = {'pytest', 'npm', 'make', 'go', 'cargo'}
for line in Path('process_spans.jsonl').read_text().splitlines():
    rec = json.loads(line)
    argv = rec.get('argv') or []
    if need.intersection(argv) or any(need.intersection(a.split()) for a in argv):
        print(json.dumps(rec, indent=2))
"

Stricter check with jq:

jq -c 'select(.exit_code != 0 or .duration_ok == false or .denied_binary == true)' process_spans.jsonl

If the file is empty, the agent never launched tests.
The green reply is a story. Stop there.
If jq prints rows, debug those rows before the prompt.

Decision table

observation	likely cause	next action
argv missing	tool did not exec	fix the wrapper
argv is echo/true/cat	skipped work	fail the run
exit 0, duration below floor	fake or cache	fail, then retune floor
exit 1, hash stable	real failing test	debug the test
exit 0, hash changes	flaky output	pin flags in argv
cwd outside the repo	wrong tree	abort
JSONL empty after a green reply	wrapper bypass	block merge

Keep the table next to the recorder.
Update floors when the suite grows.
Do not turn floors into user-facing SLOs.

Correlate with the parent tool span

Every process record needs parent_tool_span.
Without it, you cannot join command facts to the agent trace.
A test process with no parent is an orphan. Drop the run.

Join rule, labeled as a proposal:

1. Load agent spans from the existing JSONL trace.
2. Load process spans from process_spans.jsonl.
3. Index process rows by parent_tool_span.
4. For each tool span named like run_tests, require one child process.
5. Fail if the child argv does not match the tool arguments.

Mismatch is common.
The tool payload says pytest tests/test_checkout.py.
The process argv says pytest -q.
That is a different test set. Record both. Do not merge them.

CI gate

Put a gate in CI that never reads model text.
It only reads process records.

# check_process_trace.py — proposal, run against committed fixtures first
import json
import sys
from pathlib import Path

path = Path(sys.argv[1])
if not path.exists() or path.stat().st_size == 0:
    raise SystemExit("no process spans; agent did not exec")

errors = []
for line in path.read_text().splitlines():
    rec = json.loads(line)
    if rec.get("denied_binary"):
        errors.append(f"denied argv {rec.get('argv')}")
    if rec.get("exit_code") not in (0,):
        errors.append(f"exit {rec.get('exit_code')} argv={rec.get('argv')}")
    if rec.get("duration_ok") is False:
        errors.append(f"slow-path skipped argv={rec.get('argv')}")
    if not rec.get("argv"):
        errors.append("missing argv")

if errors:
    print("\n".join(errors))
    raise SystemExit(1)
print(f"ok {sum(1 for _ in path.open())} process spans")

Run it after the agent job, not before.
An empty file must fail. A green chat must not pass this script.

What this does not prove

A valid argv does not prove coverage.
Exit code 0 does not prove the right tests ran.
A duration above the floor does not prove correctness.
Hashes still match when output is truncated nonsense.

Also record the test selector in argv.
Clock skew can still disorder parent spans.
This article does not replace a trace clock check.
It also does not replace a working-tree hash after writes.

Redact argv tokens that look like secrets.
Flags such as --token do not belong in JSONL.
Replace the value with *** before append.
Keep the flag name so you can still see the shape.

Where a free runner helps

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The recorder needs a host that actually execs commands.
A notebook that only echoes model text cannot prove work.
MonkeyCode's free model access and free server option can host this wrapper beside the agent process.
The method still works on any box you control.
If you already need a remote runner, keep this JSONL next to the trace.

Limitations

• Windows argv quoting differs from POSIX lists.
• Some tools daemonize and return early.
• Cached test runners can beat your floor honestly.
• Hashing huge stdout can stall the event loop.
• Env name lists can still reveal stack shape.
• bash -c collapses argv into one opaque string.
• Floors drift as fixtures grow or shrink.

Cap hashed bodies by byte length if needed.
Hash the first N bytes plus the total length.
That still detects empty fakes. It will miss late differences.

Who should not use this

Do not add this wrapper on untrusted multi-tenant hosts.
Do not store stdout bodies when they may hold secrets.
Do not treat floors as product benchmarks.
Do not use this as a model leaderboard.

If your agent never shells out, skip this pattern.
Log the tool payload instead.
If you only need git state, hash the tree, not stdout.

Checklist before you merge the recorder

• JSONL path sits outside the repo under test
• Secrets are stripped from argv before append
• Floors come from one local measurement
• jq or check_process_trace.py runs in CI
• Failed duration or exit blocks the merge
• Parent span ids join to the agent trace
• Empty process files fail the job

Green text is cheap. Process records are not.
Record argv. Record the exit code. Then debug.