FAQ: Five Myths About "The Tool Call Succeeded"

The chat claimed the tool call succeeded tonight. You copied that claim straight into staging anyway. Guess what the host actually changed?

Often nothing. Sometimes the wrong file. Tool calling looks like engineering from the outside. A story about a tool call is not.

What this FAQ is actually about

Developers now paste function-call JSON like courtroom proof. Models narrate curl, apply_patch, and deploy steps with confidence. Then a human merges because the box looked green.

Who verified the host? Who stored argv, cwd, and the real exit code? This piece is a myth check, not a tour.

You will leave with a receipt logger you can run. You will also leave with a decision table. Keep both next to the loop.

Myth 1: The JSON in chat is a receipt

The claim: "The model emitted a tool-call block. Therefore the tool ran."

Why it spreads: Chat UIs render tool JSON in a confident card. That card looks like a kernel log. It is still just text.

What to check instead:

• Did a host process actually spawn and exit?
• What were argv, cwd, and the visible env?
• What exit code did wait() return to you?
• Where did stdout and stderr bytes go?

Corrected model: Chat JSON is a request, maybe a paraphrase. A receipt is a host-written record after wait(). Without wait(), you still have no receipt.

Ask yourself one rude question before you merge. Could the model have invented that JSON block? If yes, you do not have proof.

Myth 2: Exit 0 is the business outcome

The claim: "The process exited zero. The job is done."

Why it spreads: Unix trained us to trust status codes. HTTP trained us to trust 2xx. Agent loops mix both and then shrug.

Counterexamples worth staging on a scratch box:

• curl -s returns 0 on a 404 HTML page
• A migrate command no-ops because DATABASE_URL was empty
• A deploy script echoes done and never calls the API
• A shell used ; so true wiped a failed compile

Corrected model: Exit 0 means the wrapper did not crash. You still need an invariant after the process. Check a row, a SHA, a file, or a metric.

Would you accept "the compiler returned 0" without an artifact? Then stop accepting tool stories as releases.

Myth 3: A free host does not need traces

The claim: "It is a throwaway box. Skip the logging."

Why it spreads: Tracing feels like a production tax. Free sandboxes feel disposable and unofficial. People skip receipts when the invoice is zero.

That instinct is backwards. Cheap loops fail in noisier ways. They also hide those failures in chat summaries.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention MonkeyCode for one practical reason here. Its free model access keeps the rehearsal cheap. The free server option is a scratch host. The same receipt method still works on any SSH box.

Corrected model: Price does not change process physics. A free server still has cwd drift and leftover env. Dirty worktrees do not become clean because billing is zero.

Would you skip git because the repo is small? Same energy. Don't skip the witness file.

Myth 4: Truncated stdout is good enough

The claim: "The model summarized the last twenty lines. We are covered."

Why it spreads: Context windows are finite, so agents tail output. Humans read that tail and relax too early. The scary line was four hundred lines up.

What the tail hides on purpose:

• A warning printed long before the final ok
• A second request that actually mutated state
• Mixed stdout from two tools that overlapped
• Binary noise decoded as a success sentence

Corrected model: Store hashes of full stdout and stderr. Store byte counts beside those hashes. Store a tiny head for humans only.

If you must truncate, truncate the chat, not the log. I want both views on disk. Head for debugging. Hash for equality later.

Myth 5: One lucky run is a regression test

The claim: "It worked on the free box once. Ship the prompt."

Why it spreads: Fast loops reward the first success screenshot. People freeze that prompt like a specification. Then they call the screenshot a suite.

A prompt is not a test. A green anecdote is not coverage. Host state from that lucky run is already gone.

Corrected model: Pin the invariant in something that is not prose. Replay the tool under the receipt logger. Compare hashes, cwd, and exit codes on the next pass.

Speed is not the villain in this story. Calling a narrator a test runner is.

Artifact: a JSONL receipt logger

Label: this is example code, not a published benchmark. Run it locally or on a remote host you own. It wraps one command and appends one JSON line.

#!/usr/bin/env python3
"""Append a host-side receipt after wait(). Example only."""
from __future__ import annotations

import hashlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path

LOG = Path(os.environ.get("AGENT_RECEIPT_LOG", "receipts.jsonl"))


def run(argv: list[str], cwd: str | None = None, timeout: int = 60) -> dict:
    started = time.time()
    cwd = cwd or os.getcwd()
    error = None
    code = None
    stdout = b""
    stderr = b""
    try:
        completed = subprocess.run(
            argv,
            cwd=cwd,
            capture_output=True,
            timeout=timeout,
        )
        code = completed.returncode
        stdout = completed.stdout or b""
        stderr = completed.stderr or b""
    except subprocess.TimeoutExpired as exc:
        error = "timeout"
        stdout = exc.stdout or b""
        stderr = exc.stderr or b""
    receipt = {
        "ts": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
        "argv": argv,
        "cwd": cwd,
        "exit_code": code,
        "duration_ms": int((time.time() - started) * 1000),
        "stdout_sha256": hashlib.sha256(stdout).hexdigest(),
        "stderr_sha256": hashlib.sha256(stderr).hexdigest(),
        "stdout_bytes": len(stdout),
        "stderr_bytes": len(stderr),
        "error": error,
        "stdout_head": stdout[:240].decode("utf-8", "replace"),
        "stderr_head": stderr[:240].decode("utf-8", "replace"),
    }
    LOG.parent.mkdir(parents=True, exist_ok=True)
    with LOG.open("a", encoding="utf-8") as handle:
        handle.write(json.dumps(receipt, ensure_ascii=False) + "\n")
    return receipt


if __name__ == "__main__":
    if len(sys.argv) < 2:
        print("usage: tool_receipt.py <command> [args...]", file=sys.stderr)
        sys.exit(2)
    row = run(sys.argv[1:])
    print(json.dumps(row, indent=2))
    if row["error"] == "timeout":
        sys.exit(124)
    sys.exit(row["exit_code"] if row["exit_code"] is not None else 1)

Commands to type after you save it

export AGENT_RECEIPT_LOG="$PWD/receipts.jsonl"
python3 tool_receipt.py true
python3 tool_receipt.py curl -sS -o /tmp/body -w "%{http_code}" https://example.com
python3 tool_receipt.py git -C "$PWD" status --porcelain
python3 -c "print(open('receipts.jsonl').read())"
wc -l receipts.jsonl

Each line is boring on purpose. You can diff it tomorrow. You can grep cwd after a bad loop. You cannot gaslight a SHA256 of stdout.

Decision table: when may I trust the tool?

Chat said	Receipt says	Extra invariant	Trust?
"called curl"	no row in JSONL	none	No. Narrative only.
"deployed"	exit 0, 12-byte stdout	no SHA of the release	No. Wrapper only.
"wrote file"	exit 0	missing path on disk	No. Story leaked.
"wrote file"	exit 0	file SHA matches expected	Yes, for that file.
"HTTP 200"	curl wrote 200 and a body hash	schema check on the body	Yes, for that response.

Print this table. Keep it near the agent loop. I am not joking about the printout.

A one-hour test plan (labeled, unexecuted)

Do this on a scratch worktree. Do not point it at production data.

1. Run a command that prints ok and returns 1.
2. Confirm any chat summary still sounds like success.
3. Confirm receipts.jsonl stores exit_code 1.
4. Run curl -sS against a URL that 404s.
5. Confirm process exit 0, then fail your HTTP invariant.
6. Change cwd on purpose and rerun the same argv.
7. Compare the cwd field across the two JSONL rows.
8. Truncate the chat view. Keep the JSONL file whole.

Pass means the receipt disagrees with the story. That disagreement is the lesson you wanted.

Limitations

This logger does not wrap syscalls inside another language runtime. It does not prove idempotency across retries. It does not isolate tenants on a shared host.

A JSONL file can be deleted. So can a worktree. Humans still choose the invariant. Hashes only help after you pick the right bytes.

I am not naming models, quotas, GPUs, or uptime numbers here. A free server is not your compliance boundary. Assume it can vanish between loops. Assume it may be shared. Keep secrets off it.

Who should skip this approach

Skip this if you need attested production audit logs. Skip this if customer data cannot leave your network. Skip this if you cannot SSH to the host that ran the tool.

Also skip it if every tool already emits real traces. You do not need a toy JSONL then. Use the pipeline you already trust.

What I want you to remember

The agent is a narrator. The host is the witness. Exit 0 is a clue, not a verdict.

Run the logger once on a scratch host you control. Compare one chat sentence to one JSON line. That mismatch is the whole FAQ.