Your Agent's Tool Call Is Still a Claim

Did the file actually land on disk this time?

I keep hearing the same four confident answers. They sound finished. They collapse when you inspect the machine, not the chat.

This FAQ is that inspection. It is not a pep talk.

Why this FAQ exists

Agents now call tools in public demos every week. A green function name looks like shipped work. It is still only a claim about work.

I write this as a checklist. Treat every snippet as a proposed harness. I am not inventing your incident, your quota, or a scoreboard.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I reach for MonkeyCode only when free model access and a free server option help me practice the receipt loop. The product is not the proof. The bytes on disk are the proof.

Myth 1: The tool ran, so the side effect exists

Claim you will hear: "It called write_file. We are done."

Would you merge a PR because someone said tests ran? A transcript can show a successful tool name. The write can still miss the path you care about.

Collect evidence in two minutes. List the path. Hash the file. Compare expected bytes against actual bytes.

# proposed check, not a recorded benchmark
TARGET="./out/report.md"
test -f "$TARGET" && echo PRESENT || echo MISSING
test -s "$TARGET" && echo NONZERO || echo EMPTY
sha256sum "$TARGET" 2>/dev/null || true

Permissions fail quietly. Sandboxes remap directories. Models quote paths they never opened. JSON ok: true is still testimony.

Corrected mental model: the chat is testimony. The filesystem is evidence.

• Tool name in the log means attempt.
• Tool payload means you received a blob.
• File presence means the side effect is checkable.

Myth 2: Cheap inference means you can skip stop conditions

Claim you will hear: "The model is free. Let it loop."

Free model access does not make an unbounded loop polite. A free server option does not make a fork bomb safe. Tokens can cost you nothing and still burn the night.

You need a cap before the first prompt. Max steps. Wall clock. Max writes. Put them outside the model. The model will narrate caution and continue.

# proposed watchdog around any agent command
# label: unexecuted example
timeout 90s env AGENT_MAX_STEPS=12 your-agent-cmd
echo "watchdog_exit:$?"

Exit 124 means the clock won. That is a signal. It is not a vibe. Who pays when the loop never returns? You pay, in dirty files and a wedged box.

Corrected mental model: stop conditions are interface, not manners.

1. Set wall clock before launch.
2. Set max steps in the wrapper, not the prompt.
3. Set a write budget for the workdir.
4. Treat timeout as a failed receipt, not a retry hint.

Myth 3: If the agent is unsure, it will ask

Claim you will hear: "It will clarify. Models are careful now."

Have you ever seen a missing env var get a confident default? The pattern is boring. Gap appears. Guess appears. The task continues. Questions look like failure to an agent. Silence looks like permission.

Do not hope for manners. Force a gate the model cannot talk past.

# proposed gate: refuse work without an explicit confirm file
# label: unexecuted example
from pathlib import Path
import sys

confirm = Path(".agent-ok")
if not confirm.exists():
    print("REFUSE: missing .agent-ok")
    sys.exit(2)

print("GATE: explicit confirm present")

Put the gate in your wrapper. Keep it out of the system prompt. A prompt can say "ask when unsure" and still write a guessed path.

Corrected mental model: uncertainty is a missing artifact, not a speech act.

• No question in the log means assume a guess.
• No confirm file means do not start tools.
• No expected path on disk means do not celebrate.

If you did not get a question, assume a guess.

Myth 4: The schema in the prompt is the live schema

Claim you will hear: "We pasted the tool schema. It is wired."

Pasted JSON is documentation. Runtime is a process, a port, a version. Those drift without a speech. A prompt can describe search_code. The server can expose search_repo. The agent will still "succeed" by calling a ghost.

Check the live list. Do not check the system prompt. Would you trust a README over ls?

# proposed: ask the runtime, not the prompt
# swap the URL for the inspector you actually have
curl -sS localhost:3000/tools | python -m json.tool

I am not asserting your stack speaks that URL. The point is independent discovery. If discovery fails, the agent does not get a consolation prize.

Corrected mental model: schemas are deployed artifacts. Prompts are comments.

You observed	Do not conclude	Conclude instead
Tool name in the chat	Side effect landed	The model attempted a call
Tool returned JSON	The JSON is true	You received a payload
Agent process exit 0	The repo is healthy	The process finished
Chat says "done"	Tests passed	You still owe a check
Schema text in the prompt	Runtime matches	You still owe discovery

The artifact: a four-point side-effect receipt

Here is a proposed workflow for any free-box agent task. Copy it. Change the paths. There are no latency numbers here because I am not fabricating any.

What the receipt must answer

1. Did the expected path exist after the run?
2. Was the file non-empty, if emptiness is failure?
3. Did git status move only the paths you allowed?
4. Did the wrapper hit timeout or max-steps?

If the chat says yes and the receipt says no, believe the receipt.

Proposed receipt script

#!/usr/bin/env bash
# proposed harness: side-effect receipt
# label: unexecuted example, adapt paths
set -euo pipefail

TASK_ID="${1:?task id}"
ROOT="${2:-.}"
LOG="/tmp/agent-receipt-${TASK_ID}.txt"
ALLOWED="^out/"

{
  echo "task=${TASK_ID}"
  echo "cwd=$(pwd)"
  echo "time=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  git -C "$ROOT" status --porcelain || true
  git -C "$ROOT" diff --stat || true
  find "$ROOT/out" -type f 2>/dev/null | head
} > "$LOG"

missing=0
test -f "$ROOT/out/report.md" || missing=$((missing + 1))
test -s "$ROOT/out/report.md" || missing=$((missing + 1))

extra=$(git -C "$ROOT" status --porcelain | awk '{print $2}' | grep -vE "$ALLOWED" || true)
if [[ -n "$extra" ]]; then
  echo "RECEIPT:FAIL extra_paths log=$LOG"
  echo "$extra"
  exit 2
fi

if [[ "$missing" -gt 0 ]]; then
  echo "RECEIPT:FAIL missing=$missing log=$LOG"
  exit 1
fi

echo "RECEIPT:PASS log=$LOG"

Run it after the agent, not inside the agent. The agent will happily grade itself.

chmod +x receipt.sh
./receipt.sh demo-001 .
echo "harness_exit:$?"

Save the tool log as a file, not a screenshot. Screenshots are for group chats. Files are for diffs.

# proposed: keep raw tool I/O beside the receipt
mkdir -p /tmp/agent-io
cp ./agent-tools.jsonl /tmp/agent-io/demo-001.jsonl

How I walk a failure

Do not retry first. Retry hides the miss. You will "fix" a race you never saw.

1. Freeze the box. Leave the workdir dirty.
2. Copy the tool log off the chat UI.
3. Run receipt.sh against the same workdir.
4. Diff expected paths versus actual paths.
5. Only then rerun, with timeout already set.

# proposed diff of expected versus actual
diff -u expected-paths.txt <(git status --porcelain | awk '{print $2}')

Why freeze first? Because a second green chat teaches you nothing. The first dirty tree is the only honest patient.

What a receipt is not

A receipt is not a model leaderboard. A receipt is not a host ranking. A receipt is not a quota claim, a GPU claim, or a latency claim.

A passing receipt means the observable effect matched the contract. It does not mean the prose in the chat was wise. It does not mean the next task will pass.

Ask the disk. Ask the process table. Ask the clock. Stop asking the chat to grade itself.

Limitations

This FAQ does not measure model quality. It does not rank servers. It does not promise that free access lasts, or what hardware sits under it.

The harness assumes a workdir you control. It assumes timeout, git, and a checkable side effect. Pure network effects need a different probe. Truncated tool stdout can lie too. Hash the resource, not the speech.

I am not giving you a benchmark. I am giving you a refusal to take testimony as proof.

Who should not use this

Skip this if you pair live and watch every write. Skip this if the agent cannot touch a filesystem. Skip this if real CI already gates agents with artifact checks.

Do not treat a free server option as production control. A proposed timeout 90s is a seatbelt. It is not a security boundary. Do not point this wrapper at secrets, prod credentials, or customer data.

Four lines to keep

• Tool call means attempt, not effect.
• Free inference still needs a stop condition.
• Silence is a guess, not consent.
• Live schema beats pasted schema.

If you try the harness, paste the exit code. Skip the victory screenshot.