FAQ: Did the Tool Run, or Did the Model Only Describe It?

Did the agent run the command, or only narrate it?

I keep hearing that mix-up in reviews. A pretty tool block shows up. Someone ships the vibe. Then CI fails on a file that never moved.

This FAQ is not about chat style. It is about side effects you can falsify.

I treat every tool call as a proposal. The runner, the exit code, and a fixture do the proving. No fixture? No claim.

Why this FAQ, not another demo

Agent talk is loud this week. Terms multiply. MCP posts multiply. Standup still repeats the same five claims.

None of those claims need a GPU story. They need a shell, a hash, and a dirty worktree check.

I use a free model only to draft the proposal. I use a free server only as one place to execute it. Scoring stays in files I control.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am writing this as product outreach, not as a lab paper. MonkeyCode's free model access and free server option are the cheap loop I describe. I will not name models, quotas, or boxes I cannot verify.

How to read each answer

Every myth has three parts.

• Claim: what people repeat.
• Evidence: what you can actually inspect.
• Corrected model: what I keep in my head.

Skip the poetry. Steal the harness later.

Myth 1: A tool-call block is a syscall

Claim: "It called pytest, so tests ran."

Evidence: A transcript can emit a JSON blob and stop. The blob is text. Text is not a process.

Did you store argv? Did you store the working directory? Did a parent process wait on an exit code?

If those three are missing, you watched theater.

Corrected model: A tool call is a request. A runner may accept it, rewrite it, or drop it. Only the runner's wait status is evidence.

# proposed capture, not a vendor API
printf '%s' "$TOOL_JSON" > /tmp/tool-request.json
python3 - <<'PY'
import json, os, subprocess, hashlib, pathlib
req = json.loads(pathlib.Path("/tmp/tool-request.json").read_text())
argv = req["argv"]  # must be a list, never a shell string
proc = subprocess.run(argv, capture_output=True, text=True)
blob = {
  "argv": argv,
  "cwd": os.getcwd(),
  "code": proc.returncode,
  "stdout_sha": hashlib.sha256(proc.stdout.encode()).hexdigest(),
  "stderr_sha": hashlib.sha256(proc.stderr.encode()).hexdigest(),
}
pathlib.Path("side-effect.json").write_text(json.dumps(blob, indent=2))
raise SystemExit(proc.returncode)
PY

Label this as a template. Wire your own allowlist. Never interpolate the model into bash -lc.

Myth 2: Valid JSON means the side effect landed

Claim: "The arguments parsed. We are done."

Evidence: Schema checks prove shape. Shape is not durability. An HTTP 200 can still write nothing. A CLI can print "ok" and return 1.

Would you merge a migration because the SQL looked valid?

I would not. I want a row count, a file hash, or a failing assertion.

Corrected model: Split three greens.

1. Parse green: JSON matches the tool schema.
2. Process green: exit code is the one you expected.
3. World green: a fixture file or query matches.

Score them apart. A parse green with a world red is still red.

# proposed fixture layout
fixtures/
  echo-tool.world.json     # expected hashes / paths
  echo-tool.allow.txt      # argv[0] allowlist

If the world file is absent, the eval did not happen. You had a conversation.

Myth 3: A free server starts empty, so residue cannot bite

Claim: "It is a scratch box. Yesterday is gone."

Evidence: Did you measure the worktree at session start? Caches linger. node_modules linger. Docker layers linger. A leftover .env is not a personality quirk. It is state.

I ask four questions before I trust a free server run.

• What is git status --porcelain=v1 right now?
• What is HEAD?
• Which paths are ignored but present?
• Which processes still hold files in this tree?

No answers? Then "clean" is a mood.

Corrected model: Isolation is an assertion, not a brochure line. Snapshot first. Then let the agent touch disk.

# proposed preflight — run before any tool
set -euo pipefail
git rev-parse HEAD > preflight.head
git status --porcelain=v1 > preflight.status
find . -name node_modules -prune -o -name .venv -prune -o -type f -print \
  | sort | sha256sum > preflight.tree.sha

Compare the same three files after the session. If they drift and you did not expect it, the run is contaminated. Do not debug the model yet. Debug the dirt.

Myth 4: The final English answer is the eval

Claim: "It explained the fix well, so the fix is good."

Evidence: Language models are good at sounding finished. Finished English can wrap a no-op. It can wrap a partial write. It can wrap a test command it never launched.

Would you accept a PR description with no diff?

Then why accept a transcript with no side-effect.json?

Corrected model: Score the artifact, then read the prose. Prose is a comment. The comment can be wrong even when the patch is right. The inverse is worse.

A tiny decision table I keep in the repo:

Signal in the chat	What I record	What I decide
Tool JSON only	request bytes	not executed
"Tests passed"	nothing	not evidence
Exit code 0, no fixture	process green only	still blocked
Exit code 0 + world match	three greens	eligible for review
World mismatch	hashes + diff	fail, ignore the speech

Pin the table next to the harness. Argue with the row, not the vibe.

Myth 5: Draft quality means you can skip capture

Claim: "It is only a free model. Capture is overkill."

Evidence: Cheap proposals still write files. Files still enter reviews. Reviews still leak into main. The cost of the tokens was never the risk. The risk is an untracked edit that looks intentional.

Free does not mean fictional. A free server can hold secrets you pasted. A free model can emit a curl | bash you almost ran.

Corrected model: Price picks the generator. The harness picks the truth. If the loop is cheap, I want more capture, not less. Cheap loops are how junk multiplies.

The artifact: a one-directory harness

Here is a compact layout you can copy. Treat it as a proposal. Run it on a throwaway clone first.

harness/
  allow.txt          # one executable name per line
  run.sh             # the only entrypoint
  fixtures/world.json
  out/               # gitignored verdicts

allow.txt example:

python3
pytest
git

fixtures/world.json example:

{
  "expect_code": 0,
  "must_exist": ["src/app.py", "tests/test_app.py"],
  "must_not_exist": [".env", "secrets.json"],
  "stdout_must_contain": ["passed"]
}

run.sh — short, strict, boring:

#!/usr/bin/env bash
set -euo pipefail
cd "$(dirname "$0")/.."
mkdir -p harness/out

req=${1:?usage: run.sh tool-request.json}
allow=harness/allow.txt
world=harness/fixtures/world.json

python3 - "$req" "$allow" "$world" <<'PY'
import json, sys, subprocess, os, pathlib, hashlib
req_path, allow_path, world_path = sys.argv[1:4]
req = json.loads(pathlib.Path(req_path).read_text())
allow = {line.strip() for line in pathlib.Path(allow_path).read_text().splitlines() if line.strip()}
argv = req["argv"]
if not argv or argv[0] not in allow:
    raise SystemExit(f"blocked argv0: {argv[:1]!r}")
proc = subprocess.run(argv, capture_output=True, text=True)
world = json.loads(pathlib.Path(world_path).read_text())
verdict = {
    "argv": argv,
    "code": proc.returncode,
    "expect_code": world["expect_code"],
    "stdout_sha": hashlib.sha256(proc.stdout.encode()).hexdigest(),
    "missing": [p for p in world["must_exist"] if not pathlib.Path(p).exists()],
    "forbidden_present": [p for p in world["must_not_exist"] if pathlib.Path(p).exists()],
    "stdout_misses": [s for s in world["stdout_must_contain"] if s not in proc.stdout],
}
verdict["pass"] = (
    verdict["code"] == verdict["expect_code"]
    and not verdict["missing"]
    and not verdict["forbidden_present"]
    and not verdict["stdout_misses"]
)
pathlib.Path("harness/out/verdict.json").write_text(json.dumps(verdict, indent=2))
print(json.dumps({"pass": verdict["pass"], "code": verdict["code"]}))
raise SystemExit(0 if verdict["pass"] else 1)
PY

Sample request file:

{
  "argv": ["python3", "-m", "pytest", "-q", "tests/test_app.py"]
}

What this does not do: it does not grade English. It does not pin a model. It does not claim the tests are good. It only refuses the myth that speech equals execution.

Run it like this:

chmod +x harness/run.sh
./harness/run.sh /tmp/tool-request.json
cat harness/out/verdict.json

If pass is false, I do not argue with the model. I open the verdict. Then I open the tree.

A 20-minute drill you can actually finish

Do not turn this into a platform. Use one clone.

1. Copy the harness into a throwaway repo.
2. Commit a failing fixture on purpose.
3. Ask any model, free or not, for a tool JSON that should fix it.
4. Refuse to paste that JSON into a shell.
5. Feed it to run.sh.
6. Read verdict.json before you read the chat.

What should happen? The first pass fails on purpose. Good. You just proved the harness can say no. The second pass should either go green on the world file or stay red with hashes. Both outcomes are useful. A fluent apology is not.

Limitations, said plainly

This workflow does not make a model deterministic. Free models will still drift. Re-run the same request and you may get a different argv. That is why the allowlist exists.

This workflow does not prove a free server is single-tenant. I did not audit anyone's isolation. If the box is shared, treat secrets as already burned. Do not paste production tokens. Do not mount your real home directory.

This workflow does not replace CI. A local pytest green is still local. Promote the same verdict.json shape into the pipeline if you care. If you skip that, you only tested your laptop theater.

The fixture can be weak. stdout_must_contain: ["passed"] is easy to game. Tighten it. Prefer file hashes. Prefer query counts. Prefer golden snapshots you wrote by hand.

I am not giving timings, model names, or hardware claims. Those go stale overnight. The wait status does not.

Who should not use this

Skip this if you need a formal eval suite with raters and pinned seeds. This is a side-effect gate, not a research protocol.

Skip this if your tools must hit production APIs. Point the allowlist at pytest and python3, not at deploy scripts.

Skip this if you cannot keep secrets off the box. A free server is the wrong place for customer data. Period.

Skip this if your team already captures argv, cwd, exit code, and world hashes in CI. You do not need my script. You need to stop arguing with transcripts.

The mental model I want you to steal

Ask one question after every agent session.

What changed on disk, and who measured it?

If the answer is "the model said so," you have a story. If the answer is a verdict file, you have a chance. Stories do not merge. Verdicts can.

Keep the generator cheap if you want. Keep the scorer boring. Boring is the point.

If you already have a MonkeyCode free-server session open, drop harness/run.sh in and read verdict.json before the next tool blob. That is the only ask.