Did the Tool Run? An Agent Side-Effect FAQ

Did the tool run, or did the model only narrate?

I keep getting that question after messy agent demos. The chat window looks finished, but the ticket stays open.

This FAQ is about that gap in control flow. It is not about model leaderboards or vendor charts.

Why another FAQ on agents?

This account already argued with free-server folklore before. Those posts covered machines, billing, and false greens.

This one is about control flow and false completions. Agents make people assume the work already happened.

Do you treat a chat loop like a job queue? That habit breaks in quiet and expensive ways.

What I mean by a dry-run loop

You have a task, and a model proposes steps. Something in that loop might mutate real state.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option. I treat that pair as a scratch pad for agent drills.

I do not treat it as a hidden worker fleet. I will not invent quotas, model names, or hardware. I will not paste latency wins I never measured.

Please treat today's availability as unknown again tomorrow. Do not build a company on a scratch pad.

The mental model I want

A request is still a request, not a queue. Mixing those ideas duplicates charges, files, and comments.

Evidence lives outside the transcript every single time. You should ask where the proof actually sits.

Myth 1: "Done" in chat means the side effect ran

Why trust a paragraph that sounds so completely sure? It copies the calm tone of real dashboards.

The model is not your database or issue tracker. English confidence is not the same as a commit.

The corrected model

Assistant text is only a claim on screen. You still need an independent probe on disk.

• The last sentence is a claim, not a receipt.
• A planned tool call is not a completed call.
• A model HTTP 200 is not your mutation.

Here is a proposed check, not a scored benchmark. Copy it before you trust the chat output.

# proposed: freeze evidence before you believe "done"
mkdir -p ./out
cp ./out/app.conf ./out/app.conf.before
sha256sum ./out/app.conf.before

After the agent claims success, compare the hashes. Do not skip the diff because the prose looks confident.

# proposed: the file must change, or the claim failed
sha256sum ./out/app.conf
diff -u ./out/app.conf.before ./out/app.conf

No diff showed up in that output at all? The narrative lied, and the work is still open.

Myth 2: Free tokens are a retry budget

Did the call drop halfway through a tool plan? People hammer retry because the meter looks free.

Free model access does not make retries safe. It only makes those extra retries feel tempting.

A retry is a second attempt at the same side effect. Without a key, you just create twin writes.

What to do instead of hope

• Retry the model talk, not the tool blindly.
• Tools need idempotency keys before the first attempt.
• "It might have worked" is a branch, not a vibe.

# proposed_idempotency.py — example workflow, not a live run
import json, pathlib, time

STORE = pathlib.Path("./out/idempotency.json")

def already_applied(key: str) -> bool:
    if not STORE.exists():
        return False
    data = json.loads(STORE.read_text())
    return key in data

def mark_applied(key: str, evidence: str) -> None:
    data = json.loads(STORE.read_text()) if STORE.exists() else {}
    data[key] = {"evidence": evidence, "ts": int(time.time())}
    STORE.parent.mkdir(parents=True, exist_ok=True)
    STORE.write_text(json.dumps(data, indent=2))

def apply_once(ticket_id: str, mutate) -> str:
    key = f"close-ticket:{ticket_id}"
    if already_applied(key):
        return "skip: already applied"
    evidence = mutate()
    mark_applied(key, evidence)
    return evidence

Would you rerun mutate() because the model timed out? Do that only when the key says no.

Myth 3: The free server is agent memory

Is the box a brain, or a rented shell with a prompt? People collapse those two ideas far too fast.

Prompt memory dies when the session finally dies. Files on a free server can vanish or collide.

A free server option is useful for short drills. It is not a durable long-term memory service.

Where state should live

• Put long-term state in a store you control.
• Treat remote scratch disks as hostile and shared.
• If you need recall, write a record, then read it back.

# proposed: do not trust "I will remember that"
mkdir -p ./out
printf '%s\n' "$TASK_ID" > ./out/task_id.txt
test -s ./out/task_id.txt
cat ./out/task_id.txt

Can you SSH back tomorrow and find that file? Design the drill as if you cannot.

Myth 4: A tool-call block means the tool process ran

Why paste a JSON tool call and call it done? Because that blob looks like a finished RPC.

The block is a request the runtime might ignore. It can be truncated, refused, or simulated in prose.

The corrected model

A tool ran only with runtime evidence. Prose cannot stand in for that receipt.

• A fenced JSON blob is still only model text.
• A function tag is not a process identifier.
• You need the tool's file, exit, or HTTP trace.

Here is a proposed receipt check on disk. Fail the drill if this file is missing.

# proposed: require a tool-side receipt
test -f ./out/tool_receipt.json
jq -e '.ok == true and .key != null' ./out/tool_receipt.json

If jq fails, the tool never committed work. The pretty JSON in chat does not count.

Myth 5: Lab agents can skip isolation

It is only a lab, so shared credentials are fine, right? That is how tokens leak into public pastebins.

A shared free box is not a private tool host. Agent processes read whatever files you left behind.

Isolation is part of the drill

Corrected model: isolation is part of the experiment itself. It is not a later luxury for labs.

• Use a throwaway token with a tight scope.
• Give each drill its own working directory.
• Never point a lab agent at production inboxes.

# proposed: one directory per drill, nothing clever
DRILL="./drills/$(date +%Y%m%d)-$RANDOM"
mkdir -p "$DRILL"
cd "$DRILL"
umask 077
pwd

Would you paste a production bot token into that folder? Then do not run the drill at all.

Myth 6: The loop can replace a worker queue

This is the parent myth, and the others feed it. Why build Kafka energy out of a chat textarea?

Queues retry with backoff, visibility timeouts, and poison handling. Chat loops retry with hope and extra tokens.

A model call has no delivery guarantee you can prove from the text. You must build that guarantee around it yourself.

A blunt split

• If the work must happen once, use a queue plus a lock.
• If the work can be a demo, say it is a demo.
• If you cannot observe it, you cannot ship it.

# proposed decision, not a vendor scorecard
need exactly-once side effect? -> real queue + idempotency store
need a planning sketch?        -> model loop is enough
need overnight memory?         -> not the chat, not scratch disk
need isolation?                -> dedicated workspace, scoped tokens

A 25-minute canary you can copy

This is a proposed test plan without a leaderboard. I am not attaching timings I never captured.

Steps

1. Create a throwaway directory and a nonce file.
2. Define one side effect with an idempotency key.
3. Ask the agent to perform that single side effect.
4. Ignore the final paragraph until evidence exists on disk.
5. Run the hash diff and the idempotency store check.
6. Force a retry and confirm the side effect does not double.
7. Delete the workspace and confirm you did not need its disk.

# proposed_canary.sh — example workflow
set -euo pipefail
ROOT="./drills/canary-$$"
mkdir -p "$ROOT/out"
echo "nonce-$RANDOM" > "$ROOT/out/nonce.txt"
BEFORE=$(sha256sum "$ROOT/out/nonce.txt" | awk '{print $1}')

# pretend the agent closed the task here
# your tool should write evidence, not a slogan
echo "nonce-$RANDOM-applied" > "$ROOT/out/nonce.txt"
echo '{"close-ticket:1":{"evidence":"nonce-applied"}}' > "$ROOT/out/idempotency.json"
echo '{"ok":true,"key":"close-ticket:1"}' > "$ROOT/out/tool_receipt.json"
AFTER=$(sha256sum "$ROOT/out/nonce.txt" | awk '{print $1}')

test "$BEFORE" != "$AFTER"
test -f "$ROOT/out/idempotency.json"
jq -e '.ok == true' "$ROOT/out/tool_receipt.json" >/dev/null
echo "canary steps finished locally"

If step six doubles the write, you built a loop. You did not build a worker at all.

Decision table

Keep this table in the repo, not in the prompt. Walk it before you trust another "done".

Claim you hear	Probe	If the probe fails
"The agent updated config."	Hash diff the file.	Treat the task as open.
"Retry, it is free."	Check the idempotency key.	Do not call the tool again.
"It will remember later."	Read state from your store.	Rewrite the record now.
"The tool JSON ran."	Find tool_receipt.json.	Treat it as not executed.
"Lab isolation can wait."	List tokens in the workspace.	Stop the drill.
"The loop is the queue."	Name a visibility timeout.	Move the work to a queue.

Limitations

This workflow does not create reliability on its own. It only refuses a fake kind of reliability.

Free model access can change without a blog post. A free server option is still a borrowed machine.

I did not measure p50, cost, or token burn. Those numbers would be fiction in this FAQ.

This canary will not catch semantic bugs in patches. A file can change and still be wrong.

It also will not secure a hostile tenant. Isolation here is a drill habit, not a hard sandbox.

Who should not use this approach

Do not use a chat loop for paid user side effects. Do not use a scratch server as the system of record.

Skip this if you cannot inspect the tool's real output. Skip this if your agent already sits on a real queue.

If you need guaranteed retention, bring your own store. If you need a private tool host, bring your own isolation.

What I want you to keep

Ask one question after every agent run today. Where is the evidence for that side effect?

If the evidence is only English, the work is not done. If a retry cannot prove uniqueness, you do not have a worker.

A scratch pad is still useful for these drills. Try MonkeyCode's free model access and free server option. Then leave the queue to software that is actually a queue.