The Tool Succeeded, the Repo Did Not: A Myth-Busting FAQ

The standup recording still had the spinner on screen when the engineer muted the call. Twenty tool calls had completed in the shared log, and every status line said success. The repository on disk told a quieter story: git status printed nothing but a clean working tree. That gap between a busy agent trace and an unchanged index is the myth this FAQ is built to break.

Teams that adopted coding agents this year often score a session by how long the spinner moved. The analogy is a kitchen that sounds busy because every drawer is opened, while the oven stays cold. Tool success is an I/O event, not evidence that a behavior in the codebase changed. Until a diff, a test, or a failing assertion moves, the loop has only grown context.

A practical way to keep that distinction honest is a small trace ledger that classifies each step before anyone praises the model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter only as a place to reproduce the ledger. They are not a substitute for review, and they do not make a busy trace equivalent to a patch. The examples that follow are labeled procedures, and they are not production benchmarks from this account.

Myth: a successful tool call means the agent understood the system

When an MCP server or a shell wrapper returns a 200 and a JSON blob, people narrate that as comprehension. The corrected model is simpler and less flattering: the process received bytes and appended them to the window. Comprehension would show up later as a minimal diff that a human can explain without the log. If those bytes never constrain the next edit, the call was tourism through the filesystem.

Consider a labeled session in which list_directory and read_file both return quickly on a service that talks MCP. The trace looks healthy because every envelope has a result key and no exception. That is the same evidence a curl gives when a CDN answers; it is not evidence that the handler behind the URL is the handler you meant. Treat the tool result as untrusted input until a later step mutates the tree or a test assertion flips.

# labeled procedure: snapshot the tree before praising any tool result
git status --porcelain
git diff --stat HEAD
git rev-parse HEAD

If those three commands are identical before and after a dozen “successful” calls, the agent did I/O, not software change. The mental model is a mailroom, not a senior engineer: packages arrived, labels were scanned, and nothing was filed into the product.

Myth: extra loop iterations are free when the server is free

A complimentary machine still meters context, because every retry concatenates prior tool output into the next prompt. The free server removes a credit-card form; it does not remove the cost of unread logs sitting in the window. Treating retries as experiments only works when each retry starts from a recorded snapshot, not from an inflated transcript. Otherwise the second hour is spent rereading the first hour under a different spinner.

The analogy that holds is a whiteboard that nobody erases. Adding another equation is cheap in isolation, and then the board becomes unreadable, so the next visitor copies the clutter. Free-tier loops fail the same way: the hardware bill is zero while the prompt bill is a pile of stack traces, directory listings, and repeated plans. Stop the loop when the new step cannot name a file, a test, or a commit hash that the previous step lacked.

# labeled procedure: bound a retry by whether the input actually changed
BEFORE=$(git write-tree)
# ... one agent iteration ...
AFTER=$(git write-tree)
test "$BEFORE" = "$AFTER" && echo "retry produced no tree change"

A free server that still has your process is not the same object as a clean experiment. If you need isolation, snapshot the worktree; do not assume the next iteration is a new machine just because the product page said free.

Myth: a written plan is already the patch

Agents on free models often emit numbered steps that read like a design review. Prose that names files is not the same object as a hunk that Git can apply. Keep a compiler in mind: the plan is commentary, and the only artifact that ships is the object file, here a diff. If the session ends with a plan and a clean index, the compiler never ran.

This myth spreads because language models are fluent, and fluency looks like commitment. A paragraph that says “update invoice.py to reject negative totals” can be pasted into a ticket, and it still does not reject a negative total. Ask for the patch as if you were talking to a build system: no credit for intended object files. The review starts at git diff, not at the bullet list in the chat transcript.

# labeled procedure: refuse to review prose that never became a hunk
git diff --cached --exit-code; echo "index_exit:$?"
git diff --exit-code; echo "worktree_exit:$?"

Both exit codes zero means there is nothing to review, regardless of how confident the plan sounded. File the transcript as notes if it helps the next human; do not file it as a change.

Myth: rerunning the same red test is a new experiment

Re-running a failing test without changing fixtures, seed, or code is a heartbeat, not an experiment. The log will look industrious because the runner prints the same traceback again. Charge that step to confirmation, then stop the loop unless the input changed. Otherwise the free-tier window fills with duplicate stacks that crowd out the one file that needed an edit.

A useful analogy is a smoke alarm with a drained battery. The chirp is data the first time, and it is noise the twelfth time, even though each chirp is a “successful” sensor read. Duplicate test output should collapse into a single fingerprint. If the fingerprint is unchanged, the loop owes you a mutation or a halt, not another green tool envelope.

# labeled procedure: fingerprint the failure so duplicates are visible
pytest -q tests/test_invoice.py --tb=short -p no:cacheprovider \
  | tail -n 40 | sha256sum

Store that hash beside the git tree hash. Two matching pairs in a row means the agent is polling, not debugging.

Artifact: a trace ledger that scores I/O against mutations

The workflow is deliberately boring, which is the point. Export each agent step as JSONL, then score it with a local script that cannot be impressed by spinners. The schema below is a proposal, not a vendor format, and you can adapt field names to whatever your runner already prints.

{"ts":"2026-09-14T10:02:11Z","kind":"tool_result","name":"list_directory","ok":true,"bytes":4812,"tree":"unchanged"}
{"ts":"2026-09-14T10:02:19Z","kind":"tool_result","name":"read_file","ok":true,"bytes":2201,"tree":"unchanged"}
{"ts":"2026-09-14T10:03:04Z","kind":"model_message","name":"plan","ok":true,"bytes":1604,"tree":"unchanged"}
{"ts":"2026-09-14T10:03:40Z","kind":"tool_result","name":"apply_patch","ok":true,"bytes":388,"tree":"mutated"}
{"ts":"2026-09-14T10:04:12Z","kind":"tool_result","name":"pytest","ok":false,"bytes":933,"tree":"unchanged"}

# labeled example: classify a session without treating tool ok as progress
from __future__ import annotations

import json
from pathlib import Path

WORK = {"apply_patch", "write_file", "git_commit", "edit"}
CHECKS = {"pytest", "npm_test", "cargo_test", "lint"}

def score(path: str) -> dict:
    io_ok = 0
    mutations = 0
    check_flips = 0
    duplicate_bytes = 0
    seen = set()
    last_check = None
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        ev = json.loads(line)
        kind = ev.get("kind")
        name = ev.get("name", "")
        blob = (name, ev.get("bytes"), ev.get("ok"), ev.get("tree"))
        if blob in seen:
            duplicate_bytes += int(ev.get("bytes") or 0)
        seen.add(blob)
        if kind == "tool_result" and ev.get("ok"):
            io_ok += 1
        if ev.get("tree") == "mutated" or name in WORK:
            mutations += 1
        if name in CHECKS:
            fingerprint = (name, ev.get("ok"), ev.get("bytes"))
            if last_check is not None and fingerprint != last_check:
                check_flips += 1
            last_check = fingerprint
    return {
        "io_ok": io_ok,
        "mutations": mutations,
        "check_flips": check_flips,
        "duplicate_bytes": duplicate_bytes,
        "productive": mutations > 0 or check_flips > 0,
    }

if __name__ == "__main__":
    print(json.dumps(score("session.jsonl"), indent=2))

Run it against the sample and the numbers stay unromantic: several successful tools, one mutation, no check flip after the patch, and no excuse to call the session done. Add a human gate that refuses a merge request when productive is false, even if the chat transcript reads like a victory lap. That gate is the whole method; the script is only a scale that does not flatter the cook.

python3 score_trace.py
git diff --stat

A compact decision rule sits behind those commands. If tools succeeded and the tree is unchanged, you observed I/O. If the tree changed and no check flipped, you observed an untested edit. If a check flipped and the fingerprint is new, you observed an experiment. If none of those are true, you observed a spinner.

Limitations, and who should not use this

The ledger cannot see side effects outside Git, so database migrations, object-store writes, and remote feature flags will look like tourism. It also cannot prove that a mutation is correct; it only proves that something in the tree moved. Teams that need production SLAs, secret-scanning guarantees, or isolated tenancy should not treat a shared free server as that environment, because this article does not claim hardware isolation, quotas, or retention. Do not paste credentials into tool arguments to “help” the model, then ask a free server to forget them.

The method is a poor fit for exploratory interviews with a codebase you do not intend to change. In that setting, successful reads are the work, and a clean git tree is the desired ending. It is also a poor fit when your runner does not emit structured traces, because reconstructing JSONL from screenshots will create a second myth: that a hand-built log is the session. If you cannot export events, watch git write-tree instead and ignore the spinner entirely.

Free model access remains useful for rehearsing the ledger on throwaway branches, especially when you want a second loop without standing up local GPUs. Keep the rehearsal on a fork that contains no production secrets, and throw the worktree away when the hashes stop moving. The corrected mental model is small enough to reuse anywhere: activity is context, and review begins at the diff.

If you want a disposable place to run the same scoring loop, MonkeyCode’s free model access and free server option are one way to stage a throwaway branch. Export the JSONL, keep the gate, and let the index, not the spinner, decide whether the session existed.