The Chat Log Is Not git status: Five Agent Myths

Did your agent actually change the repo today?
I keep seeing the same victory lap in chat.
The model prints a tool call and someone ships.
Then git status stays clean and nothing landed.
Why do we trust the transcript more than disk?
This FAQ names five claims I still hear weekly.
Each myth gets a check and a better mental model.
I am not posting leaderboards or invented timings here.

Why these myths spread now

Agent UIs render JSON as if it were a commit.
That choice trains us to skip the filesystem.
Cheap model hops also make the gap wider.
People swap models mid loop and schemas drift.
A free server can look like a durable box.
Treat it as a scratch runner until you persist.
Does any of that setup sound painfully familiar?

How to use this FAQ

Read one myth and then run the matching check.
Do not skip the command because it is the point.
I label every script as a proposed local check.
Your paths will differ so wire them to your logs.

The four beats

• The claim people keep repeating in reviews
• Why that claim spreads in agent UIs
• A command you can run against disk
• The mental model I want you using

Keep notes on one page if that helps you.

Myth 1: The tool-call JSON wrote the file

The claim. People say the write_file call created the file.

Why it spreads. Agent UIs show a green check on the call.
That green check means the model emitted well-formed JSON.
It does not mean your runner applied a real write.
Did anyone stat the path after the model turn?

The check. Run these commands against the claimed path.

# proposed local check — replace PATH
test -f PATH && echo "exists" || echo "missing"
stat -c '%y %s %n' PATH 2>/dev/null || true
git status --porcelain -- PATH

If PATH is missing then the transcript simply lied.
If git status is empty then the write never staged.

Corrected model. Treat every tool call as a proposal only.
Your runner is the only writer and disk is proof.
JSON is never a side effect on its own.

Myth 2: The free server kept my working tree

The claim. People say leftover files wait on the free server.

Why it spreads. SSH muscle memory says cloud disks persist.
A free server option is still just a runner.
Assume the tree can vanish unless you snapshot it.
Did you commit and push or did you only chat?

The check. Run this on the runner before you leave.

# proposed local check on the runner
pwd
git rev-parse --show-toplevel
git status --porcelain
git log -1 --oneline
ls -la

An empty porcelain list plus a missing path means amnesia.
Do not rebuild state from the model's spoken summary.

Corrected model. Scratch storage is not a real workspace.
Persist with git or do not expect files later.
The chat history is not a backup of the tree.

Myth 3: "Done" means the tests passed

The claim. People say the suite is green because it said done.

Why it spreads. Models love to narrate success in plain English.
Narration is cheap while a real pytest run is not.
Who actually ran the runner, the model or you?

The check. Replay your own test command and keep the exit.

# proposed local check — pick YOUR test command
python -m pytest -q
echo "exit:$?"
git diff --stat

Capture that exit code in CI, never in prose.
If you cannot replay the command, it did not happen.

Corrected model. Success is an exit code you can replay.
A sentence that contains passed is still only a sentence.
Ask for the exact command and then run it yourself.

Myth 4: Any free model honors the same tools

The claim. People say a model swap keeps the same tools.

Why it spreads. OpenAI-style schemas look portable across hosts.
The names match while required fields often do not.
One model omits a path and another invents a flag.
Did you log the raw call before you swapped models?

The check. Keep JSONL logs and diff two models on one prompt.

# proposed example: schema_diff.py
import json, sys

def load(path):
    with open(path) as f:
        return [json.loads(line) for line in f if line.strip()]

a, b = load(sys.argv[1]), load(sys.argv[2])

def keys(rows):
    out = set()
    for r in rows:
        args = r.get("arguments") or {}
        out.add((r.get("name"), tuple(sorted(args.keys()))))
    return out

print("only_a", keys(a) - keys(b))
print("only_b", keys(b) - keys(a))

Run it as python schema_diff.py model_a.jsonl model_b.jsonl.
A silent schema drift will break the agent later.

Corrected model. Tool schemas belong in the deployment contract.
Pin the model id and pin the schema together.
Diff that contract on every hop before you merge.
Free access is not a reason to skip the diff.

Myth 5: More free retries converge on truth

The claim. People say they will loop until the repo looks right.

Why it spreads. Retries feel free so hope starts to look cheap.
Loops amplify invented success and they do not search.
The model can rewrite the story of the working tree.
Without disk checks you optimize for confident looking text.

The check. Cap the loop and audit after every few turns.

# proposed local check
N=3
for i in $(seq 1 "$N"); do
  echo "turn:$i"
  git status --porcelain
  git diff --stat
done

Stop if porcelain flaps with no real diffs at all.
You are negotiating with a narrator, not with a compiler.

Corrected model. Retries only help when an oracle is present.
Git status and tests are oracles while chat is not.
A free loop without oracles is just fan fiction.

Artifact: an assumption audit

Here is the workflow I want you to copy locally.
It is small and mean and it stays reproducible.

Step 1 — Log proposals, not vibes

Write each tool call to JSONL before any execution.
Treat the next script as a stub for your runner.

# proposed example: log_call.py
import json, time, sys

def log_call(path, name, arguments, source="runner"):
    row = {
        "ts": time.time(),
        "name": name,
        "arguments": arguments,
        "source": source,
    }
    with open(path, "a") as f:
        f.write(json.dumps(row) + "\n")

if __name__ == "__main__":
    log_call(sys.argv[1], "write_file", {"path": "README.md"})

Hook log_call into the runner before any tool executes.
If you cannot log the proposal, you cannot audit later.

Step 2 — Reconcile against git

Reconcile claimed paths against git from the repo root.
Read the printed lists and do not argue with them.

# proposed example: reconcile.py
import json, os, subprocess, sys

log_path = sys.argv[1]
claimed = []
with open(log_path) as f:
    for line in f:
        row = json.loads(line)
        if row.get("name") in {"write_file", "edit_file", "apply_patch"}:
            claimed.append((row["arguments"] or {}).get("path"))

porcelain = subprocess.check_output(
    ["git", "status", "--porcelain"], text=True
)
dirty = {line[3:].strip() for line in porcelain.splitlines() if line}

print("claimed_paths", claimed)
print("dirty_paths", sorted(dirty))
print("claimed_missing_on_disk", [p for p in claimed if p and not os.path.exists(p)])
print("claimed_not_dirty", [p for p in claimed if p and p not in dirty and os.path.exists(p)])

Run python reconcile.py agent.jsonl from the repo root.
Four lists beat one confident paragraph every time.

Step 3 — Decision table

Print this table next to the agent UI during reviews.
When the chat and the table disagree, the table wins.

Claim in chat	Oracle to run	Trust the agent if	Stop the loop if
File written	test -f plus stat	Path exists and mtime moved	Path is missing
Repo changed	git status --porcelain	Path is dirty or committed	Clean tree, proud chat
Tests passed	Your test command exit code	Exit 0 on a replay	Prose "green" only
Tools still work after a hop	schema_diff.py	Name and arg keys match	Name or args drifted
Loop is making progress	Porcelain plus git diff --stat	Real paths change once	Status flaps, diffs empty

Tape that table to the review. Seriously.
The oracle column is the only column that ships.

Where a free model and free server fit

I keep this audit off production traffic on purpose.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode provides free model access and a free server option.
I treat both as a scratch harness for the audit.
The free model is for generating tool JSON to inspect.
The free server is for running reconcile.py in isolation.
Neither one is a persistent workspace, which is Myth 2.
If those options are not what you need, run locally.
The workflow does not depend on the product brand.

Limitations

This audit will not catch deep semantic bugs later.
A file can exist on disk and still be wrong.
Git status cannot see a bad algorithm by itself.
Schema diffs cannot prove that a tool is safe.
Short loops will still miss flaky tests in CI.
Do not use this checklist as a security review.
Do not use this checklist as a license audit.
Do not use this checklist as production monitoring either.

Who should skip this

Skip it if you already pin models and replay CI.
Skip it if your agent cannot execute tools at all.
Skip it if you only want chat, not a repo.
Skip it if you cannot run commands on the runner.
If you cannot see disk, you cannot close these myths.

What I want you to remember

The transcript is a story and git is a ledger.
Tool JSON is a proposal and your runner is the actor.
Free retries are not a search algorithm on their own.
Model hops are deploys so you should diff the contract.
Did git status move after the turn at all?
If not, nothing happened on disk this time.
Run the audit once on a real repository you own.
Then decide which myths you still want to keep.

Comments

[Reid Marlow]

The nastiest variant of Myth 1 is partial patch rejection. A tool call fails a fuzzy match or drops an offset hunk, returns an error string to the context window, and the model treats the attempt as an edit anyway. If the runner does not assert an exact porcelain diff or check the tool exit code before scheduling the next turn, the agent spends three more turns writing tests and documentation for code that never hit disk.