The Chat Rendered a Perfect Diff. sha256sum Did Not Budge.

Have you ever trusted a model's unified diff because the chat window made it look already applied? I did that on a remote scratch box, and the working tree still contained the original failing function. The conversation felt complete, with a tidy explanation sitting next to a green-looking pytest paste. The disk disagreed, and that disagreement sent me hunting through the wrong layer for two days.

These notes cover a forty-eight hour loop, not a launch announcement or a ranked list of tools. I was iterating on a small Python service with a model proposing edits and a remote box running commands. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I used MonkeyCode's free model access and free server option as that remote scratch box. After that, I treated the git tree as the only source of truth that could close the loop.

The question I should have asked first

If the model shows a patch, did those bytes land on the machine that will run pytest? That sounds obvious until you watch a chat render +return cents // 100 while sha256sum still prints yesterday's digest. Did the tool write a file, or did it only narrate a write for my benefit? I stopped arguing with the traceback and started arguing with hashes, collection counts, and git status.

Hours 0–8: I believed the rendered diff

I pasted a failing assertion, asked for a fix, and received a confident unified diff against src/billing.py. The model restated the rounding rule in plain English, which made the reply feel already reviewed. I ran pytest from muscle memory in another terminal that was still pointed at an older clone. Green output appeared, so I treated the incident as closed and walked away from the keyboard.

Morning showed the same AssertionError on the server, because that other terminal had never been the free server. Have you noticed how easily we conflate "I ran this" with "the environment that matters ran this"? I had done exactly that, and the chat transcript still looked like a successful review.

What I actually typed

# local laptop — the terminal I forgot was local
pytest tests/test_billing.py -q
# illustrative output I treated as proof
# ....                                                                 [100%]

# the server that actually hosted the loop
pwd
# /home/runner/work/billing
ls -l src/billing.py tests/test_billing.py
sha256sum src/billing.py

The two trees were cousins, not twins, and both used the same relative paths. Chat had been talking about src/billing.py as if that string could only mean one inode. Why do we still let path strings stand in for file identity when hashing is one command?

Hours 8–20: I blamed pytest collection

Once I finally ran pytest on the server, the output went quiet in a different, nastier way. Collection printed no node ids I recognized, and the summary still felt like a pass if you skimmed it. Did the test module fail to import, or did the collector refuse the filename? I have a bad habit of dropping sketches named testBilling.py onto a box and then wondering why discovery stays empty.

pytest --collect-only -q
echo "exit=$?"
# illustrative: empty collection, pytest exit code 5

# tests/test_billing.py — collector only sees test_*.py / *_test.py
def test_normalize_cents_to_dollars():
    from src.billing import cents_to_dollars
    assert cents_to_dollars(1200) == "12.00"

I renamed the file and still had a red test, which was honest progress at last. A collected failure is a gift because it names a node id you can argue with. A silent collection is a lie of omission, and a model will happily summarize that silence as success if you let the paste stay vague.

Field note: never accept "passed" without node ids

• Require pytest --collect-only -q to print at least one tests/...::test_... line.
• Reject a summary that only says passed, ok, or 100% with no node ids attached.
• Compare collected node ids against git ls-files 'tests/test_*.py' '*_test.py'.
• If the model only edited tests/, treat the run as failed even when the suite is green.
• Keep pwd in the log so the next paste cannot hide a different clone.

Hours 20–36: sha256sum became the code review

This is the part I would repeat on any agent loop, free model or otherwise, because chat is not a filesystem. Before I ask for a change, I snapshot the files I actually care about on the server. After the model claims it is done, I snapshot again on that same pwd. If the digest did not move, the patch did not happen, no matter how pretty the unified diff looked in the transcript.

# tools/snapshot.sh — run on the same server that will execute pytest
set -euo pipefail
mkdir -p .agent-gate
git ls-files -z -- src tests | sort -z | xargs -0 sha256sum \
  > .agent-gate/before.txt
echo "wrote $(wc -l < .agent-gate/before.txt) hashes"

# after the model says the file is fixed
git ls-files -z -- src tests | sort -z | xargs -0 sha256sum \
  > .agent-gate/after.txt
diff -u .agent-gate/before.txt .agent-gate/after.txt || true
git status --short src tests

When diff printed nothing, I finally understood the weekend in one glance. The model had described an edit with perfect hunk headers, and nobody had opened the file with a write. Would you approve a human pull request that only existed in the cover letter? I would not, and I should not have approved it from a model either.

A small Python gate I now run first

The shell snapshot is enough for a panic check, but I wanted a refusal that pytest itself could not talk over. The script below exits nonzero unless hashes moved, collection returned real node ids, and production code was not left untouched. Treat it as a runnable example in the repo, not as a performance claim.

# tools/agent_gate.py
"""Refuse 'the model is done' unless disk moved and pytest collected tests."""

from __future__ import annotations

import hashlib
import json
import subprocess
import sys
from pathlib import Path

ROOT = Path(__file__).resolve().parents[1]


def digest_tree(*parts: str) -> dict[str, str]:
    mapping: dict[str, str] = {}
    for part in parts:
        base = ROOT / part
        for path in sorted(base.rglob("*.py")):
            rel = str(path.relative_to(ROOT))
            mapping[rel] = hashlib.sha256(path.read_bytes()).hexdigest()
    return mapping


def collect() -> list[str]:
    proc = subprocess.run(
        [sys.executable, "-m", "pytest", "--collect-only", "-q"],
        cwd=ROOT,
        text=True,
        capture_output=True,
    )
    return [line.strip() for line in proc.stdout.splitlines() if "::" in line]


def main() -> int:
    snap_path = ROOT / ".agent-gate" / "before.json"
    after = digest_tree("src", "tests")
    if not snap_path.exists():
        snap_path.parent.mkdir(parents=True, exist_ok=True)
        payload = json.dumps(after, indent=2, sort_keys=True) + "\n"
        snap_path.write_text(payload)
        print("snapshot written; rerun after the supposed edit")
        return 2

    before = json.loads(snap_path.read_text())
    changed = sorted(key for key in after if before.get(key) != after[key])
    added = sorted(set(after) - set(before))
    removed = sorted(set(before) - set(after))
    nodeids = collect()

    print("changed:", changed or "(none)")
    print("added:", added or "(none)")
    print("removed:", removed or "(none)")
    print("collected:", nodeids or "(none)")

    if not changed and not added:
        print("refusing: working tree hashes did not move")
        return 1
    if not nodeids:
        print("refusing: pytest collected zero node ids")
        return 1
    touched = changed + added
    if touched and all(path.startswith("tests/") for path in touched):
        print("refusing: only test files changed; production code untouched")
        return 1
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

python tools/agent_gate.py   # first call writes the snapshot
# ... model claims it patched src/billing.py ...
python tools/agent_gate.py   # second call must print real changed paths
rm .agent-gate/before.json   # reset between loops or you will compare stale maps

Decision table I wish I had on hour one

Chat claim	Hash moved?	Collected node ids?	Only tests/ touched?	What I do
"I updated src/billing.py"	No	Irrelevant	Irrelevant	Refuse; demand a write, not a story
"pytest passed"	Yes	None	No	Refuse; collection never saw tests
"fixed the assertion"	Yes	Yes	Yes	Refuse; the bug probably still lives in src/
"rounding is corrected"	Yes	Yes	No	Run the real pytest, then read the failure
"nothing to change"	No	Yes	n/a	Believe git, not the reassurance

Does that table feel heavy for a one-file bug? It is lighter than another night of reviewing a diff that never touched disk.

Hours 36–48: what broke, and what I would repeat

What broke

1. I let a rendered hunk stand in for open(path, "w"), which is how chat and disk drifted apart.
2. I ran pytest on the laptop clone, then pasted that green summary next to a server conversation.
3. I accepted a filename the collector does not discover, so "quiet" looked like "green" under a skim.
4. I allowed a test-only edit to count as a fix, which is the cheapest way for a model to silence an assertion.

What I would repeat

1. Snapshot hashes on the same host and pwd that will run pytest, before anyone talks about a patch.
2. Require collected node ids that match git ls-files, not a prose sentence that claims success.
3. Refuse test-only diffs when the production module was the thing that raised.
4. Reset .agent-gate/before.json at the start of every loop so yesterday's map cannot bless today's no-op.

Would I still use a free model on a free server for this kind of scratch work? Yes, because the remote box made the hash mismatch visible instead of letting my laptop keep lying. The useful part was not clever prompting. The useful part was refusing to close the loop until disk, collection, and git told the same story.

Limitations, and who should skip this

This gate does not prove the new logic is correct; it only proves that bytes moved and that pytest saw tests. A model can still change src/billing.py in a vacuous way, or weaken an assertion while also touching production code. Hashing will not save you from that, and neither will a green collect-only run.

Do not put secrets, production credentials, or customer data on a free shared scratch server, even for a weekend experiment. Do not use this workflow as an incident tool when you need an isolated, audited environment with a named owner. If your assistant already has a verified write tool and you check git diff every time, you may not need agent_gate.py at all.

Skip the approach if you cannot run sha256sum or pytest on the same machine the model claims to have edited. Skip it if your tests live outside tests/ and you have not adapted the paths. Skip it if you need guaranteed model quality, quotas, or hardware details I am not going to invent here.

The artifact is the gate, not the chat. If you copy anything, copy tools/agent_gate.py and the decision table, then keep them in review even when the model sounds sure. If you need a remote scratch box for that same hash-and-collect loop, I used the free server option so the laptop terminal could not sneak back into the story.