48-Hour Field Notes: I Stopped Treating the Reply as the Run

I opened a fresh notebook because a polished reply kept arriving before any process on my side had started. The text looked complete, named a command, and even described a passing check that I had not executed. Why was I ready to copy that sentence into the log as if it were an observation? A chat transcript is not a process table, and it cannot show which binary actually ran.

That gap is the whole job for these notes, not a product tour and not a scoreboard. I wanted one place to ask a model for a patch, and another place to run a narrow command where I could read the exit myself. If those two records disagree, which one deserves the green mark in my notes? I wrote the rule down before I asked for another suggestion, because a fluent paragraph can impersonate a result.

What I tried in the first twelve hours

I started by splitting the work into two witnesses that are not allowed to grade each other. The first witness is a free model session, useful when I want a draft command or a short explanation without opening a paid console. The second witness is a free server session, useful when I need a shell that is not my laptop and not the model's imagination. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I am using those two availability claims only, and I am not treating a marketing page as a measured lab.

I did not record a model name, a quota, a machine size, or a promise that the free option will still look the same next month. Those details change, and inventing them would make these notes lie with a straight face. What I can repeat is the order of operations, which is dull on purpose and hard to fake. Ask, capture, hash the text, run an allowlisted command, then write the exit code beside the hash.

Here is the question I kept asking whenever the reply sounded finished. Did any process outside the chat actually start, and did it finish with a code I can quote? If I cannot answer that, the suggestion stays in a draft folder, even when the wording is clean. A free server is not magic here, because a free shell can still be the wrong tree, the wrong user, or a job that never left the queue.

The notebook I would actually rerun

The artifact below is a small recorder, not a benchmark and not a claim that I timed a vendor. I labeled the sample paths as examples, and I would run them on whatever host I was given rather than trust a pasted success story. You can drop the file on the server, point it at an allowlist, and refuse to print a green line unless the command exits zero. Would I merge a patch because the model said this script already passed? No, I would not, and the script is written to make that laziness slightly harder.

#!/usr/bin/env python3
"""Suggestion ledger. Example workflow, not a recorded benchmark."""

import hashlib
import json
import subprocess
import sys
from datetime import datetime, timezone
from pathlib import Path

ALLOW = {
    "py_compile": [sys.executable, "-m", "py_compile"],
    "unit_check": [sys.executable, "-m", "unittest", "discover", "-s", "tests", "-q"],
}

def stamp(label, text):
    digest = hashlib.sha256(text.encode("utf-8")).hexdigest()
    return {
        "label": label,
        "sha256": digest,
        "chars": len(text),
        "at": datetime.now(timezone.utc).isoformat(),
    }

def run_allowed(key, extra):
    if key not in ALLOW:
        raise SystemExit(f"refusing unknown job: {key}")
    cmd = ALLOW[key] + list(extra)
    proc = subprocess.run(cmd, capture_output=True, text=True, timeout=120)
    return {
        "cmd": cmd,
        "code": proc.returncode,
        "stdout_sha256": hashlib.sha256(proc.stdout.encode()).hexdigest(),
        "stderr_sha256": hashlib.sha256(proc.stderr.encode()).hexdigest(),
    }

def main():
    suggestion = Path(sys.argv[1]).read_text(encoding="utf-8")
    job = sys.argv[2]
    record = {
        "suggestion": stamp("model_reply", suggestion),
        "run": run_allowed(job, sys.argv[3:]),
    }
    record["agree"] = record["run"]["code"] == 0
    print(json.dumps(record, indent=2))
    return 0 if record["agree"] else 1

if __name__ == "__main__":
    raise SystemExit(main())

The commands I actually want in the notes are boring, and that is a feature rather than a lack of ambition. I save the raw reply first, then I compile, then I decide whether a unit discovery is even on the allowlist for this host. A model can propose a piped install, and this recorder will not grow a new key just because the paragraph sounded urgent. Why give a draft the right to invent its own shell history?

mkdir -p notes/day1
# Save the raw completion yourself. Do not let the model write this file.
cp reply.txt notes/day1/reply.txt
python3 ledger.py notes/day1/reply.txt py_compile path/to/changed_module.py
echo "ledger_exit=$?"

I also keep a second command for the case where the server and my laptop do not share a tree. I print the host, the user, and the executable before I trust a zero, because a green code on the wrong machine is just a different story. This check is about identity of the run, not a feeling about how fast the free box answered.

python3 - <<'PY'
import getpass, platform, sys
print(platform.node(), getpass.getuser(), sys.executable)
PY
# Sketch only: replace the alias with the host your panel actually shows.
ssh free-host 'python3 -m py_compile path/to/changed_module.py; echo exit:$?'

I should label that ssh line as a sketch. The host alias free-host is a placeholder, not a hostname I measured, and your free server panel may name the box differently. If the panel does not give you a shell, this branch of the notes simply stops. A free model can still draft the patch, but it cannot substitute for the missing exit code.

What broke when I let the reply grade itself

The first break was social, not technical, and it showed up as soon as the paragraph said the check already passed. I had not started the server job, yet my fingers wanted to write verified next to the suggestion hash. Who was I trying to impress with that word, the future reader or my own impatience? I deleted the word and left the hash alone until ledger_exit existed.

The second break was a mismatched command that still looked related enough to fool a tired note. The reply described unittest, while the only allowlisted job I was willing to run that hour was py_compile, so a clean compile was not the test the paragraph had claimed. A zero from the compiler does not prove the tests, and a model's summary does not prove the compiler ran. I started writing the job key in the same line as the claim, so a later reader can see the gap without guessing.

The third break was environmental, and it is the one I would repeat the warning for. A free server can boot with a different working directory, a different user, or a queue that never dispatched the command I thought I had sent. If the ledger file is missing, I do not infer success from silence, because silence is how a dropped job disguises itself. Did the process start, and did it finish, or did I only receive a sentence about finishing?

A decision table I can reuse tomorrow

I do not want a score that pretends to measure model quality, because I never built that experiment and I will not fake the rows. This table only decides whether my notes may say ran, or must say draft. The columns are evidence I can point at, not feelings about how confident the wording sounded.

What I hold	Reply only	Server log only	Both, and codes match the claim
Note status	draft	incomplete	candidate
May I say verified?	no	no	only for that job key
Next action	save hash, do not patch	rerun or fix the host	read the diff, then decide
Failure I expect	fluent false pass	wrong tree or no dispatch	still review the diff

A candidate is not a merge. It means the command I allowlisted exited in a way that matches the narrow claim, on a host I can name in the note. If the reply promised tests and I only compiled, the row stays incomplete even when the compiler was happy. Would I widen the allowlist mid-session because the model asked nicely? No, that change belongs in a separate note, after I have read the new command.

What I would repeat, and what I would not

I would repeat the capture step, the allowlist, and the refusal to write verified without ledger_exit in the same folder. I would also repeat the host line, because a free server is only useful if I can tell it apart from my laptop and from the chat. The free model access is the part I would keep for drafting, especially when I want another wording of a failing traceback before I open a paid seat. The free server option is the part I would keep for the run, because a suggestion that never meets a shell is still just a suggestion.

I would not repeat any sentence that quotes a quota, a GPU name, or a latency number I did not measure on that day. I would not pipe a model command into a shell, and I would not let the recorder accept arbitrary arguments from the reply text. I would not use this notebook to judge which vendor is best, because two availability switches are not a leaderboard. If your work needs a private dataset, a signed production host, or a change window with an on-call owner, do not rehearse that change on a free shared server just because the reply looked ready.

People who should skip this approach are easy to name, and I would rather name them than pretend the ledger fits every team. Skip it if you cannot read the server log yourself, or if policy forbids sending even a redacted traceback to an outside model. Skip it if the patch touches credentials, production data, or a service you cannot roll back before the next coffee. Skip it if you need a compliance attestation, because a field notebook is not an audit, and a free tier is not a contract I am willing to invent.

Limits I am leaving in the note

This workflow does not prove that the model understood the code, and it does not prove that the server matches production. A timeout of 120 seconds in the example is a guard for the sketch, not a service level I negotiated with anyone. py_compile catches syntax, not behavior, and unittest discover only helps if those tests already exist and already mean something. The hashes tell me whether the text changed, not whether the text was wise.

I also cannot tell you that the free model list or the free server button will remain in the same place, with the same limits, next week. Check the console you actually have, and write down what you see instead of copying these notes as if they were a status page. If the free server never gives you a shell, stop at the draft and run the allowlisted command somewhere you do control. A missing witness is a result, and it should look like a missing witness in the file.

If you already use MonkeyCode, the useful part is ordinary: open the free model path when you need a draft, open the free server path when you need a shell, and keep their outputs in separate files. I would still read the diff myself before I land anything, because agreement on an exit code is not agreement that the change is the one I wanted. The notebook is there so my future self can see what ran, what never started, and what I refused to call done.