DEV Community: Sam Yang

Borrowed Cycles Are Not a Baseline: A Myth-Busting FAQ

Sam Yang — Wed, 16 Sep 2026 03:54:35 +0000

A reviewer opened a Monday pull request that already claimed a full suite pass on a complimentary remote host. The comment thread treated that machine like a shared CI worker, even though nobody could name the image, the clock, or the compiler. By afternoon a second engineer tried to reproduce the failure locally, and the loaner box had already been recycled. The chat transcript was the only remaining artifact, which is a poor substitute for a recorded run.

This pattern is spreading because coding agents now sit one command away from a scratch kernel. Teams confuse a green stream of tokens with a baseline they can cite next week. The corrected mental model is simpler than the folklore: borrowed cycles are a notepad, and evidence is a bundle you export before the box evaporates.

Myth: a green loaner run is continuous integration

The claim sounds reasonable in Slack. An agent opened a shell, installed dependencies, invoked the test runner, and pasted a summary that ended with a zero exit. People then write “CI is green on the remote box,” as if the complimentary host were a named worker with a pinned image and retained logs. That sentence smuggles three unearned properties: identity of the machine, retention of artifacts, and a contract that the same tree will be rebuilt tomorrow.

A hotel key is not a house deed, even when the door opens on the first try. Continuous integration is a recorded contract among a commit SHA, an image digest, a command, and stored outputs. A free remote shell is usually none of those things unless you wrap it. The agent’s success message is closer to a witness statement than to a junit file stored beside the commit.

You can see the difference with commands that CI systems treat as mundane and loaner boxes often skip. The following block is a proposed snapshot, not a claim about any particular vendor image.

# Proposed local or remote snapshot — run inside the repo worktree.
git rev-parse HEAD
git status --porcelain=v1
git diff --stat HEAD
uname -a
python3 --version
${CC:-cc} --version | head -n 1
test -f poetry.lock && sha256sum poetry.lock
test -f package-lock.json && sha256sum package-lock.json

If those lines were never captured, the “green” result cannot be replayed. A later checkout may match the path and still miss the lockfile, the toolchain, or an uncommitted fixture the agent wrote only on the loaner disk. Treating that gap as CI noise is how merge queues absorb folklore.

Myth: a free model endpoint is a frozen evaluation baseline

The second claim appears when teams start weekly agent evals. They point a prompt corpus at whatever complimentary model access is currently advertised, then compare this Wednesday’s patch quality with last Wednesday’s. The spreadsheet looks quantitative. The axis on the chart is unlabeled time, not a pinned model snapshot, so the comparison is mostly weather.

Baselines need three pins that marketing pages do not provide: the model identifier as the server actually received it, the decoding settings, and the tool schema hash. Without those, a “better” week may be a quieter sampler, a shorter context, or a different hidden system prompt. The scientific analogy is calibrating a scale after someone swapped the weights and kept the same sticker on the bench.

A small provenance record makes the missing pins visible. Label this as a schema proposal you can write beside any agent run, including one that used free model access.

{
  "git_sha": "REPLACE_WITH_rev_parse",
  "tree_dirty": true,
  "prompt_sha256": "",
  "tool_schema_sha256": "",
  "model_id_reported": "unspecified",
  "endpoint_host": "unspecified",
  "temperature": null,
  "max_tokens": null,
  "started_at_utc": "",
  "finished_at_utc": "",
  "junit_path": "artifacts/junit.xml",
  "patch_path": "artifacts/agent.patch"
}

Fill empty strings from the run, not from memory. If the product UI does not disclose a model identifier, write unspecified rather than inventing a brand name. Honesty in the JSON is more useful than a confident guess that future readers will treat as ground truth.

Some teams rehearse that export step with open coding-agent projects. MonkeyCode currently offers free model access and a free server option that can host an exploratory loop while you practice the bundle. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The recorder below does not require that project; it needs a shell, a git tree, and a writable artifacts directory.

Myth: an ephemeral workspace means you can skip pinning the tree

The third claim is a cousin of the second. Engineers say the complimentary server will be wiped, so there is no point in recording git status or copying the patch off the box. Ephemerality is then used as a reason to keep less evidence, which is the opposite of what a lab notebook would do with a disposable bench.

Scratch paper still needs a photograph if the calculation matters. Before logout you want three files that survive the recycle: the exact diff the agent produced, the test XML or log, and the provenance document. Those files turn a vanished kernel into something a teammate can apply on a laptop.

#!/usr/bin/env bash
# record_run.sh — proposed recorder. Review before running on a shared host.
set -euo pipefail
ROOT="$(git rev-parse --show-toplevel)"
OUT="${ROOT}/artifacts/run-$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "${OUT}"

{
  echo "git_sha $(git rev-parse HEAD)"
  echo "git_status"
  git status --porcelain=v1 || true
  echo "uname $(uname -a)"
  echo "python $(python3 --version 2>&1)"
} > "${OUT}/env.txt"

git diff HEAD > "${OUT}/agent.patch" || true

if [[ -f "${ROOT}/.pytest_cache" ]] || command -v pytest >/dev/null; then
  pytest -q --junitxml="${OUT}/junit.xml" || true
fi

python3 - <<'PY' "${OUT}"
import hashlib, json, pathlib, sys
out = pathlib.Path(sys.argv[1])
prompt = pathlib.Path("prompt.txt")
def sha(p):
    return hashlib.sha256(p.read_bytes()).hexdigest() if p.exists() else ""
doc = {
    "git_sha_line": (out / "env.txt").read_text(encoding="utf-8").splitlines()[0],
    "prompt_sha256": sha(prompt),
    "patch_sha256": sha(out / "agent.patch"),
    "junit_present": (out / "junit.xml").exists(),
}
(out / "provenance.json").write_text(json.dumps(doc, indent=2), encoding="utf-8")
print(out / "provenance.json")
PY

Keep prompt.txt next to the repo root when you start the agent so the hash is not reconstructed from chat later. If pytest is absent, the script still stores the patch and environment, which is already more than a streamed “all tests passed.” A reviewer can then run git apply artifacts/run-*/agent.patch on a known laptop image and believe or reject the claim.

Myth: free tokens plus an open agent equal an audit trail

Open source helps you read the client. It does not automatically archive what happened on a particular Tuesday. People repeat the opposite because the repository is public, the allowance is complimentary, and the server is described as free, which feels like transparency. Transparency of code is not retention of a run.

An audit trail answers who invoked which tool against which tree with which bytes out. Chat products optimize for continuity of conversation, not for those bytes. If your compliance question is “what did we merge and why,” you still need the patch, the test artifact, and the hashes. The license on the agent cannot generate those files after the working directory is gone.

A tiny checker makes the absence obvious instead of polite. This example is executable once provenance.json exists; it does not grade patch quality.

# check_provenance.py — fail closed when evidence is missing.
from pathlib import Path
import json, sys

def main(path: str) -> int:
    doc = json.loads(Path(path).read_text(encoding="utf-8"))
    missing = [k for k in ("git_sha_line", "patch_sha256") if not doc.get(k)]
    if not doc.get("junit_present"):
        missing.append("junit_present")
    if missing:
        print("incomplete provenance:", ", ".join(missing))
        return 2
    print("provenance bundle looks complete")
    return 0

if __name__ == "__main__":
    sys.exit(main(sys.argv[1]))

Exit status two is more honest than a green paragraph in the agent transcript. Teams that adopt the checker usually discover that half of their “successful” loops never wrote junit at all, which is an evidence problem rather than a model-quality problem.

When borrowed cycles still help

Complimentary remote execution remains useful when the question is exploratory. You may want to know whether a refactor even parses, whether a flaky test blows up under a second architecture, or whether an agent can navigate a large tree without filling a laptop SSD. Those are notepad questions. They become merge questions only after the recorder has run and a human has replayed the patch on a machine the team actually owns.

Think of the free server as a wind tunnel you do not calibrate. You can learn that a shape is noisy. You should not publish the airspeed as a certified number. The same caution applies to free model access: it is a way to exercise prompts and tools, not a reason to skip pins.

If you want a scratch box for record_run.sh, the free server option mentioned above is one place to try the export without buying hardware first. Check the project’s current docs for whatever allowance is offered that week, because complimentary tiers change and this article will not freeze a quota.

Limitations, and who should not use this approach

The recorder does not make a shared kernel safe for secrets. Do not export .env files, cloud keys, or production dumps onto a complimentary host, and do not paste customer data into a free model endpoint. The script also does not replace a real CI system with image digests, signed logs, and retention policies.

Skip this workflow if you are in a regulated environment that already forbids unsanctioned runners. Skip it if you need performance numbers; nothing here is a benchmark, and no latency from a free pool should be cited as capacity planning. Skip it if the team will not replay patches locally, because the JSON file then becomes another unread attachment.

Borrowed cycles can shorten the path to a hypothesis. They cannot, by themselves, tell you what landed in the tree. Treat the complimentary box as a bench you photograph, not as the laboratory whose calibration certificate you file.

The Free Allowance Is Not the Budget: Myth-Busting the Real Cost of AI Coding Loops

Sam Yang — Mon, 14 Sep 2026 21:38:49 +0000

A reviewer once sent back a pull request with three different fixes for the same flaky parser, all of them green in CI. The agent had been free to run, so nobody had counted anything, and the repo quietly paid the difference in review time. That mismatch between what the meter shows and what the work costs is the subject of this article. It is also the myth most developers repeat without checking: that a free token allowance makes AI-assisted coding free.

The corrected mental model is narrower and more useful. A token allowance caps one input to one step, while real cost accumulates across loop iterations, rework, and environment drift that no dashboard displays by default. If you want to argue about tiers, you first need a harness that measures the loop rather than the price tag. What follows is that harness, five claims checked against it, and the situations where the whole approach is the wrong tool.

Start with a measurement, not an opinion

The harness runs the same task against the same pinned commit several times and records what actually happened, including failures, which are usually the interesting rows. It talks to any OpenAI-compatible endpoint, so the tier you are evaluating stays a configuration detail rather than an assumption baked into the script. Platforms that offer free model access, including MonkeyCode, just become another base URL you can point it at.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Create a scratch copy of the repository and export three variables before running anything. The model identifier is deliberately left as a placeholder, because the right value depends on what your provider currently serves and you should read that from their own documentation.

git clone <your-repo> sandbox && cd sandbox
export LLM_BASE_URL="https://<provider-host>/v1"   # any OpenAI-compatible endpoint
export LLM_API_KEY="<your-key>"
export LLM_MODEL="<model-id-you-are-evaluating>"

python3 run_matrix.py \
  --repo ./sandbox \
  --commit "$(git -C sandbox rev-parse HEAD)" \
  --prompt prompts/fix-flaky-parser.md \
  --test "pytest -q tests/test_parser.py" \
  --repeats 5

The script below resets the worktree, asks the model once, tries to apply the returned patch, and records the diff shape alongside the exit code of your test command. Keeping temperature at zero reduces one source of noise, though it does not eliminate provider-side nondeterminism.

#!/usr/bin/env python3
"""run_matrix.py: run one task N times and record what the loop really cost."""
import argparse, csv, json, os, re, statistics, subprocess, time, urllib.request

def ask(prompt: str):
    body = json.dumps({
        "model": os.environ["LLM_MODEL"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        os.environ["LLM_BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
                 "Content-Type": "application/json"},
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=600) as resp:
        payload = json.load(resp)
    usage = payload.get("usage", {}) or {}
    usage["seconds"] = round(time.monotonic() - started, 2)
    return payload["choices"][0]["message"]["content"], usage

def apply_patch(repo: str, text: str) -> int:
    blocks = re.findall(r"```

(?:diff|patch)\n(.*?)

```", text, re.S)
    if not blocks:
        return -1  # the model answered with prose instead of a patch
    done = subprocess.run(["git", "-C", repo, "apply", "--whitespace=nowarn", "-"],
                          input=blocks[-1], text=True, capture_output=True)
    return 0 if done.returncode == 0 else -2

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", required=True)
    ap.add_argument("--commit", required=True)
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--test", required=True)
    ap.add_argument("--repeats", type=int, default=3)
    ap.add_argument("--out", default="loop-cost.csv")
    args = ap.parse_args()

    prompt = open(args.prompt).read()
    rows = []
    for run in range(1, args.repeats + 1):
        subprocess.run(["git", "-C", args.repo, "checkout", "--force", args.commit], check=True)
        subprocess.run(["git", "-C", args.repo, "clean", "-fdx"], check=True)

        text, usage = ask(prompt)
        applied = apply_patch(args.repo, text)
        diff = subprocess.run(["git", "-C", args.repo, "diff", "--numstat"],
                              text=True, capture_output=True).stdout
        tests = subprocess.run(args.test, cwd=args.repo, shell=True, capture_output=True)

        rows.append({
            "run": run,
            "seconds": usage.get("seconds"),
            "prompt_tokens": usage.get("prompt_tokens"),
            "completion_tokens": usage.get("completion_tokens"),
            "apply_code": applied,
            "files_touched": len([line for line in diff.splitlines() if line.strip()]),
            "lines_changed": sum(int(n) for n in re.findall(r"^(\d+)\t", diff, re.M)),
            "test_exit": tests.returncode,
        })

    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    print(open(args.out).read())

if __name__ == "__main__":
    main()

The CSV header alone tells you which questions become answerable: run,seconds,prompt_tokens,completion_tokens,apply_code,files_touched,lines_changed,test_exit. Treat the values you collect as yours and nobody else's, because they describe one repository, one prompt, and one test command. A quick summary keeps the discussion grounded in medians rather than a single lucky run.

python3 - <<'PY'
import csv, statistics as st
rows = list(csv.DictReader(open("loop-cost.csv")))
print("runs:", len(rows))
print("pass rate:", sum(r["test_exit"] == "0" for r in rows) / len(rows))
print("median seconds:", st.median(float(r["seconds"]) for r in rows))
print("median lines changed:", st.median(float(r["lines_changed"]) for r in rows))
PY

Five claims, checked against the harness

Claim one: a token allowance is a spending cap. An allowance bounds one dimension of one request, and agent loops multiply that dimension by the number of attempts plus every retry your wrapper adds. The corrected model is that tokens behave like a mileage limit on a single trip while your costs live in the number of trips. If you add automatic retries, log them, because two retries are three requests and the meter will not tell you that.

Claim two: cheaper tokens mean cheaper work. Loop count and human review dominate the bill long before per-token price does. A slow run that produces a two-line patch often costs less attention than a fast run that rewrites four files, even though both passed the test command. This is why the harness records files_touched and lines_changed next to timing, and why the pair is more informative than either number alone.

Claim three: one successful run proves the tier can do the task. A single green run is a coin flip with good marketing, and the interesting signal is variance across repeats. Five runs that pass with wildly different patch shapes tell you the task is underspecified, not that the model is unreliable. Fix the prompt before blaming the endpoint, since the harness resets the worktree and cannot reset an ambiguous instruction.

Claim four: if tests pass, the patch is done. Test outcomes are necessary evidence and never sufficient evidence, because your suite only measures what someone already thought to assert. The harness surfaces this cheaply: when apply_code is -1, the model answered with prose, which is a prompt problem, and when test_exit is zero on a large diff you should read the patch before celebrating.

Claim five: the run on a hosted server equals the run on my laptop. Environment identity is part of the experiment, and a container with a different toolchain version can change both the patch and the result. Record the kernel, language runtime, and lockfile hash with each batch, or your comparison measures two variables at once. Recent debates about whether agent wrappers amount to conditionals miss this point, since the interesting difference is rarely the wrapper.

Where the free options actually fit

A harness is only persuasive when you can afford repetitions, and repetitions are exactly what per-request pricing discourages during exploration. Free model access plus a free server option, as offered by MonkeyCode, is one way to run five or ten repeats of a throwaway task without turning a measurement into a purchase decision. Verify the current terms yourself, including what the allowance covers and how long a server persists, because capacity and conditions change and this article cannot promise them. The honest framing is that these options lower the cost of finding out, not the cost of shipping.

Situation	Free model access sufficient?	Free server sufficient?	Reason
One-off refactor on a side repo	Often yes	Usually unnecessary	Few iterations, low review stakes
Nightly regression harness	Often yes	Yes if jobs fit the session limits	Volume matters more than latency
Agent on every pull request	Depends on review policy	Only with artifact retention	A lost log makes a failed run unfalsifiable
Code under residency or contract rules	Usually no	Usually no	Sending the diff offsite may be prohibited
Multi-hour interactive session	Depends on interruption limits	Rarely	State loss mid-task defeats the exercise

Limitations, and who should skip this

The harness measures one task shape against one test command, so it is not a benchmark and should never be quoted as one. Token fields come from the provider's usage block, and endpoints that omit it will leave those columns empty rather than wrong. A five-run sample cannot separate model quality from prompt quality, and comparing runs across different days quietly mixes in version changes you did not record.

Skip this approach if your work is subject to data-residency or contractual constraints that forbid sending source code to a third-party endpoint, because no amount of measurement fixes a policy violation. Skip it as well if you need deterministic, bit-identical reproducibility for compliance evidence, since hosted inference does not offer that guarantee. If you are optimizing interactive latency rather than loop cost, a batch harness answers the wrong question entirely.

If you already have a repository and a test command, the cheapest next step is to run five repeats before arguing about tiers again, then read the median row rather than the best one.

The Tool Succeeded, the Repo Did Not: A Myth-Busting FAQ

Sam Yang — Sun, 13 Sep 2026 19:58:59 +0000

The standup recording still had the spinner on screen when the engineer muted the call. Twenty tool calls had completed in the shared log, and every status line said success. The repository on disk told a quieter story: git status printed nothing but a clean working tree. That gap between a busy agent trace and an unchanged index is the myth this FAQ is built to break.

Teams that adopted coding agents this year often score a session by how long the spinner moved. The analogy is a kitchen that sounds busy because every drawer is opened, while the oven stays cold. Tool success is an I/O event, not evidence that a behavior in the codebase changed. Until a diff, a test, or a failing assertion moves, the loop has only grown context.

A practical way to keep that distinction honest is a small trace ledger that classifies each step before anyone praises the model. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option matter only as a place to reproduce the ledger. They are not a substitute for review, and they do not make a busy trace equivalent to a patch. The examples that follow are labeled procedures, and they are not production benchmarks from this account.

Myth: a successful tool call means the agent understood the system

When an MCP server or a shell wrapper returns a 200 and a JSON blob, people narrate that as comprehension. The corrected model is simpler and less flattering: the process received bytes and appended them to the window. Comprehension would show up later as a minimal diff that a human can explain without the log. If those bytes never constrain the next edit, the call was tourism through the filesystem.

Consider a labeled session in which list_directory and read_file both return quickly on a service that talks MCP. The trace looks healthy because every envelope has a result key and no exception. That is the same evidence a curl gives when a CDN answers; it is not evidence that the handler behind the URL is the handler you meant. Treat the tool result as untrusted input until a later step mutates the tree or a test assertion flips.

# labeled procedure: snapshot the tree before praising any tool result
git status --porcelain
git diff --stat HEAD
git rev-parse HEAD

If those three commands are identical before and after a dozen “successful” calls, the agent did I/O, not software change. The mental model is a mailroom, not a senior engineer: packages arrived, labels were scanned, and nothing was filed into the product.

Myth: extra loop iterations are free when the server is free

A complimentary machine still meters context, because every retry concatenates prior tool output into the next prompt. The free server removes a credit-card form; it does not remove the cost of unread logs sitting in the window. Treating retries as experiments only works when each retry starts from a recorded snapshot, not from an inflated transcript. Otherwise the second hour is spent rereading the first hour under a different spinner.

The analogy that holds is a whiteboard that nobody erases. Adding another equation is cheap in isolation, and then the board becomes unreadable, so the next visitor copies the clutter. Free-tier loops fail the same way: the hardware bill is zero while the prompt bill is a pile of stack traces, directory listings, and repeated plans. Stop the loop when the new step cannot name a file, a test, or a commit hash that the previous step lacked.

# labeled procedure: bound a retry by whether the input actually changed
BEFORE=$(git write-tree)
# ... one agent iteration ...
AFTER=$(git write-tree)
test "$BEFORE" = "$AFTER" && echo "retry produced no tree change"

A free server that still has your process is not the same object as a clean experiment. If you need isolation, snapshot the worktree; do not assume the next iteration is a new machine just because the product page said free.

Myth: a written plan is already the patch

Agents on free models often emit numbered steps that read like a design review. Prose that names files is not the same object as a hunk that Git can apply. Keep a compiler in mind: the plan is commentary, and the only artifact that ships is the object file, here a diff. If the session ends with a plan and a clean index, the compiler never ran.

This myth spreads because language models are fluent, and fluency looks like commitment. A paragraph that says “update invoice.py to reject negative totals” can be pasted into a ticket, and it still does not reject a negative total. Ask for the patch as if you were talking to a build system: no credit for intended object files. The review starts at git diff, not at the bullet list in the chat transcript.

# labeled procedure: refuse to review prose that never became a hunk
git diff --cached --exit-code; echo "index_exit:$?"
git diff --exit-code; echo "worktree_exit:$?"

Both exit codes zero means there is nothing to review, regardless of how confident the plan sounded. File the transcript as notes if it helps the next human; do not file it as a change.

Myth: rerunning the same red test is a new experiment

Re-running a failing test without changing fixtures, seed, or code is a heartbeat, not an experiment. The log will look industrious because the runner prints the same traceback again. Charge that step to confirmation, then stop the loop unless the input changed. Otherwise the free-tier window fills with duplicate stacks that crowd out the one file that needed an edit.

A useful analogy is a smoke alarm with a drained battery. The chirp is data the first time, and it is noise the twelfth time, even though each chirp is a “successful” sensor read. Duplicate test output should collapse into a single fingerprint. If the fingerprint is unchanged, the loop owes you a mutation or a halt, not another green tool envelope.

# labeled procedure: fingerprint the failure so duplicates are visible
pytest -q tests/test_invoice.py --tb=short -p no:cacheprovider \
  | tail -n 40 | sha256sum

Store that hash beside the git tree hash. Two matching pairs in a row means the agent is polling, not debugging.

Artifact: a trace ledger that scores I/O against mutations

The workflow is deliberately boring, which is the point. Export each agent step as JSONL, then score it with a local script that cannot be impressed by spinners. The schema below is a proposal, not a vendor format, and you can adapt field names to whatever your runner already prints.

{"ts":"2026-09-14T10:02:11Z","kind":"tool_result","name":"list_directory","ok":true,"bytes":4812,"tree":"unchanged"}
{"ts":"2026-09-14T10:02:19Z","kind":"tool_result","name":"read_file","ok":true,"bytes":2201,"tree":"unchanged"}
{"ts":"2026-09-14T10:03:04Z","kind":"model_message","name":"plan","ok":true,"bytes":1604,"tree":"unchanged"}
{"ts":"2026-09-14T10:03:40Z","kind":"tool_result","name":"apply_patch","ok":true,"bytes":388,"tree":"mutated"}
{"ts":"2026-09-14T10:04:12Z","kind":"tool_result","name":"pytest","ok":false,"bytes":933,"tree":"unchanged"}

# labeled example: classify a session without treating tool ok as progress
from __future__ import annotations

import json
from pathlib import Path

WORK = {"apply_patch", "write_file", "git_commit", "edit"}
CHECKS = {"pytest", "npm_test", "cargo_test", "lint"}

def score(path: str) -> dict:
    io_ok = 0
    mutations = 0
    check_flips = 0
    duplicate_bytes = 0
    seen = set()
    last_check = None
    for line in Path(path).read_text(encoding="utf-8").splitlines():
        ev = json.loads(line)
        kind = ev.get("kind")
        name = ev.get("name", "")
        blob = (name, ev.get("bytes"), ev.get("ok"), ev.get("tree"))
        if blob in seen:
            duplicate_bytes += int(ev.get("bytes") or 0)
        seen.add(blob)
        if kind == "tool_result" and ev.get("ok"):
            io_ok += 1
        if ev.get("tree") == "mutated" or name in WORK:
            mutations += 1
        if name in CHECKS:
            fingerprint = (name, ev.get("ok"), ev.get("bytes"))
            if last_check is not None and fingerprint != last_check:
                check_flips += 1
            last_check = fingerprint
    return {
        "io_ok": io_ok,
        "mutations": mutations,
        "check_flips": check_flips,
        "duplicate_bytes": duplicate_bytes,
        "productive": mutations > 0 or check_flips > 0,
    }

if __name__ == "__main__":
    print(json.dumps(score("session.jsonl"), indent=2))

Run it against the sample and the numbers stay unromantic: several successful tools, one mutation, no check flip after the patch, and no excuse to call the session done. Add a human gate that refuses a merge request when productive is false, even if the chat transcript reads like a victory lap. That gate is the whole method; the script is only a scale that does not flatter the cook.

python3 score_trace.py
git diff --stat

A compact decision rule sits behind those commands. If tools succeeded and the tree is unchanged, you observed I/O. If the tree changed and no check flipped, you observed an untested edit. If a check flipped and the fingerprint is new, you observed an experiment. If none of those are true, you observed a spinner.

Limitations, and who should not use this

The ledger cannot see side effects outside Git, so database migrations, object-store writes, and remote feature flags will look like tourism. It also cannot prove that a mutation is correct; it only proves that something in the tree moved. Teams that need production SLAs, secret-scanning guarantees, or isolated tenancy should not treat a shared free server as that environment, because this article does not claim hardware isolation, quotas, or retention. Do not paste credentials into tool arguments to “help” the model, then ask a free server to forget them.

The method is a poor fit for exploratory interviews with a codebase you do not intend to change. In that setting, successful reads are the work, and a clean git tree is the desired ending. It is also a poor fit when your runner does not emit structured traces, because reconstructing JSONL from screenshots will create a second myth: that a hand-built log is the session. If you cannot export events, watch git write-tree instead and ignore the spinner entirely.

Free model access remains useful for rehearsing the ledger on throwaway branches, especially when you want a second loop without standing up local GPUs. Keep the rehearsal on a fork that contains no production secrets, and throw the worktree away when the hashes stop moving. The corrected mental model is small enough to reuse anywhere: activity is context, and review begins at the diff.

If you want a disposable place to run the same scoring loop, MonkeyCode’s free model access and free server option are one way to stage a throwaway branch. Export the JSONL, keep the gate, and let the index, not the spinner, decide whether the session existed.

A Clean Exit Is Not a Patch: A Myth-Busting FAQ

Sam Yang — Sat, 12 Sep 2026 17:58:07 +0000

The pull request opened with a cheerful summary, a green checkbox emoji, and a complete absence of changed files in the GitHub diff. The author pasted a chat in which a coding agent invoked an MCP write tool and printed JSON with ok set to true. Reviewers asked for the actual diff, yet the branch still pointed at the same tree hash as main. A quiet process exit had been treated as a mergeable artifact, which it never was.

That pattern is getting louder as teams bolt MCP servers onto agent loops and then read the loop as if it were a build system. Public threads this week keep arguing that many so-called agents are mostly control flow wearing a tool schema, and the useful part of that claim is operational. A schema is a handshake between a model and a process. A patch is a recorded mutation of one worktree, plus a test receipt another machine can replay without trusting the chat.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. The checks below mention MonkeyCode only because its free model access and free server option give you a second machine that is not your laptop. Strip those product lines out and the receipt protocol remains the whole point of the article.

Myth: structured tool success means the file landed in the clone you will merge. The evidence people cite is almost always a pretty JSON blob, because JSON looks like an API and APIs feel like side effects. The corrected model is narrower and less comforting: a tool result is a message about what some process claimed, not a git object. If the write happened on a rented workspace, a container that later recycled, or a path the agent invented, your clone can stay untouched while the transcript still reads like a victory.

You can watch the gap with ordinary git, which does not care how confident the model sounded. After any agent session that claims a write, capture identity before you argue about quality.

#!/usr/bin/env bash
# receipt_before.sh — run in the repo you actually intend to merge
set -euo pipefail
git rev-parse HEAD
git rev-parse --show-toplevel
git status --porcelain=v1
git diff --stat
find . -name '*.py' -print0 | sort -z | xargs -0 sha256sum | sha256sum

If HEAD, the porcelain status, and the content fingerprint did not move, the handshake did not produce a patch in this tree. Teams argue about style at that point, which is wasted heat, because there is nothing to review except prose. Treat the chat as a design note and start the loop again with an explicit output path inside the repository you own.

Myth: MCP grounding stops hallucination at the tool boundary. The repeated claim is that once a model can list tools, it cannot invent the next step, because the catalog is a contract. Catalogs are closer to menus. A diner can still order a dish the kitchen does not serve, and a model can still emit a tool name, a relative path, or a success payload that no server executed. MCP reduces typing mistakes. It does not turn a probabilistic decoder into a transactional filesystem.

A small Python guard makes that distinction visible without requiring a full agent framework. The script below refuses to honor a tool payload unless three boring facts agree: the name is in an allowlist, the path stays inside the repo, and the bytes on disk change.

# tool_receipt.py
from __future__ import annotations

import hashlib
import json
from pathlib import Path

ALLOWED = {"write_file", "run_tests"}

def fingerprint(root: Path) -> str:
    h = hashlib.sha256()
    for path in sorted(p for p in root.rglob("*") if p.is_file()):
        rel = path.relative_to(root).as_posix()
        if rel.startswith(".git/"):
            continue
        h.update(rel.encode())
        h.update(path.read_bytes())
    return h.hexdigest()

def accept(payload: dict, root: Path) -> dict:
    before = fingerprint(root)
    name = payload.get("name")
    if name not in ALLOWED:
        return {"accepted": False, "reason": "name-not-allowlisted", "before": before}
    rel = payload.get("path", "")
    target = (root / rel).resolve()
    if root.resolve() not in target.parents and target != root.resolve():
        return {"accepted": False, "reason": "path-escapes-root", "before": before}
    if name == "write_file":
        target.parent.mkdir(parents=True, exist_ok=True)
        target.write_text(payload.get("contents", ""), encoding="utf-8")
    after = fingerprint(root)
    return {
        "accepted": before != after,
        "reason": "mutated" if before != after else "no-byte-change",
        "before": before,
        "after": after,
    }

if __name__ == "__main__":
    sample = json.loads(Path("tool_claim.json").read_text())
    print(json.dumps(accept(sample, Path(".")), indent=2))

Label the sample payload as an unexecuted fixture unless you generated it from a real log. The interesting failure is not a Python exception. The interesting failure is accepted: false with no-byte-change, which is what you get when a model narrates a write and the disk stays still.

Myth: when the agent loop goes silent, the job is complete. Silence is an exit condition for a process, not a definition of done for a change. Loops stop because they hit a turn limit, a token budget, a cancelled stream, a parser error, or a model that learned to say "done" as a polite closer. None of those events produce a JUnit file. If you cannot point to a command, an exit code, and a log path, you watched a conversation end.

A reproducible minimum looks like this, and it is deliberately ugly.

#!/usr/bin/env bash
# receipt_after.sh — the agent does not get to skip this
set -euo pipefail
OUT="${RECEIPT_DIR:-.receipts}/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"
{
  echo "host=$(hostname)"
  echo "pwd=$(pwd)"
  echo "head=$(git rev-parse HEAD)"
  echo "branch=$(git rev-parse --abbrev-ref HEAD)"
} > "$OUT/identity.txt"
git status --porcelain=v1 > "$OUT/status.txt"
git diff > "$OUT/worktree.diff"
set +e
python -m pytest -q --junitxml="$OUT/junit.xml"
echo "pytest_exit=$?" >> "$OUT/identity.txt"
set -e
sha256sum "$OUT"/* > "$OUT/MANIFEST.sha256"
echo "$OUT"

The directory is the artifact. Chat text can go in the ticket as color commentary. If identity, diff, and junit are missing, you do not have a failing test or a passing test. You have a story about a loop that got tired.

Myth: two agreeing tool calls are a replica. Developers repeat this after an agent "double-checks" its own write by reading the file back through another tool. That is one process talking to itself, which is closer to a person rereading a draft than to a second machine. Replication starts when a different checkout, preferably on a different host, applies the diff and reruns the same command line. Agreement inside one session is echo.

This is the narrow place where a free remote server is relevant rather than decorative. Run the receipt on your laptop, then run the same scripts on a throwaway host that never held the original chat. MonkeyCode currently offers free model access and a free server option that can play that second-host role if you already work there. The product is not the proof. The proof is that the fingerprint and the pytest exit code match across machines, or that they fail in the same named way.

Myth: a live free workspace is source of truth while the session remains open. Live workspaces are scratch paper. They vanish, migrate, or get reused, and the reuse problem is worse when many users share a free pool. If the only copy of the change lives in that scratch tree, you do not have a change. You have a lease. Copy the diff and the receipt off the box before you celebrate, the same way you would copy notes off a whiteboard before the next meeting starts.

The corrected mental model fits in one sentence that teams can put in a contributing guide. An agent may propose work; only a hashed diff plus a replayable test log may claim that work happened. Everything else is drafting. That model stays useful when the tools are local, remote, paid, free, MCP-backed, or glued together with a shell loop that would look embarrassing in a architecture diagram.

Limitations follow from the same model and should be stated without theater. The receipt scripts do not detect semantic regressions, flaky tests, or a model that edited the tests until they smiled. They also do not prove that a free server is isolated, private, or appropriate for secrets, customer data, or production credentials. People who need a hardened runner, an air-gapped build, or a compliance trail should keep using the same CI they already trust and should not substitute a chat loop for that CI. People who only want a rubber stamp for an unreviewed dump of model output should not use this protocol either, because it will keep saying no.

If you already keep a spare remote session for agent work, run the receipt scripts there once and store the JSON beside the diff before anyone writes "LGTM" under a transcript.

The Second Run Was a Different Machine: A Myth-Busting FAQ

Sam Yang — Fri, 11 Sep 2026 15:42:39 +0000

A pull request arrived just after lunch with a green local script and a commit message that claimed the suite had been rerun. The reviewer opened the job log and found a hostname that did not match the laptop that sent the patch. The same prompt had been submitted twice, yet the second transcript talked about files the first run had already deleted. That mismatch is the subject of this FAQ, because agent loops make the story easy to misread.

Teams now split a coding task across a laptop, a CI runner, and a complimentary remote workspace without recording which host answered. Free model access makes another attempt feel inexpensive, so the loop continues after the working tree has already changed underfoot. The corrected mental model is blunt: a transcript is a story about some machine, not a proof about yours. The checks below turn that sentence into commands you can run before anyone merges.

When the loop runs against a free server option, the host identity problem becomes ordinary rather than exotic. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding assistant that provides free model access and a free server option for these agent loops. Remove the product name and the receipt workflow still has to exist, because the failure mode is about identity, not branding.

Myth: the same prompt means the same machine

This claim shows up whenever someone reruns a failed agent pass and reads the second transcript as a sequel. Continuity of text is not continuity of disk, clock, installed compilers, or even repository HEAD. A free server can clone a repo at a different commit, drop you in /workspace, and still quote your original relative paths with complete confidence. Ask the process to print a receipt before it is allowed to describe a file as missing, patched, or tested.

The proposed commands are intentionally boring, because boring facts are what the chat window tends to skip. Run them on the host that will execute tools, not on the laptop that composed the prompt if those hosts differ. Save the JSON beside the transcript so a reviewer can see whether "again" meant replay or relocation.

python3 session_receipt.py --label before --out /tmp/receipt-before.json
git rev-parse --show-toplevel
git rev-parse HEAD
git status --porcelain=v1
git rev-parse --abbrev-ref @{upstream} 2>/dev/null || true
uname -n
pwd

Myth: a tool result stays true until the thread ends

Chat UIs present every tool payload as a still photograph that remains hanging on the wall. Filesystems are not galleries; they are kitchens during dinner service, and later turns keep moving the pans. An agent may list routes.py in turn two, delete it in turn five, and still argue from the old listing. The corrected model is that a tool result is a timestamped observation, and observations expire when any later write can touch the same path.

A practical analogy is a restaurant ticket that still says "table four, two soups" after table four has paid and left. The ticket is authentic, and it is still about a world that no longer exists. If you need the listing, hash the file again and compare it to the earlier receipt rather than asking the model. Memory in a thread is a compression of prior tokens, not a lock on the working tree.

#!/usr/bin/env python3
"""Proposed session receipt helper. Labelled example; not a production service."""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import socket
import subprocess
import time
from pathlib import Path

SKIP_PARTS = {".git", "node_modules", ".venv", "dist", "__pycache__"}


def git(args: list[str]) -> str:
    result = subprocess.run(
        ["git", *args],
        check=False,
        capture_output=True,
        text=True,
    )
    return result.stdout.strip()


def hash_tree(root: Path) -> str:
    digest = hashlib.sha256()
    files = sorted(
        p for p in root.rglob("*")
        if p.is_file() and SKIP_PARTS.isdisjoint(p.parts)
    )
    for path in files:
        rel = path.relative_to(root).as_posix()
        digest.update(rel.encode())
        digest.update(b"\0")
        digest.update(path.read_bytes())
    return digest.hexdigest()


def main() -> None:
    parser = argparse.ArgumentParser()
    parser.add_argument("--label", required=True)
    parser.add_argument("--out", required=True)
    args = parser.parse_args()
    top = git(["rev-parse", "--show-toplevel"]) or os.getcwd()
    root = Path(top)
    payload = {
        "label": args.label,
        "captured_at_unix": int(time.time()),
        "hostname": socket.gethostname(),
        "cwd": os.getcwd(),
        "git_head": git(["rev-parse", "HEAD"]),
        "git_branch": git(["branch", "--show-current"]),
        "git_status": git(["status", "--porcelain=v1"]),
        "upstream": git(["rev-parse", "--abbrev-ref", "@{upstream}"]),
        "tree_sha256": hash_tree(root),
    }
    Path(args.out).write_text(json.dumps(payload, indent=2) + "\n")
    print(args.out)


if __name__ == "__main__":
    main()

Myth: a clean status in chat means origin moved

Agents often paraphrase git status into English that sounds like a completed delivery to production. Working tree clean is a statement about one index on one clone, not a promise that origin moved. On a free server the clone may be a shallow fetch with no remote you can push toward. The reviewer still has to compare HEAD, the upstream ref, and the reflog before trusting a summary.

python3 session_receipt.py --label after --out /tmp/receipt-after.json
git log --oneline -5 --decorate
git reflog -5
git status -sb
git rev-list --left-right --count origin/HEAD...HEAD 2>/dev/null || echo "no-origin-count"
python3 - <<'PY'
import json
from pathlib import Path
before = json.loads(Path("/tmp/receipt-before.json").read_text())
after = json.loads(Path("/tmp/receipt-after.json").read_text())
keys = ("hostname", "cwd", "git_head", "tree_sha256")
for key in keys:
    left, right = before.get(key), after.get(key)
    state = "match" if left == right else "DIVERGED"
    print(f"{key}: {left!r} -> {right!r} ({state})")
PY

If hostname or cwd diverges, the second run is not a retry; it is a different laboratory that happened to receive similar English. If git_head matches and tree_sha256 diverges, the commit name is stable while the files are not, which is a classic dirty-or-ignored-file surprise. If both hashes match and origin has not advanced, the agent may have produced a locally clean tree that nobody else can see. None of those outcomes should be blamed on model quality until the two receipts have been compared.

Myth: cheap retries replace a pinned fixture

Free model access tempts teams to treat another generation as cheaper than writing a fixture that would make the claim testable. The cost that actually grows is reviewer time, because each retry can change host and HEAD while the prompt stays frozen. A pinned fixture is a receipt plus an assertion of hostname, cwd, and HEAD, or a note that those values may change. Without that assertion, another try is not an experiment, and it remains only a new anecdote.

The small matrix below is the whole method, and you should refuse the agent claim when a check fails. This is a proposed review gate, not a measured benchmark from a production program.

Agent claim	Required check	If it fails, the claim is
Same retry as last turn	hostname and cwd equal in both receipts	a different machine telling a sequel
Tests were run here	tree hash plus a runner exit code on that host	prose about a runner you cannot name
Ready to merge	HEAD equals the reviewed commit and origin has the same SHA	a clean island
File still missing	a fresh listing after the last write	nostalgia for turn two

What this approach is not

Do not send receipts, source, or env files to a free server if your organization forbids third-party execution of that tree. The receipt proves identity, and it does not prove confidentiality, isolation, or supply-chain hygiene on its own. People working on regulated code, unpublished keys, or customer data should keep the loop on hosts they already control. They should not treat a complimentary workspace as a harmless substitute for that operational control.

The script hashes file bytes and can become slow on huge monorepos, so scope it to the package the agent was asked to touch. It also does not detect lies inside a test runner, skipped tests, or network stubs that return theatrical success. Pair it with the tests you already trust, and distrust any summary that cannot point at a receipt pair. Treat every unmatched hostname as a new investigation rather than as extra creativity from the model.

If you already run these loops on MonkeyCode's free server option, attach the before and after JSON before you accept a second-pass summary. The useful habit is the same on any host: name the machine, name the commit, and only then argue about what the model understood. A transcript without those names is still useful as a draft, and it is not yet evidence. Reviewers can reject a green-sounding story when the two receipt files cannot be produced.

The Path Matched, the Tree Did Not: A Myth-Busting FAQ

Sam Yang — Wed, 09 Sep 2026 15:55:34 +0000

On a Tuesday incident call, a backend engineer pasted an agent transcript and declared the webhook bug closed. The pasted block showed a cat of src/billing/webhook.py and a tidy explanation of a signature check. A reviewer opened the same path on main and found a different helper and an older comment. The path names matched, yet the trees under those paths were not the same checkout.

That mismatch is becoming a default failure mode as coding agents hop between laptops, clones, and loaner runtimes. A tool call can succeed against any workspace that happens to contain a similar relative path. The transcript then reads like courtroom evidence, because reviewers treat file paths as unique identity. Paths behave like room numbers on a long hotel corridor, while the building itself can still change overnight.

This FAQ collects claims that show up in review threads when an agent already read the file. Each answer restates the claim, names the missing evidence, and offers a corrected mental model. The working artifact is a binder script that a reviewer can run before arguing about prompt quality. None of the checks require a particular vendor, and the figures in comments are illustrative rather than a published benchmark.

Myth 1: a successful cat means the agent saw your tree

Developers often treat a printed file body as proof that the agent inspected the intended checkout. The analogy is a keycard that opens room 412 in every tower on the same street. The number on the door matches, while the furniture inside can still be last year's renovation. A cat or read_file result only proves that some process opened some path that shared a name.

The corrected model is boring on purpose: bind the workspace before you debate the patch. Write a nonce the agent cannot know from training data, then require that nonce in the transcript. Record the commit, the dirty state, and digests of the files the agent claims to have read. If those fields are missing, the quotation is real prose about an unnamed edition of the tree.

Reviewers can generate a receipt with a short command that names every file the agent quoted. The script refuses paths that escape the repository root, which blocks a class of copy-paste mistakes. It also records hostname and origin URL so a loaner runtime cannot hide behind a familiar relative path. Save the JSON next to the review note rather than trusting scrollback from a chat client.

#!/usr/bin/env python3
"""Bind workspace identity before trusting an agent transcript.

Proposed reviewer harness, not an attested production control plane.
Run it in the checkout you intend to merge.
"""
from __future__ import annotations

import argparse
import hashlib
import json
import os
import subprocess
import sys
import time
from pathlib import Path


def run(cmd: list[str]) -> str:
    proc = subprocess.run(cmd, check=False, capture_output=True, text=True)
    if proc.returncode != 0:
        return f"ERR:{proc.returncode}:{proc.stderr.strip()}"
    return proc.stdout.strip()


def digest_file(path: Path) -> str:
    hasher = hashlib.sha256()
    with path.open("rb") as handle:
        for chunk in iter(lambda: handle.read(65536), b""):
            hasher.update(chunk)
    return hasher.hexdigest()


def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--root", default=".")
    parser.add_argument("--nonce-path", default=".workspace-nonce")
    parser.add_argument("--claim", action="append", default=[])
    parser.add_argument("--out", default="workspace-receipt.json")
    args = parser.parse_args()

    root = Path(args.root).resolve()
    os.chdir(root)

    nonce_path = root / args.nonce_path
    if not nonce_path.exists():
        token = hashlib.sha256(os.urandom(32)).hexdigest()
        nonce_path.write_text(token + "\n", encoding="utf-8")

    nonce = nonce_path.read_text(encoding="utf-8").strip()
    claims: dict[str, dict] = {}
    for rel in args.claim:
        path = (root / rel).resolve()
        if os.path.commonpath([str(path), str(root)]) != str(root):
            print(f"refusing path outside root: {rel}", file=sys.stderr)
            return 2
        exists = path.is_file()
        claims[rel] = {
            "exists": exists,
            "sha256": digest_file(path) if exists else None,
            "size": path.stat().st_size if exists else None,
        }

    receipt = {
        "generated_at_unix": int(time.time()),
        "hostname": run(["hostname"]),
        "pwd": str(root),
        "git_head": run(["git", "rev-parse", "HEAD"]),
        "git_branch": run(["git", "branch", "--show-current"]),
        "git_status": run(["git", "status", "--porcelain"]),
        "git_remote": run(["git", "remote", "get-url", "origin"]),
        "nonce_path": args.nonce_path,
        "nonce_sha256": hashlib.sha256(nonce.encode("utf-8")).hexdigest(),
        "claims": claims,
    }
    Path(args.out).write_text(json.dumps(receipt, indent=2) + "\n", encoding="utf-8")
    print(json.dumps(receipt, indent=2))
    return 0


if __name__ == "__main__":
    raise SystemExit(main())

Run it like this against the checkout you actually intend to merge, not against a scratch clone the agent created overnight. The command is ordinary Python, and it should fail closed when a claimed path walks outside the repository. Keep the generated nonce file out of commit history if your policy treats it as review scaffolding rather than source.

python workspace_bind.py \
  --claim src/billing/webhook.py \
  --claim tests/test_webhook.py \
  --out receipt.json

Myth 2: a clean git status in the transcript equals the branch you will ship

An agent can print an empty porcelain status from a freshly cloned copy at an older commit. Cleanliness answers whether the worktree has uncommitted edits, not whether HEAD matches the pull request. Reviewers who equate that silence with current main are reading a weather report as if it were a map. Ask for rev-parse HEAD and the remote URL in the same breath as status.

A compact identity dump belongs in the same receipt, because status without HEAD is an incomplete sentence. The commands below are ordinary Git, and they should appear before any claimed patch. If the agent cannot reproduce these values, stop blaming prompt tone and name the building that holds room 412. Paste those lines into the review note so later readers do not reconstruct identity from memory.

git rev-parse HEAD
git branch --show-current
git status --porcelain
git remote get-url origin
hostname
pwd

Myth 3: rerunning the same shell line later replays the same tree

Engineers like to just run it again when a transcript looks implausible, expecting the second execution to be a scientific replicate. A shell line is not an experiment until the working directory, commit, and file digests are fixed. Otherwise the replay is a new performance of a similar script in a possibly different theater. Compare two receipts instead of comparing two vibes from scrolling terminals.

The same checkout can emit a second document after the agent finishes, which you then diff as structured data. Hostname changes, HEAD changes, and digest changes are different kinds of drift, and they should not be collapsed. A digest change on a claimed file is a content event, while a hostname change is a venue event. Mixing those two events is how teams invent a confident story about flaky models and unlucky sampling.

#!/usr/bin/env python3
"""Compare two workspace receipts. Proposed checker, not an attestation service."""
from __future__ import annotations

import json
import sys
from pathlib import Path

KEYS = ("hostname", "pwd", "git_head", "git_branch", "git_remote", "nonce_sha256")


def load(path: str) -> dict:
    return json.loads(Path(path).read_text(encoding="utf-8"))


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: compare_receipts.py left.json right.json", file=sys.stderr)
        return 2
    left, right = load(sys.argv[1]), load(sys.argv[2])
    mismatches = []
    for key in KEYS:
        if left.get(key) != right.get(key):
            mismatches.append((key, left.get(key), right.get(key)))
    left_claims = left.get("claims", {})
    right_claims = right.get("claims", {})
    for name in sorted(set(left_claims) | set(right_claims)):
        left_digest = (left_claims.get(name) or {}).get("sha256")
        right_digest = (right_claims.get(name) or {}).get("sha256")
        if left_digest != right_digest:
            mismatches.append((f"claim:{name}", left_digest, right_digest))
    if not mismatches:
        print("receipts agree on identity fields and claimed digests")
        return 0
    print("scope mismatch; do not debate the model until these fields align")
    for key, left_value, right_value in mismatches:
        print(f"{key}\n  left:  {left_value}\n  right: {right_value}")
    return 1


if __name__ == "__main__":
    raise SystemExit(main())

Porcelain status is recorded for humans and omitted from the automated comparison on purpose. A dirty tree can still be the correct SHA, and a clean tree can still be the wrong SHA. If you compare status lines as if they were hashes, you will file identity bugs as style nits. Keep the comparison boring, and let Git own the question of commit identity.

Myth 4: parking the job on a free server cannot change workspace identity

Teams sometimes move agent work onto a complimentary remote host because local laptops are noisy during incident weeks. That move is reasonable for isolating CPU load, yet it silently creates a second checkout unless identity is rebound there. MonkeyCode is an open-source coding-agent project with operator-supplied free model access and a free server option for hosting that replay. Disclosure: this article was prepared as part of MonkeyCode's product outreach, so treat the next runtime note as optional context.

The useful workflow is not trusting the remote host merely because the extra capacity was free. Copy or clone the intended SHA onto that host, run the binder, and only then let an agent quote files. Free model access can draft a checklist from the receipt schema, which is a writing aid rather than a source of hashes. If the remote receipt disagrees with the laptop receipt, you have a scope bug, not a personality clash with the decoder.

A second optional pass looks like the following, and it should fail closed when HEAD does not match. The commands assume you already cloned the same origin, which is a precondition rather than a hidden feature. Do not skip the nonce file, because a remote box that never received it cannot honestly echo it. That negative result is more valuable than a fluent explanation of a webhook that never existed on this SHA.

EXPECTED_HEAD="$(git rev-parse HEAD)"
git fetch origin
git checkout --detach "$EXPECTED_HEAD"
python workspace_bind.py --claim src/billing/webhook.py --out remote-receipt.json
python compare_receipts.py receipt.json remote-receipt.json

Myth 5: a second free model is automatically an independent witness

Independence is a property of evidence channels, not a property of drawing a new sample from a decoder. Two models reading the same unbound workspace will often repeat the same path-shaped mistake with extra confidence. Hide the nonce from the second pass if you want a cross-check that is not just shared context parroting. If both passes echo the receipt, you have agreement about identity, which still is not agreement about correctness.

Use the free model access, when you have it, to ask only questions that the receipt has already constrained. A scoped question names the nonce and the digest, then asks which function verifies signatures in that exhibit. An unscoped question asks the model to inspect webhook.py and thereby invites a convenient edition of the file. The difference is the same as asking a witness about a labeled exhibit versus asking them to recall a hallway.

What this method does not claim

This binder does not replace code review, signed provenance, or a locked-down CI identity for merges. It will not stop a determined agent from hashing a file after rewriting it and then quoting the new digest. People who need hardware attestation or regulated production controls should not treat a JSON receipt as a root of trust. Skip the method if merges already flow only from attested CI, because you already bind identity more strongly.

The approach also wastes time on throwaway spikes where the tree is disposable and no reviewer will ship the output. It can create a false sense of science if teams collect receipts and then ignore mismatches that are inconvenient. Do not use it as a reason to stop reading diffs, especially when the claimed change is authorization or money movement. The receipt is a scope label, like a specimen sticker on a vial, and stickers do not make the chemistry true.

If the next agent patch arrives as a path-shaped story, run the binder on the checkout you intend to merge. Argue about prompts only after the nonce, the HEAD, and the file digests agree. That order keeps model debate downstream of workspace identity, which is usually the cheaper disagreement to resolve.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The Pytest Block Was Prose: A Myth-Busting FAQ

Sam Yang — Tue, 08 Sep 2026 03:33:48 +0000

Consider a typical afternoon in a shared coding workspace, where a webhook change looks finished because the agent pasted a pytest summary. The fenced block reported forty-seven passed, zero failed, and a twelve-second runtime, which is exactly the shape reviewers have learned to trust. Canonical CI then failed on an import that the chat never executed, because the “run” had been a paragraph wearing a test harness. This FAQ treats that scene as a verification problem, not a personality problem in the model, and it replaces each repeated claim with a check that leaves a file on disk.

The corrected mental model is narrow enough to keep beside a pull request. Language models emit text that resembles command output because that text is common in training and in prior turns, not because a shell has returned an exit code. A free remote session makes the resemblance cheaper to produce, while it does not make the resemblance into evidence. Evidence is an artifact written by a wrapper the model does not control, carrying a nonce the wrapper created before the agent was allowed to speak about tests.

Myth: a pytest summary in chat means pytest ran

Developers repeat this claim because the block has the right nouns, the right integers, and the right absence of a traceback. Imitating a summary is closer to finishing a sentence than to invoking an interpreter, which is why the numbers stay round and the failed tests stay polite. A useful analogy is a restaurant receipt printed by a novelist: the columns look like commerce, yet no drawer has opened and no tax authority has received a copy. Before you believe a pass count, require a JSON document that a script outside the model wrote, and refuse to parse the transcript as a result file.

The proposed wrapper below is a starting point, not production CI. It mints a nonce, runs the real test command, and writes one artifact whose name includes that nonce. The agent may call the wrapper; it should not be asked to “paste what pytest would have printed.”

#!/usr/bin/env bash
# run_tests.sh — proposed wrapper, not a host security boundary
set -euo pipefail
ROOT="$(cd "$(dirname "$0")" && pwd)"
ART="$ROOT/artifacts"
mkdir -p "$ART"
NONCE="${AGENT_RUN_ID:-$(python3 -c 'import uuid; print(uuid.uuid4())')}"
START="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
set +e
python3 -m pytest -q --tb=line "$@"
CODE=$?
set -e
END="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
python3 - "$ART" "$NONCE" "$CODE" "$START" "$END" <<'PY'
import json, pathlib, sys
art, nonce, code, start, end = sys.argv[1:6]
payload = {
    "nonce": nonce,
    "exit_code": int(code),
    "started_at": start,
    "ended_at": end,
    "argv": ["python3", "-m", "pytest", "-q", "--tb=line"],
}
path = pathlib.Path(art) / f"test-run-{nonce}.json"
path.write_text(json.dumps(payload, indent=2) + "\n", encoding="utf-8")
print(f"WROTE {path} exit_code={code}")
sys.exit(int(code))
PY

A reviewer then asks a boring question that chat cannot answer with more prose. Where is artifacts/test-run-<nonce>.json, and does its exit_code match the claim in the transcript? If the file is missing, the pass did not occur in this tree, regardless of how tidy the fenced block looked.

export AGENT_RUN_ID="$(python3 -c 'import uuid; print(uuid.uuid4())')"
./run_tests.sh tests/
ls -l artifacts/test-run-"$AGENT_RUN_ID".json
python3 -c "import json,pathlib,os; p=pathlib.Path('artifacts')/'test-run-'+os.environ['AGENT_RUN_ID']+'.json'; print(json.loads(p.read_text())['exit_code'])"

Myth: asking the agent to paste the terminal is verification

This claim survives because it feels like an extra step, and extra steps get confused with controls. Pasting is still generation: the model can copy a prior block, invent a plausible duration, or drop the one failing test that would force a design discussion. Verification starts when a process you control writes bytes that the model cannot edit without going through the same wrapper. If the agent is allowed to rewrite artifacts/, the nonce is theatre, so keep that directory append-only in the session or copy it out immediately.

A small auditor makes the rule mechanical. It does not try to understand pytest; it only refuses the story when the story has no matching file. Treat the script as a proposed gate for humans and for any later CI job that should not trust chat.

#!/usr/bin/env python3
"""verify_pass_artifact.py — proposed check against narrated test runs."""
from __future__ import annotations

import argparse
import json
import re
from pathlib import Path

PASS_CLAIM = re.compile(
    r"(\d+)\s+passed",
    re.IGNORECASE,
)

def main() -> int:
    parser = argparse.ArgumentParser()
    parser.add_argument("--transcript", required=True)
    parser.add_argument("--artifacts", default="artifacts")
    parser.add_argument("--nonce", required=True)
    args = parser.parse_args()
    text = Path(args.transcript).read_text(encoding="utf-8")
    claimed = bool(PASS_CLAIM.search(text))
    artifact = Path(args.artifacts) / f"test-run-{args.nonce}.json"
    if not claimed:
        print("no_pass_claim_in_transcript")
        return 0
    if not artifact.is_file():
        print(f"FAIL claimed_pass_without_artifact nonce={args.nonce}")
        return 1
    payload = json.loads(artifact.read_text(encoding="utf-8"))
    if payload.get("nonce") != args.nonce:
        print("FAIL nonce_mismatch")
        return 1
    if int(payload.get("exit_code", 1)) != 0:
        print(f"FAIL artifact_exit_code={payload.get('exit_code')}")
        return 1
    print(f"OK artifact={artifact}")
    return 0

if __name__ == "__main__":
    raise SystemExit(main())

python3 verify_pass_artifact.py \
  --transcript agent-session.md \
  --nonce "$AGENT_RUN_ID"

Myth: a free remote workspace is already CI because both are somewhere else

Remote execution and continuous integration share a network location in casual speech, which is how the myth travels. CI is a contract about which command ran, against which commit, with which captured logs; a coding session is a conversation that can describe those things without performing them. Free model access and a free server option change the cost of trying the wrapper in a disposable workspace. They do not turn the workspace into a source of record until you keep the nonce files and the git sha together.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode's free model access and free server option are relevant when you want the wrapper, the tests, and the agent in one short-lived machine, then copy artifacts/test-run-*.json back to the canonical clone. The product does not certify that a transcript is true, and this article does not claim durability, hardware details, or a quota. The useful move is still local and dull: no artifact, no pass.

Myth: an exit code inside a fence is an exit code

Fenced shells look executable, especially when they include echo $? or a Process finished with exit code 0 line borrowed from an IDE. Those strings are not wait statuses from your kernel; they are characters. If you need an exit code, read it from the wrapper’s JSON, or from a CI system that you already trust for other jobs. A second proposed test keeps the rule in the repo so a later agent cannot “fix” a failure by editing the chat recap.

# test_wrapper_contract.py — proposed characterization test
import json
import os
import subprocess
import uuid
from pathlib import Path

def test_wrapper_writes_nonce_file(tmp_path, monkeypatch):
    root = Path(".").resolve()
    nonce = str(uuid.uuid4())
    env = os.environ.copy()
    env["AGENT_RUN_ID"] = nonce
    # Intentionally tiny: the contract is the file, not the product suite.
    result = subprocess.run(
        ["bash", str(root / "run_tests.sh"), "-k", "test_wrapper_writes_nonce_file"],
        cwd=root,
        env=env,
        check=False,
    )
    artifact = root / "artifacts" / f"test-run-{nonce}.json"
    assert artifact.is_file(), "wrapper must write a nonce-named artifact"
    payload = json.loads(artifact.read_text(encoding="utf-8"))
    assert payload["nonce"] == nonce
    assert "exit_code" in payload

That test will look circular if you run it through the same wrapper it is describing, so keep a non-agent path in CI that invokes pytest directly on test_wrapper_contract.py. The point is not elegance. The point is that a future session cannot satisfy review by inventing a cleaner paragraph.

Myth: you can replay “the same command” from memory and call it confirmation

Re-running a command later is a different experiment unless you pin the commit, the nonce, the seed data, and the exact argv. Agents compress those details because transcripts are hostile to bookkeeping, and free sessions are often truncated when the conversation gets long. If confirmation matters, store the artifact and the git rev-parse HEAD in the same directory, then compare bytes, not recollections. The decision table below is the whole policy for most application repos.

Claim in the transcript	Treat as	Evidence before merge
“47 passed” in a fence	Narration	`artifacts/test-run-<nonce>.json` with `exit_code` 0
“I ran pytest” without a path	Narration	Wrapper argv recorded in the artifact
“CI would pass”	Forecast	The canonical pipeline on that sha
“exit code 0” as prose	Fan fiction	JSON written by `run_tests.sh`
“I pasted the terminal”	Still narration	File the model cannot mint unaided

Limitations and who should not use this

A nonce file proves that a wrapper ran and that it captured an exit code. It does not prove the tests were meaningful, that they touched the webhook path, or that the agent refrained from editing tests until they agreed with the sketch. Truncated transcripts will hide pass claims, which lowers false alarms and also hides the original lie, so keep raw logs outside the chat when the change is important. Teams with a mature CI requirement already have a stronger version of this control, and they should not replace it with a shell script living next to an agent.

Do not use this workflow as the only gate on payment, identity, or safety-critical code, because a JSON file cannot see a wrong trust boundary. People who cannot read failing tests should not let a free session become the test runner of record, even when the prose looks like a well-funded pipeline. If your last merge trusted a fenced pytest block, put the wrapper beside the repo and require the nonce file before the next argument about which model is better.

The 200 Came From a Rental

Sam Yang — Sun, 06 Sep 2026 21:11:31 +0000

A pull request arrived after midnight with a README that claimed the API was already healthy. The coding agent had started a process, requested its own localhost, and treated a 200 as proof the service would run for everyone. That response was genuine inside a short-lived workspace, yet it said nothing about the laptop waiting on Monday. The reviewer stared at a green sentence printed on a host that nobody on the team could reopen.

This pattern appears whenever a coding agent can execute commands, not merely suggest them, and reviewers misread the transcript. Developers treat the agent's shell as a preview of their laptop because both sessions speak bash and render similar fonts. The analogy fails like a hotel gym standing in for a home garage, familiar until one bolt size changes. Claims in the next sections are the ones that keep returning during review, then a fingerprint workflow that makes the rental visible.

Myth: a bound port means the service is portable

Agents love a bound port because it is a crisp success token that copies cleanly into a README. A process that answers on the sandbox does not encode libc, extra packages, file layout, or the user's group permissions. Health checks measure a moment on a host you do not retain, not a contract with the checkout that will survive merge. Treat a remote 200 as proof that some files ran once, then demand a second run on CI or a laptop.

A useful correction is to refuse README claims that cannot be replayed from a clean clone of the branch. Ask the agent for the exact command sequence, the working directory, and the non-secret environment keys it exported during the run. Then execute that sequence locally with undocumented keys unset, unless they already exist in the team's dotenv template. If the local run dies on a missing header or a path the sandbox invented, the original green check was a rental.

Myth: a free remote box is unofficial CI

Teams under schedule pressure will point at agent logs the way they once pointed at a personal developer laptop after a demo. Continuous integration is a pinned image, a known network policy, and an artifact that later jobs can download without guessing. A disposable coding workspace is usually none of those properties, even when the vendor does not charge for the minutes of compute. Billing and reproducibility sit on different axes, so a zero-dollar session can remain an unpinned host that vanishes.

Free model access plus a free server option makes retries look cheap, which confuses a rental with a pipeline. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode can host an agent session with free model access and a free server option without turning that session into merge infrastructure. Use the remote box to draft changes and capture fingerprints, then keep gates on runners the team can name and retain.

Myth: matching the language runtime is enough

Python 3.12 on both sides is a comforting sentence that hides compilers, locales, and case sensitivity in the rest of the stack. Operating system package names and default filesystems drift independently of the language version string that agents paste into summaries. Agents also invent paths like /home/sandbox/work and then hard-code them into scripts that look abstract in the pull request. The corrected model treats a runtime version as one row in a matrix, never as the entire matrix itself.

Capture rows you can observe without leaking secrets, including kernel name, architecture, package managers, and the lockfile hash the agent actually used. Do not paste environment values; paste only sorted key names so reviewers can see DATABASE_URL was assumed without copying its contents. The scripts below are unexecuted proposals you should read, then run locally and again inside the agent workspace. Commit the script on the branch so both hosts hash identical bytes rather than improvising two inventories by hand.

A fingerprint instead of a vibe

The bash script prints inventory lines and avoids credential values, which matters when the remote transcript might be retained by a vendor. Redirect stdout to a file named after the host role, such as fingerprint.local.txt or fingerprint.agent.txt, before you compare. Keep raw files out of public gists when they reveal internal hostnames, and prefer committing only a short diff summary.

#!/usr/bin/env bash
# Proposed example: secret-free host fingerprint for agent-versus-laptop diffs.
set -euo pipefail

role="${1:-unspecified}"
echo "role=${role}"
echo "date_utc=$(date -u +%Y-%m-%dT%H:%M:%SZ)"
echo "pwd=${PWD}"
echo "user=$(id -un 2>/dev/null || echo unknown)"
echo "uid=$(id -u 2>/dev/null || echo unknown)"

echo "uname=$(uname -s 2>/dev/null || echo unknown)"
echo "arch=$(uname -m 2>/dev/null || echo unknown)"
echo "kernel=$(uname -r 2>/dev/null || echo unknown)"

if command -v python3 >/dev/null 2>&1; then
  python3 - <<'PY'
import sys, platform
print(f"python={sys.version.split()[0]}")
print(f"python_impl={platform.python_implementation()}")
print(f"platform={platform.platform()}")
PY
else
  echo "python=missing"
fi

if command -v node >/dev/null 2>&1; then
  echo "node=$(node -v)"
else
  echo "node=missing"
fi

if command -v go >/dev/null 2>&1; then
  echo "go=$(go env GOVERSION 2>/dev/null || echo present)"
else
  echo "go=missing"
fi

echo "env_keys=$(env | cut -d= -f1 | sort | tr '\n' ',' | sed 's/,$//')"

for lock in package-lock.json yarn.lock pnpm-lock.yaml poetry.lock uv.lock Cargo.lock go.sum Gemfile.lock; do
  if [[ -f "$lock" ]]; then
    if command -v sha256sum >/dev/null 2>&1; then
      echo "lock_${lock}=$(sha256sum "$lock" | awk '{print $1}')"
    else
      echo "lock_${lock}=$(shasum -a 256 "$lock" | awk '{print $1}')"
    fi
  fi
done

Pair the capture with a comparer that fails review when sandbox and laptop disagree on fields your team marked required. The Python that follows is also a proposal, so change the required keys rather than treating them as universal. Exit status is the review signal; printed DRIFT lines are the conversation starter in the pull request thread.

# Proposed example: compare two fingerprint files and exit nonzero on drift.
from pathlib import Path
import sys

REQUIRED = ("uname", "arch", "python", "node")


def parse(path: Path) -> dict[str, str]:
    data = {}
    for line in path.read_text(encoding="utf-8").splitlines():
        if "=" not in line:
            continue
        key, value = line.split("=", 1)
        data[key] = value
    return data


def main() -> int:
    if len(sys.argv) != 3:
        print("usage: compare_fingerprints.py local.txt agent.txt", file=sys.stderr)
        return 2
    local = parse(Path(sys.argv[1]))
    agent = parse(Path(sys.argv[2]))
    failed = False
    for key in REQUIRED:
        lv, av = local.get(key, "absent"), agent.get(key, "absent")
        if lv != av:
            print(f"DRIFT {key}: local={lv} agent={av}")
            failed = True
        else:
            print(f"OK {key}={lv}")
    local_keys = set((local.get("env_keys") or "").split(","))
    agent_keys = set((agent.get("env_keys") or "").split(","))
    invented = sorted(k for k in (agent_keys - local_keys) if k)
    missing = sorted(k for k in (local_keys - agent_keys) if k)
    if invented:
        print("AGENT_ONLY_ENV_KEYS " + ",".join(invented[:40]))
    if missing:
        print("LOCAL_ONLY_ENV_KEYS " + ",".join(missing[:40]))
    return 1 if failed else 0


if __name__ == "__main__":
    raise SystemExit(main())

A prompt you paste into the agent is part of the same proposal and is not a guarantee that the agent will comply. Ask it to run the committed script, write fingerprint.agent.txt, and stop without starting long-lived servers as proof of success. If it starts a server anyway, treat that as data about the agent rather than a reason to skip the fingerprint file. If it rewrites the script to omit inconvenient fields, that rewrite is itself a finding you should quote in review.

Myth: small tasks do not need the second run

Short diffs are where silent path assumptions hide, because nobody wants a pipeline ceremony for a twelve-line helper. The helper that imports orjson because the sandbox already had it will fail on a laptop that only shipped the standard library. The helper that writes /tmp/agent-out may later collide with a shared runner that treats that directory as a crowded hallway. Diff size is a poor prior for environment drift, while novelty of commands and native dependencies is a better one.

If the agent touched package manifests, native modules, container files, or anything that binds a port, run the fingerprint pair. If it only rewrote comments and string copy, you can skip the ceremony without pretending the environments were proven equal. Write that split into the review guide so agents are not negotiating freshness of the rule during every late session.

The rental is not the hall

Think of the agent's workspace as a rental instrument, tuned before you arrived and reset after you leave the room. Your repository is the score, and your CI image is the hall where the performance is actually graded for merge. A free-tier session can still play the piece clearly and still be the wrong room for the exam that counts. The useful question is which host produced each claim, and whether that host will still exist tomorrow morning.

Merge only when the local or CI fingerprint satisfies the required keys and the lockfile hashes match across both captures. Agent-only environment keys must be documented in the template or removed from the code before the branch is eligible. Fail the review when the agent started a server as proof, sandbox paths leaked into scripts, or the fingerprint file is missing. Pass with comments when the drift is limited to tools the change never invokes, and record that exception in the thread.

Limitations and who should skip this

This workflow does not prove functional correctness, security, or performance, and it only reduces a class of worked-over-there mistakes. It will not help if the agent refuses to run your script, or if the remote filesystem vanishes before the fingerprint can be written. It can create false safety when required keys are too few, because two similar Linux boxes can still disagree on libpq. Expand the required set when your change touches databases, browsers, or compilers, rather than celebrating an OK python line.

Do not use a remote coding server as the system of record for regulated data, production secrets, or customer traffic. Do not treat free compute as permission to disable CI in order to save a handful of wall-clock minutes on a branch. People on air-gapped product lines should keep remote agents off the critical path, even when the models themselves are appealing. People shipping mobile or embedded artifacts need device coverage that a host fingerprint does not pretend to replace at merge time.

The approach is a poor fit for exploratory chats that never touch the repository, because there is no merge gate to protect. In that setting a fingerprint comparison adds ceremony without changing what will run on anyone else's machine later. Once files move toward a pull request, the ceremony is cheaper than a Monday failure caused by a port that existed only on their box.

If you already draft on a free-tier agent with a remote workspace, capture both fingerprints before trusting a health check. Reproduce the commands from a clean clone, and treat any remote 200 as a rental receipt rather than a production certificate. The receipt can still be useful notes for debugging, as long as nobody files it under continuous integration by accident.

The Dependency the Agent Invented: A Myth-Busting FAQ

Sam Yang — Sat, 05 Sep 2026 19:48:57 +0000

Consider a reconstructed incident that shows up when teams treat an agentic loop as an architecture review. A backend pair watched their coding agent close six steps with a confident summary and a tidy pull request. The suite was green, the changelog mentioned Redis-backed sessions, and nobody had added Redis to compose, secrets, or the runbook. The miss was not a weak model so much as a missing freeze on what the repository was allowed to assume.

Developer glossaries often present planning, tools, memory, and loops as if those nouns were safety properties. Those nouns describe a control flow, not a signed agreement about services, environment variables, or network ports. The practical question is not whether the agent sounded agentic, but whether a new dependency appeared without a prior contract line. The rest of this FAQ restates four claims teams repeat, then offers a checker that fails the build when the contract is violated.

Claim: a finished agentic loop means the design was negotiated

Engineers often read a multi-step trace as evidence that the agent and the repository reached a shared understanding. The trace usually contains file reads, a patch, a test command, and a paragraph that restates the patch in calmer language. That sequence resembles a hiker who stops when the trees look familiar rather than a committee that recorded dissent. Green tests record that the new code agrees with itself, not that Redis, object storage, or a queue existed yesterday.

A corrected mental model treats the loop as a hypothesis generator whose writes remain untrusted until an assumption freeze passes. Names that were absent from the contract file before the first tool call are invented dependencies, even when the compiler is quiet. You can still use agentic planning; you just refuse to confuse a plan with a change-control record.

Claim: a successful tool call equals the intended side effect

Teams point at a tool result that returned exit code zero and conclude the world now matches the assistant's summary. Exit codes report that a process finished, not that the process mutated the object the prompt described. A formatter can succeed on a file that should not exist, and tests can pass against a stub the agent just wrote. The analogy is a shipping label that prints without proving the crate contained the parts on the invoice.

Evidence you can collect is boring on purpose: diff the working tree against a declared allowlist after every tool batch. If docker-compose.yml, .env.example, or infrastructure modules gained a service name, that name must already live in the contract. If the name is new, the loop did not discover architecture; it smuggled a prior from training data into your repo.

Claim: reading a file once keeps the agent grounded for the rest of the loop

Developers repeat that the agent has the codebase because an early step dumped a tree or opened README.md. A transcript is a snapshot with drift, more like a tourist map folded in a pocket than a live survey of the street. Later writes can add modules the early read never saw, and later reads can miss files the agent itself created. Grounding is a time series of checks, not a single observation captured at step one of the loop.

The corrected model is to re-hash the working tree after each batch of writes and compare it with the contract, not with the chat. Chat memory can narrate files that the index never staged, which is how invented caches survive a code review that only skims the summary. If your review process starts from the assistant's closing paragraph, you are reviewing literature, not the tree.

Claim: a large free-tier token budget removes the need for an assumption budget

A related claim says that once tokens are plentiful, the agent will eventually read enough files to stop inventing services. Token volume only measures how much text moved through the loop, not how many new runtime names were authorized. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode provides free model access and a free server option with an operator-stated ten million token allowance, which makes long loops feel inexpensive.

Use that server as a place to run the checker beside the agent, not as a reason to skip the contract file. Split the token allowance so the last pass must read assumptions/contract.yaml and the diff, even if the feature summary already sounds complete. If the ledger pass cannot fit, the loop was not cheap; it was unpaid review shifted onto the next human.

A proposed assumption ledger

The proposed checker is a local fixture, not a benchmark, and it has not been executed against a private production monorepo in this article. Copy it into a throwaway app that already has compose, tests, and a small API so you can see the failure mode. The contract is the customs form; the agent may pack the suitcase, but undeclared names should not cross the boundary. The script reads a unified git diff, then rejects service names, environment prefixes, ports, and imports that the YAML file does not list.

# assumptions/contract.yaml
# Proposed fixture: commit this before the agent starts a loop.
version: 1
allowed_compose_services:
  - api
  - postgres
  - worker
allowed_env_prefixes:
  - APP_
  - DATABASE_
allowed_ports:
  - 8000
  - 5432
allowed_python_imports:
  - fastapi
  - sqlalchemy
  - pydantic
  - psycopg
datastore_hints:
  redis: redis
  memcached: memcached
  mongo: mongodb
  elastic: elasticsearch
  kafka: kafka
  rabbitmq: rabbitmq
  minio: object-storage
  s3://: s3

#!/usr/bin/env python3
"""Proposed fixture: fail a diff that invents undeclared runtime names.

Label: unexecuted example. Run in a copy of your repo. Do not point it
at production kubeconfigs, secret files, or live credentials.
"""
from __future__ import annotations

import re
import subprocess
import sys
from pathlib import Path

try:
    import yaml
except ImportError:
    sys.stderr.write('pip install pyyaml\n')
    raise SystemExit(2)

CONTRACT_PATH = Path('assumptions/contract.yaml')
ENV_ASSIGN = re.compile(r'\b([A-Z][A-Z0-9_]{2,})=')
PORT_PAIR = re.compile(r'\b(\d{2,5}):(\d{2,5})\b')
IMPORT_LINE = re.compile(r'^\s*(?:from|import)\s+([A-Za-z0-9_]+)')
COMPOSE_SERVICE = re.compile(r'^\s{2}([A-Za-z0-9_-]+):\s*$')


def git_diff() -> str:
    result = subprocess.run(
        ['git', 'diff', '--unified=0', 'main...HEAD'],
        check=False,
        capture_output=True,
        text=True,
    )
    return result.stdout


def added_lines(diff: str) -> list[str]:
    lines = []
    for line in diff.splitlines():
        if line.startswith('+') and not line.startswith('+++'):
            lines.append(line[1:])
    return lines


def main() -> int:
    contract = yaml.safe_load(CONTRACT_PATH.read_text())
    added = added_lines(git_diff())
    joined = '\n'.join(added)
    failures: list[str] = []

    allowed_services = set(contract['allowed_compose_services'])
    allowed_prefixes = tuple(contract['allowed_env_prefixes'])
    allowed_ports = {int(p) for p in contract['allowed_ports']}
    allowed_imports = set(contract['allowed_python_imports'])
    hints = contract['datastore_hints']

    for hint, label in hints.items():
        if hint.lower() in joined.lower():
            failures.append(f'invented datastore hint {label!r} via {hint!r}')

    for line in added:
        match = COMPOSE_SERVICE.match(line)
        if match and match.group(1) not in allowed_services:
            failures.append(f'invented compose service {match.group(1)!r}')
        for env_name in ENV_ASSIGN.findall(line):
            if not env_name.startswith(allowed_prefixes):
                failures.append(f'invented env {env_name!r}')
        for host_port, _container in PORT_PAIR.findall(line):
            port = int(host_port)
            if port not in allowed_ports:
                failures.append(f'invented host port {port}')
        imported = IMPORT_LINE.match(line)
        if imported and imported.group(1) not in allowed_imports:
            failures.append(f'invented import {imported.group(1)!r}')

    unique = sorted(set(failures))
    if unique:
        sys.stderr.write('assumption freeze failed\n')
        for item in unique:
            sys.stderr.write(f'- {item}\n')
        return 1
    sys.stdout.write('assumption freeze passed\n')
    return 0


if __name__ == '__main__':
    raise SystemExit(main())

Run the commands from a clean branch so the diff window matches the loop you actually want to audit. If check_assumptions.py exits with a nonzero status, the agent invented a dependency even when pytest remains quiet. Keep the contract in the same pull request as the feature so reviewers see the freeze and the code together.

git checkout -b audit-invented-deps
python3 tools/check_assumptions.py; echo exit:$?
git diff --name-only main...HEAD
rg -n 'redis|memcached|mongo|elastic|kafka|s3://' -g '!assumptions/**' -g '!tools/**'

# tests/test_assumption_contract.py
# Proposed test. Label: unexecuted example.
from pathlib import Path
import yaml

REQUIRED = (
    'allowed_compose_services',
    'allowed_env_prefixes',
    'allowed_ports',
    'allowed_python_imports',
    'datastore_hints',
)

def test_contract_has_required_keys():
    data = yaml.safe_load(Path('assumptions/contract.yaml').read_text())
    for key in REQUIRED:
        assert key in data and data[key], key

When the checker fails, classify the hit before you widen the allowlist, because widening is how invented caches become permanent. If the name is a real product decision, add it to the contract in a separate commit with a human sentence of intent. If the name arrived from a model prior, delete the code and spend tokens on a narrower prompt that cites the contract path. Do not enlarge allowed_python_imports to silence a red build; that is how Redis became a session store in the reconstructed incident.

This approach will not help teams that lack a declared runtime, because a blank contract either blocks all work or rubber-stamps everything. Regular expressions over diffs miss generated manifests, helm templates, and configuration assembled at deploy time from concatenated strings. The ledger also ignores semantic debt: an allowed postgres service can still receive a schema the team cannot operate. Security review, load testing, and data-retention analysis remain outside the fixture, and green ledger output should not be sold as those reviews.

Skip this workflow if your organization already enforces infrastructure policy as code through a real admission controller. Skip it for throwaway prototypes that will never open a port, and skip it if nobody can edit the contract except the agent. An agent-owned contract is a diary, not a freeze, and it will happily legalize every invention after the fact.

Agentic language will keep spreading because it names a real control flow that many developers are seeing for the first time. The useful move is to keep the vocabulary and still demand a freeze on names that cost money, pages, or incident hours. If you already run a free-tier coding agent, commit the contract before the loop, then let the remaining tokens write code that can pass it.

The Agent Could Still Open .env

Sam Yang — Fri, 04 Sep 2026 17:37:10 +0000

A reviewer opened a Friday pull request whose description claimed the coding agent never touched secrets. The agent had listed three application files, a passing test command, and a reminder that .env remained gitignored. Staging failed an hour later when another process printed a live key that had never left disk. Ignore files constrain version control, not every workspace reader that can still call open().

That incident is a composite of a pattern that appears once coding agents share ordinary developer checkouts. Teams repeat a handful of workspace claims because those claims sound like kernel guarantees rather than conventions. They remain hypotheses until you record the paths the tools actually opened during the session. This FAQ treats each repeated claim as a testable statement and replaces it with a corrected mental model.

What the agent wrote is not a filesystem trace

Most session summaries are after-the-fact narratives stitched from tool JSON, not append-only logs of open and exec. An agent can name a directory and still later read a gitignored child through a shell. Fluency is cheap on current coding models, and omission stays cheap when nobody asked for a read inventory. If you cannot point to a path list, you do not yet know the working set.

Reproducing the checks requires a checkout, a tool-using agent, and a writable trace file, not a private GPU. Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source coding-agent project with free-tier model access and an optional free server for scratch workspaces. Those options matter only as a disposable host for the audit, which does not depend on a named model.

Myth: gitignore means the agent cannot read the file

Developers treat .gitignore as a privacy boundary because git status hides the same paths they care about. Git ignore rules only change which paths enter the index, the object database, and some porcelain listings. A coding agent that receives a shell can run cat .env without asking git for permission. Treating gitignore as a sandbox is like treating a packing list as a lock on the suitcase you already opened.

You can demonstrate the split on any clone that still has a local env file. The first command shows git's opinion of the path, while the second shows the kernel's opinion. If the Python one-liner prints content, an agent with the same user identity can print it too. Asking the model not to read secrets remains a request, not a syscall filter.

git check-ignore -v .env || echo "git does not ignore this path"
python -c "print(open('.env','r',encoding='utf-8',errors='replace').read()[:80])"

If the second command prints anything, the agent can print it under the same user. Policy text in the prompt will not survive a later tool call that looks more useful than obedience. Teams that need a real boundary have to move the file out of the reachable tree or drop the unrestricted shell. Logging the successful read still helps after the fact, but that record remains forensics rather than prevention.

Myth: naming three folders means the agent loaded the monorepo

Session write-ups often list apps/web, packages/ui, and services/api as if those labels implied recursive ingestion. Listing a path is a speech act, whereas reading a path is an I/O act with a byte count. Many agents grep first, open the hits, and then summarize from those hits plus priors about similar trees. The corrected model treats every claimed file as unread until a trace shows a successful open.

A small JSONL log is enough to keep that distinction honest during a session. Each tool wrapper should append one record before it returns bytes to the model. Until that file exists, repository understanding is just a fluent list of plausible names. Counting unique resolved paths tells you more than counting tokens in the explanation.

# read_trace.py — labeled example for a local tool wrapper
import json, os, time
from pathlib import Path

TRACE = Path(os.environ.get("AGENT_READ_TRACE", "agent_reads.jsonl"))

def record_open(path: str, ok: bool, nbytes: int, via: str) -> None:
    rec = {
        "ts": time.time(),
        "path": str(Path(path).resolve()),
        "ok": ok,
        "nbytes": nbytes,
        "via": via,
        "pid": os.getpid(),
    }
    with TRACE.open("a", encoding="utf-8") as fh:
        fh.write(json.dumps(rec) + "\n")

The wrapper is deliberately boring so you can paste it around open, read_file, or a shell cat helper. It does not parse git, and it does not redact values, which keeps the failure mode visible. If wrapping every tool is too heavy, start with the shell tool, because that is where ignore rules usually die. Review the JSONL file before you spend time reviewing the agent's markdown summary.

Myth: a successful tool object means the command did the useful thing

Agents frequently return a zero exit for npm test, pytest, or cargo check inside a compact JSON object. Exit zero can mean tests ran, a stub script echoed success, or the working directory was empty. A zero exit is a receipt for a process start, not a photograph of the meal you thought you ordered. Without a binary path, a cwd, and an output digest, a green object is just another fluent sentence.

# labeled example: bind the command to cwd, binary, and output hash
cd /workspace/app
readlink -f "$(command -v pytest)"
pytest -q --maxfail=1 | tee /tmp/pytest.out
sha256sum /tmp/pytest.out

Compare the digest with whatever the agent pasted into the chat transcript after the tool returned. Disagreement means the summary drifted or the workspace moved under the process. Either failure is cheaper to catch on the digest than after a green badge in production. Keep the tee file beside the trace so later reviewers can replay the same bytes.

Myth: a free shared server isolates uncommitted files from model context

Teams sometimes assume a complimentary remote workspace behaves like a personal laptop with slightly better uptime. A free server is still a computer with a user, a disk, and a tool broker sitting in front of that disk. Isolation comes from accounts, containers, and network policy, not from the adjective free. Uncommitted env files and old traces stay readable until you delete them or start from a fresh tree.

The useful habit is to treat each finished session as a working set that you enumerate on disk. After the agent stops, join the trace with git ignore rules and a cheap secret-shaped name check. The classifier below is a proposal you can run locally; it is not a claim about any hosted scanner. A non-zero exit should mean more than a polite closing sentence from the model.

# workspace_read_audit.py
"""Classify recorded agent reads against git ignore rules and simple secret names."""
from __future__ import annotations

import json, re, subprocess, sys
from pathlib import Path

SECRETISH = re.compile(
    r"(\.env($|\.)|credentials|id_rsa|id_ed25519|\.pem$|\.p12$|secret)",
    re.I,
)

def git_ignored(path: str) -> bool:
    r = subprocess.run(
        ["git", "check-ignore", "-q", "--", path],
        capture_output=True,
    )
    return r.returncode == 0

def classify(path: str) -> str:
    name = Path(path).name
    if SECRETISH.search(path) or SECRETISH.search(name):
        return "secretish"
    if git_ignored(path):
        return "gitignored"
    return "tracked_or_unknown"

def main(trace: Path) -> int:
    if not trace.exists():
        print(f"missing trace: {trace}", file=sys.stderr)
        return 2
    counts = {"secretish": 0, "gitignored": 0, "tracked_or_unknown": 0}
    for line in trace.read_text(encoding="utf-8").splitlines():
        rec = json.loads(line)
        if not rec.get("ok"):
            continue
        bucket = classify(rec["path"])
        counts[bucket] += 1
        print(f"{bucket:20} {rec.get('nbytes', 0):7} {rec['path']}")
    print("---")
    print(counts)
    return 0 if counts["secretish"] == 0 else 1

if __name__ == "__main__":
    sys.exit(main(Path(sys.argv[1] if len(sys.argv) > 1 else "agent_reads.jsonl")))

Run the classifier from the same checkout the agent used, because path resolution follows the host the process actually saw. A non-zero exit means at least one successful read matched a secret-shaped path. That signal beats a closing apology in the transcript, and it is cheap enough to repeat after every session. Store traces as paths and sizes only unless you are prepared to treat the log as secret material.

python workspace_read_audit.py agent_reads.jsonl
echo $?

Myth: a policy sentence in the system prompt is a sandbox

Prompts such as never read environment files fail the way linter comments fail, because they are text competing with other text. Tool-using agents optimize for finishing the user task, and reading .env.local is often the fastest path to a boot. The corrected model puts prevention outside the model, using a dedicated user, a bind-mounted subset, or a command allow-list. If you cannot change the sandbox yet, fail closed on the audit for agent-authored branches.

# labeled CI fragment, not a complete pipeline
git checkout "$AGENT_BRANCH"
python workspace_read_audit.py agent_reads.jsonl
test "$(python workspace_read_audit.py agent_reads.jsonl >/dev/null; echo $?)" -eq 0

That gate will not stop a determined shell from printing a key during the live session itself. It will stop the team from merging a session that already looked unclean in the trace. Detection after the fact is not a sandbox, and calling it one will recreate the Friday outage. Use the gate as a review aid while you work toward an actual allow-list.

A corrected working set, not a wiser ghost

The mental model that survives these checks is intentionally boring and slightly unkind to chat UIs. A coding agent is a program that proposes edits and sometimes runs tools against a tree. The workspace is a disk with permissions, git ignore is a commit filter, and model context is a truncated buffer. Those four objects collapse into one transcript, which is why the myths keep getting airtime.

This approach is the wrong tool if you need hard multi-tenant isolation or a guarantee that prompts never leave your rack. Detection scripts do not replace seccomp profiles, ephemeral virtual machines, or secret managers that never write values into the tree. People who copy production env files into a shared free-tier directory to speed the agent should stop first. The samples above assume a disposable clone and an operator who already knows which files are toxic.

If you want a disposable host for this audit, MonkeyCode's free-tier models and free server can host a throwaway clone. That is an optional rehearsal environment, not a requirement for the classifier to work. The value remains the log you keep, not the brand sitting on the workspace you happened to use. Delete the clone after the audit so yesterday's env file cannot become tomorrow's model context.

MonkeyCode provides free models that can run this workflow.

Green But Wrong: Mutation-Testing Agent-Generated Test Suites

Sam Yang — Fri, 04 Sep 2026 11:57:28 +0000

The test suite passed in forty-two seconds, which made the production incident that followed especially embarrassing. Our coding agent had generated a set of unit tests alongside a refactor of the patient registration flow, and every assertion was green when the pull request merged. Two days later a compliance report flagged that a 120-year-old patient had been registered, violating a policy that should have rejected anyone at or above that age. The root cause traced back to a boundary condition the tests should have caught, and the worst part was that the tests were not missing; they were simply asserting the wrong boundary.

That shared assumption is the signature weakness of agent-generated tests, because the model encodes one mental model into both the code and its verification. A misunderstanding in the implementation gets faithfully reproduced in the test, producing a false green where the suite passes and the coverage report looks healthy. The bug sails through because the test asserts the buggy behavior as if it were the specification, and coverage percentages only measure which lines executed. They do not measure whether the assertions would notice a broken line.

Mutation testing closes that gap by asking a different question, one that measures how much code the tests can distinguish from a broken version of itself. The tool introduces small faults, called mutations, into the source and runs the suite against each one; if the tests pass with the mutation in place, the mutation survives, and that survival is a direct signal of a testing blind spot. A suite with a high mutation survival rate is not merely incomplete, it is actively misleading, because it reports green while broken code remains undetected. This makes mutation survival a more honest metric than coverage for evaluating agent-generated tests.

I ran this workflow on the registration validator using the open-source MonkeyCode project's current free tier, which includes 10 million tokens and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The setup took about twenty minutes and required no local dependencies beyond an SSH client, because the free server provided an isolated Linux environment where I could install the mutation tooling without touching my laptop's Python installation. The token allowance covered the entire session, including a conversation where I asked the model to explain each surviving mutation and classify it as a real gap or a false alarm.

The module under test was a small age validator that parses a string, rejects negative values, and rejects implausibly large ones:

# validator.py
def validate_age(age_str):
    age = int(age_str)
    if age < 0:
        raise ValueError("Age cannot be negative")
    if age > 120:
        raise ValueError("Age cannot exceed 120")
    return age

The agent generated three tests: one for a valid input, one for a negative input, and one for an input that exceeds the upper bound. The suite passed and coverage reported one hundred percent, which made the mutation results more interesting. The tool introduced a mutation that changed age < 0 to age <= 0, and the suite passed anyway, because no test exercised the boundary value of zero. Another mutation changed age > 120 to age >= 120, and again the suite stayed green, because the test for the upper bound used 121 rather than 120 itself. The survival rate for this tiny module was roughly thirty percent, and every surviving mutation traced to a boundary condition.

The fix was not a longer test file but a sharper testing strategy, and I added two boundary tests, one for zero and one for 120, which dropped the mutation survival rate to zero for this module. The interesting part is that the agent's original tests were not lazy; they were simply shaped by the same boundary blindness that shaped the implementation. The model had written age < 0 in the code and then written tests for -1 and 121, which are the values that make the comparisons true, not the values that probe the comparison itself. Mutation testing exposed that pattern in minutes, where a code review might have missed it entirely.

The same workflow scales to larger codebases with a few practical adjustments, and the first is computational cost, because mutation testing runs the full suite once per mutation. A large project can take hours on a modest machine, so the free server handled this small module comfortably but would struggle with a serious codebase. The free tier's token allowance is generous for interactive sessions, yet it is not a substitute for dedicated test infrastructure, and teams should treat it as a sandbox for experiments rather than a permanent pipeline. Mutation tools also produce false positives, where a surviving mutation reflects equivalent code rather than a real gap, so the results need human interpretation rather than blind automation.

The reusable lesson is that agent-generated tests deserve the same skepticism as agent-generated code, and mutation survival rate is a concrete metric for that skepticism. A green suite from an agent is a starting point, not a conclusion, and the boundary-blindness pattern I found here is common enough to warrant a quick mutation run on any critical module. If you want to reproduce this workflow, the MonkeyCode free tier is a reasonable sandbox for the experiment, though the technique itself is tool-agnostic and will serve you with any mutation framework you already use.

MonkeyCode provides free models that can run this workflow.

A Cost-Aware Shadow Test for Trending Model Claims

Sam Yang — Fri, 04 Sep 2026 10:57:06 +0000

One Friday afternoon, a team pointed its production pipeline at a newly released model after reading a launch thread that called it both cheaper and better on common benchmarks. By Monday, the support queue had filled with malformed JSON and missed tool calls. The model was not broken; it was simply tuned for a different distribution than the prompts that actually reached the service. That gap is easy to close if you look at your own workload before you flip the switch.

Launch threads are not evaluations. When a name such as DeepSeek-V4-Pro-0813 or Grok 4.6 starts trending, the useful question is not whether the public benchmark table is accurate. The useful question is whether that table predicts success on your exact prompts, output shapes, and failure tolerances. A shadow test answers that question with a small amount of code and a list of representative requests.

The workflow below compares an incumbent model with a candidate on the same prompts, then records latency, token usage, and a lightweight pass or fail signal. It does not require a production rollout, and it can run on a free OpenAI-compatible endpoint so the experiment itself does not become a budget discussion. A free OpenAI-compatible endpoint from MonkeyCode can act as the baseline, and its free server option can host the runner. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Start by collecting prompts that cause real failures, such as a customer request that returned invalid JSON, a code patch with an empty diff, or a tool call that used the wrong schema. These examples are worth more than a thousand generic benchmark rows because they encode the shape of your system's errors. The harness runs each prompt twice, once through the current model and once through the candidate, then compares the results.

import asyncio, json, os, re, time
import httpx

async def call_model(client, cfg, prompt):
    started = time.perf_counter()
    body = {
        'model': cfg['model'],
        'messages': [{'role': 'user', 'content': prompt}],
        'temperature': 0.0,
        'max_tokens': 512,
    }
    url = cfg['base_url'] + '/chat/completions'
    headers = {'Authorization': 'Bearer ' + os.environ.get('API_KEY', '')}
    response = await client.post(url, headers=headers, json=body)
    elapsed = time.perf_counter() - started
    payload = response.json()
    usage = payload.get('usage', {})
    text = payload['choices'][0]['message']['content'] if payload.get('choices') else ''
    return {
        'model': cfg['model'],
        'latency_s': round(elapsed, 3),
        'input_tokens': usage.get('prompt_tokens', 0),
        'output_tokens': usage.get('completion_tokens', 0),
        'text': text,
    }

def check(text, expected):
    if expected.get('is_json'):
        try:
            json.loads(text)
            return True
        except Exception:
            return False
    if expected.get('regex'):
        return re.search(expected['regex'], text, re.IGNORECASE) is not None
    return True

async def run(cfg, prompt, expected):
    async with httpx.AsyncClient(timeout=30) as client:
        result = await call_model(client, cfg, prompt)
        result['pass'] = check(result['text'], expected)
        return result

async def main(path):
    rows = [json.loads(line) for line in open(path) if line.strip()]
    baseline = {'model': os.environ['BASELINE_MODEL'], 'base_url': os.environ['BASELINE_BASE_URL']}
    candidate = {'model': os.environ['CANDIDATE_MODEL'], 'base_url': os.environ['CANDIDATE_BASE_URL']}
    results = []
    for row in rows:
        prompt = row['prompt']
        expected = row.get('expect', {})
        results.append(await run(baseline, prompt, expected))
        results.append(await run(candidate, prompt, expected))
    print(json.dumps(results, indent=2))

if __name__ == '__main__':
    asyncio.run(main(os.environ.get('PROMPT_FILE', 'prompts.jsonl')))

Create a prompt file with two representative failures. The exact model strings are placeholders, so confirm the provider's current list before you run the command.

cat > prompts.jsonl <<'EOF'
{"prompt": "Extract order id from: order #AB-1234 was placed", "expect": {"regex": "AB-1234"}}
{"prompt": "Return a JSON object with keys total and currency for 14 USD", "expect": {"is_json": true}}
EOF

export BASELINE_BASE_URL=https://your-baseline-endpoint.example/v1
export BASELINE_MODEL=your-current-model
export CANDIDATE_BASE_URL=https://candidate-endpoint.example/v1
export CANDIDATE_MODEL=deepseek-v4-pro-0813-or-grok-4.6-placeholder
export API_KEY=your_key_here
python shadow_test.py prompts.jsonl

A useful decision table keeps the comparison short. Suppose the baseline returns 48 passing results with a median latency of 1.2 seconds, while the candidate returns 44 passing results at 0.7 seconds. The candidate is genuinely faster, but it introduces four regressions in the exact failure modes you care about. That is not a reason to reject it; it is a reason to inspect the failing prompts and decide whether they are safe to retrain, route differently, or leave on the incumbent.

Model	Pass / N	Median latency	Input tokens	Output tokens	Failure notes
baseline	48 / 50	1.2s	12,400	3,800	2 invalid JSON
candidate	44 / 50	0.7s	12,400	3,200	4 invalid JSON, 2 missed regex

This harness is deliberately narrow. It does not measure long-context behavior, multi-turn tool use, retry behavior, or non-deterministic failures that appear only under load. A free endpoint also carries limits that can change without notice, so treat the baseline as a convenience rather than a production mirror. If the candidate model string is wrong or the provider renames it, the same script will fail loudly instead of silently testing the wrong model.

Teams with an existing evaluation suite or live canary traffic should not replace those systems with this script. If you already collect production pass rates and can roll back quickly, a small canary deployment will catch problems an offline dataset cannot. This shadow harness is for the earlier stage, when the only evidence you have is a trending title and a vendor's benchmark table.

The next time a model name starts trending, treat the launch post as a lead, not a lease. Collect a small set of your own failing prompts, run the harness against both models, and let the decision table tell you whether the switch is worth a canary. A free server option from MonkeyCode is one zero-cost place to keep that harness running while you collect evidence.

MonkeyCode provides free models that can run this workflow.