DEV Community: Sam Yang

A Snapshot Is Not a Rebuild: Five Myths About Free Dev Boxes

Sam Yang — Tue, 27 Oct 2026 13:01:10 +0000

The script passes on the machine where you wrote it and fails on the machine you borrowed, and the traceback rarely names the real difference. Most teams respond by pinning one more version in requirements.txt, rerunning, and calling it fixed when the color turns green. That habit works often enough to feel like engineering, which is exactly why the underlying myths survive contact with production. This piece takes five claims developers repeat about free or borrowed development boxes and tests each one against a concrete artifact you can run this week.

Myth 1: The same base image means the same box

Two instances built from the same image can still disagree about the C library, the locale, the timezone database, and the global package tree. A wheel compiled against a newer glibc will install on one host and silently resolve to a source build on another, and sorting behavior changes the moment LC_ALL moves from C to en_US.UTF-8. Neither difference announces itself in a stack trace, because both are environmental facts rather than code paths. The image is a starting point, not a fingerprint.

Myth 2: Pinning your direct dependencies is enough

Pinning top-level versions constrains the graph you declared and says nothing about the graph your package manager chose. Transitive resolution depends on the resolver version, the index state at install time, and sometimes on the CPU architecture of the box doing the install. You can prove this cheaply by hashing pip freeze output on two supposedly identical hosts and comparing the digests. If the digests differ while the pins match, the pins were never the contract you thought they were.

Myth 3: A snapshot is a backup of the environment

A snapshot restores a disk, which is useful, and it does not restore the reasoning that produced the disk. Six weeks later nobody remembers why libssl was downgraded by hand at 2 a.m., and the snapshot faithfully reproduces an accident along with the fix. The rebuildable artifact is the script, not the image, because the script is reviewable in a diff and the image is not. Free boxes make this worse by disappearing on someone else's schedule, so the only durable asset you own is the recipe.

The artifact: a manifest, a noise filter, and a diff

The drill is small enough to fit in a repository and boring enough to run on every dependency change. Start by emitting a sorted fingerprint of the box, then strip the fields that legitimately vary between hosts, then diff one fingerprint against another.

#!/usr/bin/env bash
# manifest.sh — emit a stable fingerprint of a dev box
set -euo pipefail
out="${1:-manifest.txt}"
{
  echo "## kernel";  uname -srm
  echo "## distro";  grep -E '^(ID|VERSION_ID)=' /etc/os-release | sort
  echo "## libc";    (ldd --version 2>/dev/null || true) | head -n1
  echo "## python";  command -v python3 >/dev/null && python3 -VV
  echo "## node";    command -v node   >/dev/null && node --version
  echo "## pip-digest"; command -v pip3 >/dev/null && pip3 freeze 2>/dev/null | sort | sha256sum
  echo "## npm-global"; command -v npm >/dev/null && npm ls -g --depth=0 --parseable 2>/dev/null | sort
  echo "## locale";  locale 2>/dev/null | sort
  echo "## tz";      cat /etc/timezone 2>/dev/null || date +%Z
  echo "## limits";  ulimit -a | sort
  echo "## cpu";     nproc; grep -m1 'model name' /proc/cpuinfo 2>/dev/null || true
  echo "## env";     env | grep -E '^(LANG|LC_|TZ|PYTHONHASHSEED|NODE_OPTIONS)=' | sort
} > "$out"
echo "wrote $out ($(wc -l < "$out") lines)"

The interesting part is what you throw away before comparing, because a noisy diff trains people to ignore diffs. Hostnames, inode counts, load averages, and core counts all vary legitimately between a laptop and a small remote box, and none of them explain a test failure. Filter them explicitly rather than eyeballing the output.

# strip sections whose values are expected to vary across hosts
strip_volatile() {
  awk '/^## (kernel|cpu|limits)$/ {skip=1; next} /^## / {skip=0} !skip' "$1"
}

strip_volatile a.txt > a.norm
strip_volatile b.txt > b.norm
diff -u a.norm b.norm | tee drift.diff || true
test ! -s drift.diff && echo "OK: no drift outside filtered sections"

Run manifest.sh on the box that works, provision a clean free box, replay your bootstrap.sh, and run manifest.sh again there. The diff is the entire deliverable: every line in it is either a fact your bootstrap forgot to pin or a field your filter should have dropped. Commit the diff alongside the fix, because the diff is the bug report you will want next quarter.

Myth 4: If it ran twice, it is reproducible

Running twice on the same host tests almost nothing, since the second run inherits every mutation the first one made. Reproducibility is a property of the pair (recipe, clean host), and the clean host is the hard half of that pair. This is where a free server option earns its place, not as a deployment target but as disposable verification capacity you are willing to destroy. A box you can throw away is more scientifically useful than a box you are afraid to touch.

Myth 5: The model can repair the environment for you

Free model access is genuinely useful for the tedious half of this loop, which is turning a drift diff into a first-draft bootstrap.sh and explaining why a package was pulled in. It is not useful as an authority on what your box should contain, because it cannot see the host and will happily invent a plausible apt-get line. I use a three-strike rule for generated setup scripts: each line must name the package it satisfies, map to a diff hunk or a manifest entry, and survive one clean replay. Lines that fail the rule get deleted rather than debugged.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. In this workflow I use its free model access to draft and repair the bootstrap script described above, and its free server option as the throwaway clean host that the drill rebuilds into; both are operator-supplied availability claims, so check the project page for current limits before you plan around them. The article stands on its own without either, since the manifest and the diff are plain shell.

When the drill is worth the hour

Situation	Run the drill?	Why
Pure-Python service, lockfile committed	Once per quarter	Catches resolver drift cheaply
Native extensions or CUDA wheels	Every dependency bump	glibc and driver drift dominate
Data or state lives on the box	No — fix that first	Rebuild cannot restore data
Regulated audit requirements	No	Use signed, attested images instead
Teaching juniors about environments	Yes, once	The diff teaches faster than prose

Limitations and who should skip this

The drill verifies environment shape, and it says nothing about data durability, secret handling, or whether the free box will exist tomorrow. Free tiers change their limits and availability windows without notice, so treat any borrowed host as a cache and never as a system of record. If your team already ships pinned container images through an attested pipeline, you have solved this problem with better tooling and should not add shell scripts for symmetry. Individuals writing one-off analysis scripts with no dependencies should skip it too, because there is nothing to rebuild.

The corrected mental model is unglamorous: a box is a claim, and a rebuild is the evidence. Keep the manifest in the repository, keep the diff in the pull request, and let the second host be the reviewer that never gets tired. If you want to try the loop without provisioning anything, run the two scripts on the machine in front of you and read what the filter removes — that list is usually longer than expected.

Token Budgets Are Not Task Budgets: A Free-Tier Accounting FAQ

Sam Yang — Mon, 26 Oct 2026 11:33:39 +0000

A developer opens a free coding console on a Monday, asks the agent to rename one config key across forty files, and finds the session closed before the last file is touched. Nothing crashed, no model refused, and the tool reported success each turn; the allowance had simply been spent re-reading the same repository over and over. That small scene explains most of the confusion around free tiers, because the quantity being metered is not the work you asked for.

Four claims circulate in issue threads and conference hallways, and each one survives because it is almost true. Below, every claim gets a counter-measurement, a corrected mental model, and a probe you can run against your own tasks. All arithmetic here is labeled arithmetic rather than measurement, and none of it is a vendor benchmark.

Myth 1: "A token quota is a task quota"

This claim treats the allowance like a fuel tank, where one liter of driving consumes one liter whether you idle or accelerate. Agent sessions do not work that way, because the conversation history and any retrieved repository context are typically re-sent with every turn. The multiplier is turns, not tasks, and a task that needs nine corrections can cost an order of magnitude more than the task it was cloned from.

A rough planning formula keeps the shape honest even when the constants are wrong. Expected input is roughly the sum of system prompt, injected context, and task text, multiplied by the number of turns, plus the growing transcript from earlier turns.

expected_input ≈ (system + context + task) × turns + transcript_growth

If your system prompt and pinned context total 6,000 tokens and a mundane rename takes six turns, you have already spent about 36,000 input tokens before writing a single line of code. Sixty tasks of that shape consume well over two million, which is the arithmetic that surprises people who divided a large allowance by an imagined per-task price.

The probe: measure tasks per 100k tokens

Guessing from character counts is unreliable, because tokenizers differ from the ones you test with and providers report usage differently. The honest approach is to record what the tool actually reports and divide by outcomes rather than attempts. The script below wraps any CLI agent invocation that can emit a usage file, then computes successes per 100k tokens so that a chatty run cannot flatter itself with volume.

#!/usr/bin/env python3
"""budget_probe.py: tasks-per-token accounting for any CLI coding agent.

Usage file contract: the wrapped command must write JSON like
{"input_tokens": 61234, "output_tokens": 812}
If it cannot, the run is recorded with zero tokens and marked unusable.
"""
import json, subprocess, time
from dataclasses import dataclass

@dataclass
class Attempt:
    task_id: str
    ok: bool
    seconds: float
    tokens_in: int
    tokens_out: int
    metered: bool


def run_one(task, cmd, usage_path):
    start = time.monotonic()
    proc = subprocess.run(cmd, shell=True, capture_output=True, text=True)
    elapsed = round(time.monotonic() - start, 2)
    usage = {}
    try:
        with open(usage_path) as f:
            usage = json.load(f)
    except FileNotFoundError:
        pass
    # exit code alone is not correctness; task["verify"] is the judge
    verify = subprocess.run(task["verify"], shell=True, capture_output=True)
    return Attempt(
        task_id=task["id"],
        ok=verify.returncode == 0,
        seconds=elapsed,
        tokens_in=int(usage.get("input_tokens", 0)),
        tokens_out=int(usage.get("output_tokens", 0)),
        metered=bool(usage),
    )


def summarize(attempts):
    metered = [a for a in attempts if a.metered]
    total = sum(a.tokens_in + a.tokens_out for a in metered)
    wins = sum(1 for a in metered if a.ok)
    if not total:
        return {"note": "no usage data; fix the usage file before trusting anything"}
    return {
        "tasks": len(metered),
        "successes": wins,
        "tokens_total": total,
        "successes_per_100k": round(wins / total * 100_000, 2),
        "tokens_per_success": round(total / wins, 0) if wins else None,
    }

Run ten to fifteen small tasks from your own repository, each with a machine-checkable verify command, and store the JSON beside the commit hash. The number that matters is successes_per_100k, because it folds correctness, verbosity, and retry cost into one comparable figure. A cheap model that succeeds after four retries is often more expensive than a slower one that succeeds immediately.

Myth 2: "The free server is a free CI runner"

A session box is built for interactive agent loops: you land in it, run a command, inspect the diff, and continue. A CI runner is built for queued, parallel, hermetic jobs that must terminate on their own with artifacts attached. Confusing the two produces the classic complaint that a free environment is useless because it cannot run a forty-job matrix overnight, which is like complaining that a workshop cannot serve as a warehouse.

The operator-provided free server option is genuinely useful for the first role and generally wrong for the second. Use the decision table below as a starting filter, and treat every row as a hypothesis rather than a rule.

Task shape	Where to run	Why
Single-file edit with a fast test	Local shell	Feedback loop measured in seconds
Multi-file refactor needing repo context	Server session	Keeps the laptop cool and the context stable
Nightly matrix across runtimes	Real CI	Queueing, parallelism, and artifact retention
Reproducing a user bug in a clean image	Server session	Disposable environment, pinned toolchain
Publishing signed release artifacts	Real CI with secrets	Audit trail and credential isolation

Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode exposes free model access and a free server option, and the allowance advertised at the time of writing is ten million tokens; treat any figure you read here, including that one, as expiring, and confirm the live quota in the console before planning around it.

Myth 3: "A bigger context window produces better patches"

Larger windows raise the ceiling on what can be attended to, not the floor on what gets attended. Past a certain size, unrelated files compete with the one function under repair, and the model's effective attention spreads thinner across a longer field. The practical symptom is a patch that is locally reasonable and globally wrong, such as renaming a symbol consistently in code and inconsistently in tests.

Measure context as a variable rather than a virtue. Run the same task set twice, once with a curated three-file context and once with the whole directory tree pasted in, and compare successes_per_100k from the probe above. If the larger context does not pay for itself, the smaller one is not a compromise but an optimization, and the accounting finally says so out loud.

Myth 4: "Free means disposable, so state does not matter"

Ephemeral environments tempt people into skipping pins, and then the same task passes on Tuesday and fails on Thursday for reasons nobody records. The environment is a variable in the experiment, so pin the toolchain explicitly and print it into the run log alongside your results. Three lines of setup cost less than an afternoon of arguing about a flaky failure.

node --version >> run.log 2>&1
python3 --version >> run.log 2>&1
git rev-parse HEAD >> run.log 2>&1

When a run regresses, you can now ask whether the task changed or the ground moved beneath it. Without that line in the log, every disagreement becomes a matter of memory, and memory loses.

Limitations and who should skip this

The probe is only as good as its verify commands, and exit codes are optimistic witnesses that will happily certify a broken build. Token counts also come from two different counting regimes, so compare runs made on the same tool rather than across vendors. Small samples are noisy, cached dependencies skew timings, and quotas move faster than articles do.

Skip this approach if you need contractual guarantees, if your repository context cannot leave your network, or if your workload is a scheduled batch job that belongs in CI. Skip it too if you cannot write a verify command for your task, because without one you are measuring enthusiasm rather than outcomes.

If you want to try the probe somewhere the meter is already free, the MonkeyCode free model access and free server option make a low-stakes starting point, and fifteen tasks are enough to learn something you can defend in a review.

The Repo Is Not the Service: A Portability FAQ for Free AI Coding Tiers

Sam Yang — Tue, 20 Oct 2026 09:39:53 +0000

On a Tuesday, a developer on a small platform team pulled a clean patch out of a free hosted coding tier and used it to open a demo PR. By Thursday, the same task against a self-hosted clone of the same open-source project produced a patch that imported a package nobody could install. The model had not changed, the prompt had not changed, and yet the result had; the seam between a repository and a running service had simply moved. That seam is what this FAQ is about, because most disagreements about free AI coding tiers are really disagreements about which seam people are standing on.

The numbers I ask you to collect below are yours, not mine. Where a figure appears in this article it is illustrative and labelled as such, and the probe is built so that your own ledger can contradict me within an hour. That is the only kind of evidence worth publishing about availability claims.

A repository is a recipe, not a kitchen

A repository tells you how the author cooked; a service tells you what the kitchen looks like at 2 a.m. on a Wednesday. When you clone an open-source coding agent you inherit the recipe, the tool definitions, and the tests, but you do not inherit the model endpoint, the sandbox image, the queue, or the meter. Those four things are the service, they are rarely in the README, and they are exactly what your CI will collide with.

The distinction is not pedantry, it is a budget line. A patch that arrives in four seconds from a hosted tier and ninety seconds from your laptop changes how many iterations a developer attempts before lunch. Same code, same task, completely different loop.

MonkeyCode is an open-source coding agent project that, as of this writing, advertises free model access and a free server option, with a free token allowance the operator states as ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims are operator-supplied and time-sensitive, so treat them as hypotheses to test in your first ten minutes rather than as constants you plan capacity against.

What makes a project like that a useful test subject is not the price, it is that the free path and the self-host path sit side by side. That layout forces you to notice where they diverge instead of discovering it during a sprint review.

The probe: three trials, one ledger

Before arguing about patch quality, record what actually happens when your agent runs. The script below executes any command you hand it three times inside a throwaway clone, resets the working tree between trials, and appends one JSON line per trial with wall-clock time, exit code, touched files, and a hash of the produced patch. It is deliberately dumb: it does not know your provider's billing unit, it does not parse token usage, and it will record a timeout as a perfectly valid result.

#!/usr/bin/env python3
'''parity_probe.py - record what a coding agent did, trial by trial.

Run inside a throwaway clone. The probe resets the tree between trials,
so anything uncommitted will be lost.
'''
import hashlib
import json
import os
import subprocess
import time
from datetime import datetime, timezone
from pathlib import Path

WORK = Path(os.environ.get('WORK', './sandbox')).resolve()
CMD = os.environ['PROBE_CMD']
TRIALS = int(os.environ.get('TRIALS', '3'))
LEDGER = Path(os.environ.get('LEDGER', 'ledger.jsonl'))
TIMEOUT = int(os.environ.get('TIMEOUT', '900'))


def sh(cmd, timeout=120):
    started = time.monotonic()
    proc = subprocess.run(
        cmd, cwd=WORK, shell=isinstance(cmd, str),
        capture_output=True, text=True, timeout=timeout,
    )
    return proc, round(time.monotonic() - started, 2)


def snapshot():
    _, head = sh('git rev-parse HEAD', 30)
    _, status = sh('git status --porcelain', 30)
    _, diff = sh('git diff --binary', 60)
    return {
        'head': head.stdout.strip()[:12],
        'files': sorted({line[3:] for line in status.stdout.splitlines()}),
        'patch_sha': hashlib.sha256(diff.stdout.encode()).hexdigest()[:12],
    }


def reset():
    sh('git checkout -- .', 120)
    sh('git clean -fdq', 120)


def main():
    LEDGER.write_text('')
    for trial in range(1, TRIALS + 1):
        reset()
        try:
            proc, seconds = sh(CMD, TIMEOUT)
            exit_code, stderr = proc.returncode, proc.stderr[-400:]
        except subprocess.TimeoutExpired:
            seconds, exit_code, stderr = TIMEOUT, 'timeout', 'probe timeout'
        record = {
            'trial': trial,
            'at': datetime.now(timezone.utc).isoformat(timespec='seconds'),
            'seconds': seconds,
            'exit': exit_code,
            'stderr_tail': stderr,
            **snapshot(),
        }
        with LEDGER.open('a') as fh:
            fh.write(json.dumps(record) + '\n')
        print(f'trial {trial}: exit={exit_code} {seconds}s '
              f'files={len(record[chr(39)+chr(39)] if False else record["files"])}')


if __name__ == '__main__':
    main()

Fix that print line to files={len(record['files'])} in your copy, then wire it to whatever command drives your agent. A typical session looks like this, and it costs you three runs of your own task rather than a benchmark you did not design:

python3 -m venv .venv && . .venv/bin/activate
git clone --depth 1 git@github.com:your-org/your-repo.git sandbox
cd sandbox && git checkout -b probe-base && git commit -am 'probe baseline'
cd ..
WORK=./sandbox TRIALS=3 TIMEOUT=900 \
  PROBE_CMD='your-agent run --task tasks/add-validation.md' \
  python3 parity_probe.py

Then read the ledger as a table instead of a vibe:

jq -r '[.trial, .seconds, .exit, .patch_sha, (.files|length)] | @tsv' ledger.jsonl

If the three patch hashes differ, you have measured variance on your own task. If they match but the file counts differ wildly, you have measured a different problem: the agent is reaching the same answer through unstable routes, and your reviewer is paying for that instability.

Four myths, corrected

Myth one: open source means the hosted tier and my clone are interchangeable

They are three separate contracts pretending to be one: the source you can read, the endpoint you are pointed at, and the runner that executes tool calls. Version drift in any of the three produces a patch that looks authored by a different engineer. The corrected model is to version all three explicitly, then record them in the same ledger entry the probe already writes.

Myth two: the free tokens are the free tier

The token allowance is the visible meter, and visible meters are excellent at hiding the invisible ones. On a shared free server the scarce resource is usually when you get to run and how often your job is retried, not whether the tokens exist. Think of free parking downtown: the space is free, the tow truck is not, and the tow truck is what your retro will discuss.

Myth three: the same model and prompt produce the same patch

Sampling settings, tool-schema versions, context assembly order, and even the order files appear in a directory listing all nudge the output. This is not a scandal, it is how probabilistic systems behave under a deterministic-looking interface. The corrected habit is to log the knobs alongside the diff, so that a surprise on Thursday can be traced to a changed default rather than to a colleague's imagination.

Myth four: one clean run proves the tier works for your team

A single green run is a sample of one, and samples of one are how teams end up with a demo that cannot survive a second developer. Three trials give you a range, a range gives you a review budget, and a review budget is the number that actually decides adoption. The measured wall clock matters less than the honest answer to whether anyone will read the diff.

Choosing where the loop runs

Placement	You control	You inherit	Expect this failure	Skip it when
Self-hosted clone	endpoint, runner image, cache	upgrade churn, on-call, GPU or CPU bill	your config drifts from upstream within weeks	you cannot staff the maintenance
Hosted free tier	repo, task text, ledger	queue, capacity, tomorrow's defaults	a fast Tuesday and a slow Thursday	you need a contractual uptime figure
Local small model	everything, offline	RAM, heat, patience	plausible patch, wrong import	the task needs more context than your box holds

The table is a decision aid, not a ranking; each row is somebody's correct answer on a different day. The free tier is the cheapest place to falsify the portability claim, which is precisely why the probe belongs there first.

Limitations and who should not bother

The probe does not measure your code review time, which is usually the largest cost in the loop and the hardest to attribute. Renamed files collapse into a single path in the porcelain parsing, so treat the file list as a hint rather than an inventory. Three trials are a range, not a distribution, and the probe itself consumes tokens and queue time while producing nothing shippable.

Skip this approach if your workload requires data residency guarantees you cannot verify, if you need an uptime commitment attached to the agent loop, or if your repository cannot survive a git clean -fd in a throwaway clone. It is also the wrong tool for anyone who will not read the ledger, because an unread ledger is just a slower way to guess.

If you want to test the portability claim rather than accept it, start where the setup cost is lowest and the record-keeping is identical. That is the honest role MonkeyCode's free model access and free server option can play in this experiment: they remove two steps the probe would otherwise have to document, finding an endpoint and finding a machine. The ledger is what stops you from mistaking a short walk to the first patch for a short project.

模型不是瓶颈：关于免费远程编码盒的四个误区

Sam Yang — Mon, 19 Oct 2026 08:21:50 +0000

想象一个只在 CI 中复现的 bug：单元测试在笔记本上通过，集成套件在干净的 Linux 机器上失败，而差异就隐藏在时区设置、缺失的系统包或者你想不到的旧锁文件之中。直觉会说“把整个环境搬走”，于是你把 agent 指向一台远程机器，让它在那边工作，然后等着一个确定性的通过结果回来。你等到的却是一个不同的失败，它比第一个失败更难阅读，因为它已经把机器、网络和模型合并成了一条你无法归因的单一时间线。

MonkeyCode 就是打包了这套设置两部分的项目之一：免费模型访问和免费服务器选项，所以尝试这套工作流的代价主要是你的时间，而不是预算额度。披露：本文是作为 MonkeyCode 产品推广的一部分撰写的。确切的额度，以及免费服务器能持续多久，都是可能不经通知就变化的商业条款，所以请到项目页面上阅读它们，而不是相信任何博客文章里的数字，包括这篇。本文其余部分是关于如何测量，无论这些条款最终是什么样都仍然有用。

误区一：免费远程盒把你的笔记本从等式中移除

它没有移除任何东西，只是重新定位了约束。每次工具调用都要经过网络，因此一个 150 毫秒的往返时间在 40 次调用上，在执行任何模型工作之前就带来了六秒的纯等待。在被测量之前，这种开销是不可见的，因为它分布在数千条日志行中，并且在你的终端里看起来就像模型在思考。

在评估其他任何东西之前，先测量链路：

for i in $(seq 5); do
  /usr/bin/time -f '%e s to a no-op' ssh -o BatchMode=yes box true
done

如果这个数字与你的本地延迟相当，那么托管只是出于环境原因才有意义，例如一致的发行版、干净的缓存，或者有足够的 RAM 运行完整测试套件。如果它比本地延迟高出一个数量级，那么你的 agent 正在为一个空闲循环付出的代价，比它从模型中节省的还要多。修正后的思维模型是：免费盒牺牲了往返时间，换来了环境一致性和更少的本地资源压力。

误区二：免费模型就是付费模型，只是排在队列后面

这是两种不同的现象，而将它们混为一谈会导致你调整错误的旋钮。输出方差意味着相同输入在不同运行之间产生不同的文件树，这是你需要用 diff 去追查的。服务延迟意味着相同输入产生相同输出，但实际耗时从 20 秒变成了 200 秒，这是你需要用墙钟时间去追查的。队列可能加剧第二种症状，却对第一种毫无影响，而模型能力可能改善第一种，却对第二种毫无作用。

最便宜的区分方式是进行三次完全相同的运行并比较结果，而不是仅仅阅读输出。大多数对免费层的失望，都是有人用两个变量诊断一个问题，然后归咎于没有改变的那一个。把每次运行都当作一次实验，而不是一次抽奖。

误区三：token 额度就是预算

预算是墙钟时间乘以重试次数，而 token 计数会系统性地低估重试的成本。一个失败两次后成功的运行，花费了三次审查时间、三次构建时间，以及三次分散你注意力的上下文切换。如果你只记录 token，那么当真正的成本在增长时，账本看起来却风平浪静。

所以记录要围绕结果和所花时间，而不是围绕消耗量：

TIMEFORMAT='%3R s to first green'
time bash -c './bootstrap.sh && npm test --silent'

一个月后再把这个数字与同一分支上的数字进行比较。设置漂移很少会以错误的形式宣告自己，它通常表现为同一套件的墙钟时间缓慢上升，而所有检查仍然通过。

误区四：如果盒是免费的，就把它当作一次性的

一次性盒子在冷启动上最贵，在漂移上也最糟。每次你把它扔掉，就要重新付出安装、索引和依赖解析的成本，而且你无法知道今天的绿色与上周的绿色是否在同一环境中产生。把它当作半持久的：记录它是什么。

一个环境记录只需要四行代码，而且足够短，你真的会去填写：

{ uname -srm; node -v 2>/dev/null; python3 -V; date +%Z; } > env.txt
sha256sum package-lock.json >> env.txt 2>/dev/null

如果连续两次运行的 env.txt 不同，那么将它们的结果进行比较就是在浪费时间。如果你不希望它变化，就固定它，并写下原因。

一个同时测量以上所有内容的账本

工件是一个小工具，它会把每条主张变成一个数字。在仓库的临时副本中运行 N 次相同的命令，并记录退出码、墙钟时间、生成树的哈希以及第一个错误行。下面脚本是一个提案，并未在你的机器上进行基准测试，所以请先在一次性仓库上运行它，再信任任何输出。

#!/usr/bin/env python3
"""runledger.py TARGET_REPO -- CMD...  (one JSON row per identical run)"""
import hashlib, json, os, pathlib, shutil, subprocess, sys, tempfile, time

repo = pathlib.Path(sys.argv[1]).resolve()
cmd  = sys.argv[sys.argv.index('--') + 1:]
runs = int(os.environ.get('RUNS', '5'))

def tree_hash(root):
    h = hashlib.sha256()
    for p in sorted(root.rglob('*')):
        if p.is_file() and '.git' not in p.parts:
            h.update(str(p.relative_to(root)).encode())
            h.update(p.read_bytes())
    return h.hexdigest()[:12]

rows = []
for i in range(runs):
    tmp = pathlib.Path(tempfile.mkdtemp(prefix='run-'))
    shutil.copytree(repo, tmp / 'repo', ignore=shutil.ignore_patterns('.git'))
    t0 = time.monotonic()
    p = subprocess.run(cmd, cwd=tmp / 'repo', capture_output=True, text=True)
    rows.append({'run': i, 'exit': p.returncode,
                 'wall_s': round(time.monotonic() - t0, 2),
                 'tree': tree_hash(tmp / 'repo'),
                 'first_err': (p.stderr.splitlines() or [''])[0][:80]})
    shutil.rmtree(tmp)

print(json.dumps(rows, indent=2))
trees = {r['tree'] for r in rows}
walls = [r['wall_s'] for r in rows]
print(f'distinct trees {len(trees)}/{len(rows)}; wall min/median/max '
      f'{min(walls)}/{sorted(walls)[len(walls)//2]}/{max(walls)}')

如果 distinct trees 等于 1/5，你的设置是确定性的，你的问题在于延迟；如果你的结果是 5/5，那么花在时序上的任何精力都是错位的，直到你收窄任务或降低采样温度。这种区分只需要五分钟，却能节省数小时的争论。

情况	在笔记本上运行	在免费远程盒上运行
针对一个文件的紧凑编辑循环	是	否
完整的集成套件，依赖缓存干净	否	是
测量 N 次运行的输出方差	两种都可以	是，这样可以保持 CPU 空闲
客户数据，或受监管的工件	仅在受控环境中	否

这种方法在哪里失效，以及谁应该跳过

这个方法假设你能够以脚本化方式重放任务，而许多现实任务并非如此。如果 agent 的提示词依赖于聊天上下文，那么你的五次运行就不是同一个实验，数字也会很混乱，因此先把任务固定在一个文件中，然后再测量。

如果你的工作涉及合同、审计记录，或者任何带有数据处理义务的内容，就不要使用免费远程盒；这里没有任何测量可以替代数据保护审查。如果你需要具有确定生命周期和供应商支持条款的可复现结果，付费层级并不是一种浪费，而是问题的成本。如果你的团队无法维护环境记录，那么任何托管方式都无法修复其所掩盖的漂移问题。

如果你宁愿在别人的服务上失败，也不想在自己的笔记本上失败，那么这份账本值得你花一个下午。想尝试的话，可以到项目页面查看当前条款，并从三次完全相同的运行开始；之后到达的每一个数字，都会比任何公告的额度更有讨论价值。

MonkeyCode provides free models that can run this workflow. A free server option is enough to reproduce the setup.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Zero Cost Is Not Zero Variance: Four Myths About Repeating an AI Coding Run

Sam Yang — Sun, 18 Oct 2026 07:05:34 +0000

Imagine the same prompt, the same repository, and the same command, run on two different afternoons. On the first afternoon the agent finishes quickly and the test suite is green; on the second it edits four extra files and leaves two tests red. Nothing in the repository changed between those runs, and that is precisely why the second outcome is difficult to defend in a code review.

Most comparisons of free AI coding tiers begin and end with price. The question that actually changes your workflow is whether a tier behaves consistently enough that you can build a habit around it, because a cheap tool with a wide outcome spread quietly spends your review time instead. This article converts that spread into a small ledger you can run in one sitting, and it retires four claims that developers repeat whenever tiers are compared.

The format is a myth-busting FAQ, because myths are what get repeated in chat threads while the measurements stay private. Each claim below comes with a cheap check and a corrected mental model, and none of the checks requires a paid plan to be interesting.

Myth 1: The same prompt produces the same run

Repeat a task eight times against the same commit and you will rarely get eight identical diffs. Sampling settings, tool-call ordering, network retries, and the order in which files are read all push the run in slightly different directions. A single green run is therefore an anecdote about one sample, not a property of the tier.

The corrected mental model is that you are not buying a function, you are sampling a distribution. That sounds abstract until you write the distribution down, and writing it down takes less effort than arguing about it in a pull request. Once you have rows, the conversation moves from "it worked for me" to "median acceptable, tail unacceptable".

Myth 2: A green run verifies the workflow

A passing test suite proves that one sample passed, and it says nothing about how often the next sample will pass. The failure mode is subtle: the agent's diff and the test outcome are recorded together in your head, but only the happy version survives into the team's memory. Recording both columns for every run is the entire fix.

The corrected model separates process from task. Exit code zero tells you the command finished; a test result tells you the change was acceptable; a diff count tells you how much review surface the change created. Those three signals disagree often enough that collapsing them into one boolean will mislead you.

Myth 3: The median is the number that matters

Medians are comfortable and tails are expensive, and free tiers tend to produce tails. If seven runs finish in a few minutes and the eighth takes an hour, your median looks healthy while your afternoon is gone. Anyone who has waited on a stuck job knows the average is not the number that ruins the day.

The corrected model is a tail ratio: worst run divided by median run, computed from the same ledger rows. A tier with a stable median and a large tail ratio is fine for exploration and hostile for anything with a deadline. Compute both numbers before you decide which kind of tool you are holding.

The ledger: eight runs, four columns

The harness below is deliberately boring, and you should adapt the one line that invokes your agent. It records the commit you started from, resets the worktree between runs, and appends one CSV row per attempt. Treat it as a proposal to run on your own task rather than as a benchmark result, since the numbers you get will depend on your repository.

#!/usr/bin/env bash
# variance_ledger.sh — repeat one task N times, record one row per run
set -uo pipefail

N="${N:-8}"
TASK="${TASK:-tasks/fix_import_order.md}"
OUT="ledger/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$OUT"

git rev-parse HEAD > "$OUT/base_commit.txt"
git status --porcelain > "$OUT/base_dirty.txt"   # should be empty

for i in $(seq 1 "$N"); do
  start=$(date +%s)
  # Replace this line with your agent's non-interactive entry point.
  agent run --task "$TASK" --no-interactive > "$OUT/run_$i.log" 2>&1
  code=$?
  end=$(date +%s)
  added_deleted=$(git diff --numstat | awk '{a+=$1; d+=$2} END {print (a+0) " " (d+0)}')
  printf '%s,%s,%s,%s\n' "$i" "$((end-start))" "$code" "$added_deleted" >> "$OUT/ledger.csv"
  git checkout -- . >/dev/null 2>&1
  git clean -fdq
done

The reset step is what makes the rows comparable, and skipping it is the most common way to poison a ledger. Running without a clean worktree means run two starts from run one's leftovers, and the whole table then measures compounding edits instead of tier behaviour. Check base_dirty.txt before you trust anything downstream.

Summarising takes a short awk script, because pulling in a dataframe library for eight rows is unnecessary ceremony.

awk -F, '{wall[NR]=$2; n=NR}
END{
  for(i=1;i<=n;i++) for(j=i+1;j<=n;j++) if(wall[j]<wall[i]){t=wall[i];wall[i]=wall[j];wall[j]=t}
  med=(n%2)?wall[(n+1)/2]:(wall[n/2]+wall[n/2+1])/2
  printf "runs=%d min=%ss median=%ss max=%ss tail_ratio=%.2f\n", n, wall[1], med, wall[n], wall[n]/med
}' ledger/*/ledger.csv

Add the task verdict yourself, because wall time cannot tell you whether a diff was correct. A second column holding the exit status of your test command is usually enough, and it keeps the ledger honest when a fast run produces a bad patch.

Myth 4: Variance is a property of the model

Variance usually arrives from the environment wrapped around the model. A shared shell with a warm package cache, a locale that formats numbers differently, and a proxy that silently retries a request are three common sources that have nothing to do with the weights. If you change the environment between runs, you have measured the environment.

The corrected model treats the agent as one variable among several, and it forces you to hold the others fixed or record them. A warm cache and a different locale are environment variables in the literal sense, and both explain why two teams can disagree about the same tier while both are reporting honestly.

Where free access and a free server actually help

Repetition is the step everyone skips, because repetition is what costs money and occupies a machine. MonkeyCode's free model access and free server option change the arithmetic of that step, since an eight-run ledger stops being an expense and becomes an ordinary afternoon task. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The honest version of that claim is narrower than a slogan. Free access makes sampling practical at small n, and a free server keeps the loop off your laptop while you review the previous run; neither makes a tier deterministic. Check the project's own documentation for current allowances and machine characteristics, because quotas and hardware are exactly the details that go stale fastest.

Limitations worth stating plainly: eight runs is a small sample, diff size is not correctness, wall time includes network noise you cannot control, and every repetition consumes allowance you may want later. The ledger also assumes a non-interactive entry point, which not every agent exposes cleanly. Treat the output as a screen for whether a tier deserves careful measurement, not as a benchmark.

Who should not use this approach

Teams that need deterministic output, contractual latency, or regulated audit trails will not be served by sampling a free tier at all. Anyone whose task takes an hour per run should shrink the task before running any ledger. And if review capacity is the real bottleneck, variance reduction matters more than cost reduction, so pay for stability rather than collecting cheap rows.

The myth-busting habit generalises beyond any single product: write the claim down, find the cheapest observation that could refute it, and keep the rows. Eight runs and four columns will not settle a platform debate, but they will stop you from repeating a one-sample story as if it were a property. If you want to try the ledger, the free tier is enough to learn whether your own task is stable before you commit real budget to it.

Testing Free AI Coding Tiers: Four Myths and One Probe

Sam Yang — Sat, 17 Oct 2026 06:06:36 +0000

A recurring scene in team chats: one engineer posts a screenshot of a free tier answering a genuinely hard question, another posts a monthly cost report, and both are correct. The screenshot captures a single exchange on a single afternoon, while the report aggregates retries, resent context, and prompts nobody saved. What follows is usually an argument about trusting a vendor, when it could be an argument about a file in the repository.

Free AI coding tiers have become ordinary infrastructure, and the conversation around them has drifted toward vibes — visible in the recent wave of 'AI fatigue' posts as much as in launch threads. The useful correction is not more opinion; it is a smaller, timestamped claim that someone else can re-run. Four myths keep surviving contact with that standard, and each one falls to a probe that takes about half an hour to assemble.

Myth 1: open source means the model runs next to you

Most open-source AI coding projects are clients, gateways, or orchestration layers, while the weights sit behind an HTTP boundary you do not control. 'Open source' tells you which part you can audit and patch, and it says nothing about where inference happens or what happens when the network drops. That distinction matters for air-gapped review, for data-handling sign-off, and for anyone who assumes graceful local fallback.

You can settle the question with an egress trace instead of a README argument. Run strace -f -e trace=connect -o /tmp/egress.txt python3 ask.py ping and inspect the resolved addresses with grep -o 'sin_addr=inet_addr("[^"]*")' /tmp/egress.txt | sort -u. While a slow request is still in flight, ss -tunp | grep python3 shows the live sockets and remote ports, which is usually enough to answer the only question a reviewer cares about.

The corrected mental model is a thin local client plus a remote service that carries its own terms, quotas, and retention policy. You can self-host the client in minutes and you cannot self-host the model in minutes, so keep the two separate in your architecture notes.

Myth 2: a free server is a workstation you can leave running

Free compute is usually ephemeral by design: the container restarts, the disk resets, and the CPU is shared with whoever else is on the host that minute. None of that makes it useless, but it does make it a poor home for a long-running daemon or for state you cannot rebuild from the repository. The failure mode is quiet, because the first session feels identical to a paid box.

Fingerprint the environment before and after instead of trusting your memory of it:

fp() {
  { uname -a; python3 -V; nproc
    python3 -m pip freeze 2>/dev/null | sort | sha256sum
    env | grep -viE 'token|key|secret' | sort | sha256sum
  }
}
fp > /tmp/fp_t0.txt        # capture at session start
# ... a few hours later, in a fresh shell ...
fp | diff -u /tmp/fp_t0.txt - || echo 'environment drifted'

Keep volatile lines out of the fingerprint if you only care about structure, since memory and disk figures move on their own and produce noise. A drift report that fires on every run teaches you to ignore it, which is worse than having no report at all.

Myth 3: the usage field tells you what you spent

Provider usage fields are useful and they are not an invoice, because they count a request shape you may not recognise. A system prompt gets prepended, a failed call gets retried, and an agent loop resends the whole conversation on every turn, so prompt tokens can exceed your visible message by a wide margin. The gap is not fraud; it is the difference between what you typed and what was sent.

The pair of scripts below reconciles the two views. ask.py writes one JSONL record per exchange, and token_audit.py compares the provider's counts against a local count of the exact strings you transmitted.

#!/usr/bin/env python3
# ask.py - template; endpoint and model id come from your own dashboard
import datetime, json, os, time, urllib.request, uuid

BASE = os.environ['BASE_URL'].rstrip('/')      # OpenAI-compatible base URL
KEY = os.environ['API_KEY']
MODEL = os.environ['MODEL_ID']

def ask(prompt: str, out: str = 'runs.jsonl') -> dict:
    payload = {'model': MODEL, 'temperature': 0,
               'messages': [{'role': 'user', 'content': prompt}]}
    req = urllib.request.Request(
        f'{BASE}/chat/completions',
        data=json.dumps(payload).encode(),
        headers={'Content-Type': 'application/json',
                 'Authorization': f'Bearer {KEY}'})
    t0 = time.time()
    with urllib.request.urlopen(req, timeout=120) as r:
        resp = json.loads(r.read())
    rec = {'ts': datetime.datetime.now(datetime.timezone.utc).isoformat(),
           'run_id': str(uuid.uuid4()),
           'latency_ms': round((time.time() - t0) * 1000),
           'request': payload, 'response': resp}
    with open(out, 'a') as f:
        f.write(json.dumps(rec) + '\n')
    return resp

if __name__ == '__main__':
    print(ask('Reply with the single word: ok')['choices'][0]['message']['content'])

#!/usr/bin/env python3
# token_audit.py - reconcile reported usage against local counts
import hashlib, json, statistics, sys
from pathlib import Path

def count_tokens(text: str) -> int:
    try:
        import tiktoken
        return len(tiktoken.get_encoding('cl100k_base').encode(text))
    except ImportError:
        return max(1, len(text) // 4)      # coarse estimate, labelled as such

def main(path: str) -> None:
    overheads, rows = [], []
    for i, line in enumerate(Path(path).read_text().splitlines(), 1):
        if not line.strip():
            continue
        rec = json.loads(line)
        sent = '\n'.join(m.get('content', '') for m in
                         rec.get('request', {}).get('messages', []))
        choice = (rec.get('response', {}).get('choices') or [{}])[0]
        reply = choice.get('message', {}).get('content') or ''
        usage = rec.get('response', {}).get('usage') or {}
        local_in, local_out = count_tokens(sent), count_tokens(reply)
        rep_in, rep_out = usage.get('prompt_tokens'), usage.get('completion_tokens')
        oh = None if rep_in is None else rep_in - local_in
        if oh is not None:
            overheads.append(oh)
        rows.append((i, hashlib.sha256(reply.encode()).hexdigest()[:10],
                     local_in, rep_in, oh, local_out, rep_out))
    print('ln, reply_sha, local_in, reported_in, overhead, local_out, reported_out')
    for r in rows:
        print(r)
    if overheads:
        print(f'median prompt overhead: {statistics.median(overheads):+.0f} tokens'
              f' over {len(overheads)} runs')

if __name__ == '__main__':
    main(sys.argv[1] if len(sys.argv) > 1 else 'runs.jsonl')

When usage is missing entirely, the script reports that instead of guessing, and you fall back to the local estimate with a visible caveat. That honesty is the actual deliverable: a number with a method beats a number with a confident tone.

Myth 4: one good run means the tier works

A single success proves the path existed once, which is roughly what a screenshot proves. Free tiers change shape between morning and evening, model identifiers get repointed, and rate limits appear exactly when a deadline does. Time-spaced repetition is the cheapest defence, and it costs three prompts.

The plan I follow is deliberately boring. Send the same temperature-zero prompt three times, at least six hours apart, and keep every record written by ask.py. Then run python3 token_audit.py runs.jsonl and compare the reply hashes, the latencies, and the median prompt overhead across all three records. A stable hash is a bonus rather than an expectation, and a shifting overhead tells you the wrapper or the system prompt changed underneath you.

Choosing where a job should run

The table is the part I reuse most, because it converts an argument into a routing decision.

Job	Laptop plus free model access	Free server	Paid or self-hosted
One-off question or syntax check	fine	unnecessary	waste
Repo-wide mechanical edit	fine, on a branch	fine, after pushing	when the run must survive a restart
Multi-hour agent loop	risky	risky	yes
Anything touching customer data	only after review	only after review	depends on the contract
Benchmarking model quality	no	no	pinned version, pinned hardware

The rightmost column is not an argument for spending money; it is where reproducibility becomes a requirement instead of a preference. Most exploratory work never gets there, which is why the free columns genuinely earn their place.

Limitations and who should skip this

The probe only sees what your client can see, so server-side accounting, caching discounts, and queueing stay invisible to it. The token count without tiktoken installed is coarse enough to be wrong by double digits on unusual text, and it measures volume rather than response quality. A free server that wipes its disk will also wipe the runs.jsonl you forgot to commit, so push early and push often.

Skip this approach if you need a contractual uptime guarantee, if your data cannot leave a controlled network, or if the real question is whether one model beats another. Quality comparisons need a pinned model version, a fixed evaluation set, and more patience than a thirty-minute probe can supply.

Where MonkeyCode fits in this workflow

MonkeyCode is an open-source AI coding project offering free model access and a free server option, and both availability claims belong to the operator rather than to this article's measurements. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Inside the workflow above they serve a narrow purpose: free model access lets ask.py produce the first three records without a billing conversation, and the free server gives you a second location to fingerprint, which is precisely the drift check from Myth 2.

Treat any quoted allowance as a dated claim rather than a permanent property, because free-tier numbers printed in older posts go stale faster than the code does. Check the project's current documentation before you plan a week of work around it, and write the date next to the number in your notes. If you run the three probes and keep the output, you will hold a review comment worth more than any screenshot, which is the only endorsement I am comfortable making here.

If you do try it, the useful thing to share back is not 'it worked' but the overhead column from token_audit.py.

Audit Your Free AI Coding Tier: Five Claims, One Run Ledger

Sam Yang — Fri, 16 Oct 2026 05:04:22 +0000

It is 23:40, and the same prompt has produced a clean patch in one browser tab and a mangled one in another, so you start deleting variables: model, timezone, working directory, that stray file from three commits ago. Two hours later you have a diff you cannot explain and a suspicion that the free tier you are leaning on is not the thing you think it is. The uncomfortable part is that most of these questions do have answers, but only if you recorded evidence while the run was still happening.

The five claims below are the ones I keep seeing repeated in free-tier discussions, and each one gets the same treatment here: state it plainly, try to falsify it with a command you can run, then write down the mental model that survives. The artifact is a small run ledger you can drop into any repository in about fifteen minutes. Disclosure: This article was prepared as part of MonkeyCode's product outreach. I mention the project's free model access and free server option later because they are what I used to run the loop, but the harness itself is provider-agnostic and wraps any CLI you can invoke.

Myth 1: The free token allowance is a budget

The claim sounds reasonable enough: you have a fixed amount of free usage, therefore you have a budget, therefore you can plan work around it. What a token allowance actually limits is consumption, while the thing that decides whether your week goes well is the number of runs that end in an accepted change. Two developers can burn identical token counts and get wildly different value, because one retries a failing prompt eleven times while the other spends one run writing a test first. Replace the budget metaphor with a yield metric, and the allowance stops being a planning input.

The metric worth tracking is tokens or runs per accepted change, where "accepted" means a human reviewed the diff and kept it. That annotation is one extra line after review:

# after human review, mark the run as kept
printf '{"run_id":"%s","kept":true}\n' "$RUN_ID" >> run-ledger.jsonl

Myth 2: A free server is a pinned environment

The claim goes: if the code executes on the provider's machine, the environment is fixed, therefore the results are reproducible. A remote machine fixes where the process runs, not what the process sees, and those two things are easy to confuse when the workspace persists between sessions. Timezone, locale, installed tool versions, and a dirty working tree all float unless something captures them at run time. The model that survives is simple: a server is a location, a fingerprint is an environment, and only the fingerprint travels with the run.

Myth 3: The same model label means the same behaviour

If both runs report the same model name, the outputs should be comparable, or so the claim goes. Sampling defaults, context truncation, hidden system prompts, and routing between backends all break that assumption quietly, and none of them appear anywhere in the model label. Compare runs by input and environment fingerprint instead, and treat the label as a hint rather than a guarantee. When two runs disagree, the fingerprint tells you whether you changed something or the service did.

Myth 4: A zero exit status means a verified run

Exit status 0 only means the process finished without signalling an error, which is fully compatible with an empty patch, a patch to the wrong file, or a test suite that quietly skipped itself. Hash the artifacts and compare them against an expected digest, and the green checkmark becomes evidence rather than vibes. The check is one line in CI or your shell:

sha256sum -c expected.sha256   # fails loudly when the artifact drifts

Myth 5: Free means the run costs nothing

The tokens may cost nothing, but the review never does, because every unrecorded run pushes work onto a human who must reconstruct context later. Recording a fingerprint and a log hash while the run is in flight costs a few seconds and removes most of that future reconstruction. The number that matters is not price per token but elapsed human minutes per accepted change.

The artifact: a fifteen-minute run ledger

The harness wraps any CLI invocation, computes an environment fingerprint, times the run, and appends one JSON line per execution. Treat it as a template rather than a tested release: run shellcheck over it, and adapt the command capture to whatever your agent CLI actually accepts.

#!/usr/bin/env bash
# run-ledger.sh — record what one AI coding run actually depended on.
set -euo pipefail

LEDGER="${LEDGER:-./run-ledger.jsonl}"
RUN_ID="${RUN_ID:-$(date -u +%Y%m%dT%H%M%SZ)-$$}"
RUN_CMD="${RUN_CMD:?export RUN_CMD to the CLI invocation you want to audit}"

fingerprint() {
  {
    uname -srm
    printf 'TZ=%s LC_ALL=%s LANG=%s\n' "${TZ:-unset}" "${LC_ALL:-unset}" "${LANG:-unset}"
    python3 -c 'import platform;print("python",platform.python_version())' 2>/dev/null || true
    node --version 2>/dev/null || true
    git rev-parse HEAD 2>/dev/null || echo no-git
    git status --porcelain 2>/dev/null | sha256sum | cut -c1-16
  } | sha256sum | cut -c1-16
}

FP="$(fingerprint)"
started="$(date +%s)"
set +e
LOG="$(eval "$RUN_CMD" 2>&1)"
STATUS=$?
set -e
finished="$(date +%s)"

python3 - "$RUN_ID" "$FP" "$STATUS" "$((finished - started))" "$LOG" "$LEDGER" <<'PY'
import hashlib, json, sys
run_id, fp, status, seconds, log, ledger = sys.argv[1:7]
record = {
    "run_id": run_id,
    "env_fingerprint": fp,
    "exit_status": int(status),
    "wall_seconds": int(seconds),
    "log_sha256": hashlib.sha256(log.encode()).hexdigest()[:32],
    "log_bytes": len(log.encode()),
}
with open(ledger, "a", encoding="utf-8") as fh:
    fh.write(json.dumps(record, sort_keys=True) + "\n")
print(json.dumps(record, sort_keys=True))
PY

Two runs, one variable, and a comparison that takes four seconds is the whole point of the exercise:

RUN_CMD='your-agent-cli --prompt "add a retry test for fetch_user"' ./run-ledger.sh
TZ=Asia/Tokyo RUN_CMD='your-agent-cli --prompt "add a retry test for fetch_user"' ./run-ledger.sh
jq -r '[.run_id,.env_fingerprint,.exit_status,.log_sha256]|@tsv' run-ledger.jsonl

If both rows carry the same fingerprint but different log hashes, you now have evidence that behaviour moved outside your control, which is a claim you could not have made last night. Running a loop like this needs two things: somewhere to execute it and something to execute it with, which is where the project's operator-supplied free tier becomes relevant, since it advertises free model access for the agent call and a free server option for hosting the ledger job. Token allowances, model line-ups, and server specifications change without notice, so read the project's current page before you plan a quarter around any of them. If you want to try the harness end to end, wrap a single command first and confirm one accepted change before automating anything.

Choosing your level of evidence

Situation	What the ledger alone buys you	What you should add
Learning an API, one-off scripts	Drift detection between attempts	Nothing more; keep it disposable
Shared repository with several contributors	A shared record of what each run saw	Commit fingerprints in the PR template
Recurring flaky agent behaviour	Proof that inputs were identical	Raw fingerprint text, not only the hash
Deterministic fixtures or snapshot tests	Almost nothing	Pinned toolchain and a fixed sampling setup
Regulated audit evidence	Not sufficient on its own	Your existing signed, tamper-evident tooling

Limitations and who should not use this

The ledger only measures what you wrap, so any work that happens inside a chat window and never reaches the command line stays invisible to it. Hashing a log detects drift but says nothing about correctness, which means the harness is a complement to tests you already trust, never a replacement. Because the fingerprint is a hash, it tells you that something changed rather than which variable changed, so print the raw block as well while debugging. Anyone who needs a signed, tamper-evident audit trail should keep using their compliance tooling instead of a personal JSONL file, and anyone whose runs are purely interactive and never revisited should skip the recording tax entirely, since it buys nothing without review.

These myths survive because each one saves a little effort today and charges interest tomorrow, which is exactly the trade a free tier makes easy to accept. If you keep only two habits from this, keep the fingerprint and the log hash, because together they answer the question that started the night. Then run the same prompt twice tomorrow with a different TZ, and let the ledger tell you which part of the system you actually control.

Cloneable Is Not Reproducible: Four Myths About Free AI Coding Tiers

Sam Yang — Thu, 15 Oct 2026 11:17:39 +0000

At 9:40 on a Tuesday, two engineers cloned the same repository at the same commit, ran the same test command, and got different results. Nothing in the repository explained the difference, because the deciding variable lived in one laptop's shell profile and a cache directory the second machine had never populated. The pair had spent the previous week wiring an AI coding loop on free model access and a free server, and nobody had asked whether the loop was reproducible at all.

That scene is a composite illustration rather than a logged incident, and the scripts below exist so you can replace it with your own measurements. The distinction matters, because recent AI news has followed a familiar shape: model capability improves quickly while the surrounding practice of verifying and reproducing changes lags behind it. Free tiers widen the gap, since a convincing demo costs nothing and a reproducible workflow costs discipline.

MonkeyCode is an open-source project that advertises free model access and a free server option, and the operator-supplied brief for this article cites a free allowance of 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Allowances, available models, and server limits change with time and load, so treat every figure in this paragraph as a claim to verify on the project page rather than as a constant you can design around.

The four myths below circulate widely in issue threads, code reviews, and internal documentation. Each one is cheap to believe and expensive to act on, and each one has a measurement that settles the argument.

Myth 1: If the repository clones, the environment reproduces

Cloning copies files, not the machine state that decides whether those files run. Dependency resolution, locale, timezone, toolchain patch versions, and untracked caches all change outcomes while leaving the git tree identical. This is why "works on my machine" survives every migration, and why a fresh server can fail a suite that passed locally minutes earlier. The corrected mental model is narrow and testable: reproducibility is a recorded set of inputs plus a deterministic entrypoint, and anything unrecorded is a coin flip.

The script below records those inputs without leaking secrets, because it stores environment variable names rather than their values. Note that a manifest is not a lock: it tells you when two machines differ, not how to make them identical, and secret values still have to arrive through your existing channel.

#!/usr/bin/env python3
"""env_manifest.py - record the inputs that decide whether a fresh clone behaves."""
import hashlib, json, os, platform, subprocess, sys, time


def shell(cmd: str, timeout: int = 20) -> str:
    try:
        done = subprocess.run(cmd, shell=True, capture_output=True,
                              text=True, timeout=timeout)
        return done.stdout.strip()
    except Exception as exc:
        return f"unavailable:{exc.__class__.__name__}"


def file_sha256(path: str) -> str:
    try:
        with open(path, "rb") as handle:
            return hashlib.sha256(handle.read()).hexdigest()[:16]
    except OSError:
        return "missing"


manifest = {
    "recorded_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "git_rev": shell("git rev-parse HEAD"),
    "git_dirty": bool(shell("git status --porcelain")),
    "python": sys.version.split()[0],
    "platform": platform.platform(),
    "locale": os.environ.get("LANG", ""),
    "timezone": shell("date +%Z"),
    "lockfile_sha256": file_sha256("requirements.txt"),
    # Names only, never values: values leak credentials into build logs.
    "env_names": sorted(k for k in os.environ if k.startswith(("APP_", "CI_"))),
}
print(json.dumps(manifest, indent=2, sort_keys=True))

Run that on the laptop and on the free server, diff the two JSON files, and the conversation stops being about vibes. In practice the diff almost always shows a different Python patch version, a dirty working tree, or a missing lockfile hash. Two lines of output are usually enough to explain a week of "flaky" failures.

Myth 2: Free model access makes the model the bottleneck

The intuition here is reasonable and usually wrong. Teams assume the constraint is model latency or the token allowance, then discover that the real consumer is re-sent context, since every failed test round ships the entire conversation back to the endpoint. A loop that converges in two rounds and a loop that thrashes for nine can differ by an order of magnitude in tokens while producing the same single commit. The unit worth optimizing is therefore tokens per verified change, not tokens per request.

The ledger below is a template written against an OpenAI-compatible /v1/chat/completions endpoint; response field names vary between providers, so check the parsing against your own server before trusting the totals. It logs the exact model string the endpoint returns, because a stable model name in your config does not guarantee a stable model behind the endpoint.

#!/usr/bin/env python3
"""loop_ledger.py - log tokens against verified changes, not against requests."""
import json, os, subprocess, time, urllib.request

ENDPOINT = os.environ["LLM_ENDPOINT"].rstrip("/")
MODEL = os.environ["LLM_MODEL"]
GATE = os.environ.get("GATE_CMD", "pytest -q --maxfail=1")
LEDGER = "loop_ledger.jsonl"


def call(messages, **params):
    body = json.dumps({"model": MODEL, "messages": messages, **params}).encode()
    request = urllib.request.Request(
        f"{ENDPOINT}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
        },
    )
    with urllib.request.urlopen(request, timeout=180) as response:
        payload = json.load(response)
    usage = payload.get("usage") or {}
    # Assert instead of defaulting if silent zeroes would mislead your totals.
    if "prompt_tokens" not in usage:
        raise RuntimeError("endpoint returned no usage block")
    return payload["choices"][0]["message"]["content"], {
        "prompt_tokens": usage["prompt_tokens"],
        "completion_tokens": usage.get("completion_tokens", 0),
        "model_returned": payload.get("model", "unknown"),
    }


def gate():
    started = time.monotonic()
    done = subprocess.run(GATE, shell=True, capture_output=True, text=True)
    return done.returncode, round(time.monotonic() - started, 2)


def record(task, round_no, usage, exit_code, gate_seconds, wall_seconds):
    row = {"task": task, "round": round_no, "exit_code": exit_code,
           "gate_seconds": gate_seconds, "wall_seconds": wall_seconds, **usage}
    with open(LEDGER, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(row) + "\n")

Two derived numbers carry most of the insight. Divide the summed prompt and completion tokens of rounds whose exit_code was zero by the count of tasks that reached green, and you get tokens per verified change. Divide the prompt tokens in a task's final round by the prompt tokens in its first round, and you get context amplification, which is the number that quietly eats a free allowance. Truncating or summarizing context attacks amplification directly, while a faster endpoint does nothing for it.

Myth 3: A free server is an environment

A running process is not an environment, however convenient the shell feels. Free servers tend to be ephemeral by design, so databases, caches, and installed system packages may disappear between sessions while your mental model still assumes they are yours. The rule that survives this is blunt: anything you cannot rebuild from the repository does not exist on the server, and anything that exists only on the server is a future incident. Snapshot discipline is what converts borrowed capacity into a usable runner.

#!/usr/bin/env bash
# bootstrap.sh - install, or rebuild from zero and prove the rebuild is clean.
set -euo pipefail

install() {
  python3 -m venv .venv
  . ./.venv/bin/activate
  pip install --require-hashes -r requirements.txt
}

verify() {
  git clean -xfd          # drop every untracked file, including caches
  install
  pytest -q --maxfail=1
}

case "${1:-install}" in
  install) install ;;
  verify)  verify ;;
  *) echo "usage: $0 [install|verify]" >&2; exit 2 ;;
 esac

Run verify in a scratch checkout, never where uncommitted experiments live, since git clean -xfd is deliberately destructive. A clean rebuild that passes is the only evidence that your environment lives in version control. The same command then works on a laptop, in CI, and on the free server, which is usually the moment a team stops debugging machines.

Myth 4: Faster tokens mean faster delivery

Token throughput is the least interesting number in the pipeline, because wall clock is dominated by test runtime, review latency, and queueing. A loop that emits a thousand tokens per second and needs nine correction rounds still lands behind one that emits slowly and converges once. The honest metric is elapsed time from task opened to change merged, measured on the same tasks over time rather than on a single lucky afternoon. Once you accept that framing, the choice between free and owned capacity becomes a question about blast radius.

Situation	Free model access plus free server	Owned or paid environment
Prompt and script prototyping	Fits: disposable work, low blast radius	Unnecessary
Generating tests for existing modules	Fits when a human reviews the assertions	Same, provided review exists
Mechanical refactors behind a strong gate	Fits: the gate is the safety net	Required when no gate exists
Latency-sensitive user paths	Does not fit: shared capacity, variable timing	Required
Regulated or customer-identifying data	Does not fit without a data-handling review	Required
Unattended multi-hour autonomy	Does not fit without an external stop condition	Required

The corrected mental model

Three ideas replace the four myths. Record your inputs, because a reproduction you cannot diff is a rumour rather than a fact. Measure tokens per verified change, because convergence is the variable that free allowances actually react to. Treat borrowed capacity as a runner you can throw away, because state that cannot be rebuilt from the repository is state you do not own.

None of this requires paid infrastructure, and all of it survives a change of provider. That property matters more than any single endpoint, since free availability is exactly the part of your stack that can change without notice. If your ledger and your manifest live in the repository, a migration costs an afternoon instead of a rewrite.

Limitations and who should skip this

This workflow adds bookkeeping, and small solo prototypes rarely earn it back. Teams handling regulated or customer-identifying data should resolve the data-handling question before optimizing anything else, and latency-bound user paths generally do not belong on shared free capacity at all. The ledger also only counts what the endpoint reports: if usage fields are missing, you will see zeroes and a falsely cheap loop, which is why the template raises instead of defaulting. Finally, skip the whole approach if you cannot commit to a verification gate, because every myth above assumes a test command stands between the model and main.

If you want to try the loop on a repository of your own, MonkeyCode's free model access and free server option are a low-cost place to run the first few hundred rounds. Clone the project, run bootstrap.sh verify, point loop_ledger.py at your endpoint, and let the ledger tell you whether the free tier is a bottleneck or a bargain.

A Shared Shell Is Not an Environment Record: A Myth-Busting FAQ

Sam Yang — Tue, 13 Oct 2026 00:48:21 +0000

A platform reviewer opened a pull request whose description already read like a finished release note for a small path fix. The author wrote that a coding model had drafted the patch and that a free remote server had turned the suite green. The diff changed a path join, a short timeout, and a helper that built temporary directories for the tests. Nothing in the thread recorded the operating system, the lockfile digest, the clock source, or the exact command.

That gap appears often once free model access and a free server option make the first run feel cheap. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those two availability claims come from the operator for this draft, and they are not a hardware sheet or a frozen quota. This article will not name models, token caps, machine sizes, or durations, because no primary excerpt was attached and those figures move.

The useful question is which facts a later reader can still check after the remote session has disappeared. A myth-busting pass helps here, because the repeated claims sound practical and still smuggle in an identity the log never had. The corrected picture is closer to a lab notebook than to a stamp on a finished package. What follows keeps that notebook small enough that a reviewer can paste it beside an ordinary pull request.

A green remote command is not a signed machine

Developers often treat a successful remote command as if the machine itself had signed the result. A shared coding server can be rebuilt, rescheduled, or returned to a pool between two working afternoons. The same shell string can then pass for a different reason than the one the author still remembers. A corrected model treats the green line as a claim that still needs a fingerprint captured beside it.

A model draft plus a pass is not a review

Another repeated claim says the model already looked at the code, so human review can stay light. A fluent patch can still hide a path assumption, a race, or a test that passes in only one directory layout. Free model access lowers the cost of a first draft, and it does not lower the cost of a production mistake. The corrected model is that the assistant proposes text, while the reviewer still owns the failing case.

Yesterday's free server is not today's bench

A third claim treats yesterday's free server session as a stable bench for today's later comparison. Without a recorded image identity, two runs can differ in package mirrors, compiler patches, or disk layout. Those differences rarely announce themselves inside a short success message printed by the test runner itself. The corrected model compares environment fingerprints first, and only then compares the pass or fail outcomes.

A later green retry erases the clue

A fourth claim says extra retries are harmless as long as the last attempt on the server passes. Each retry can land on a warmer cache, a different process state, or a file the previous attempt created. If you delete the failing logs, you also delete the clue that success depended on leftover state. The corrected model keeps the first failure record beside the later success record, and it reads both.

A record you can actually diff

The artifact below is a proposed workflow, and it is not a benchmark executed for this article. It writes a small record beside the test log so a reviewer can see where the green line came from. You can run it on a laptop or inside a free server session, then compare the two records. If those records diverge, the passing result is not yet a fact that both sides share.

#!/usr/bin/env bash
# Proposed, unexecuted example: write an environment record beside a test run.
set -euo pipefail

record_dir="${1:-.env-records}"
mkdir -p "$record_dir"
stamp="$(date -u +%Y%m%dT%H%M%SZ)"
out="$record_dir/run-$stamp.txt"

{
  echo "utc=$stamp"
  echo "host=$(hostname)"
  echo "uname=$(uname -srm)"
  echo "pwd=$(pwd)"
  if command -v git >/dev/null 2>&1 && git rev-parse --is-inside-work-tree >/dev/null 2>&1; then
    echo "git=$(git rev-parse HEAD)"
    echo "dirty=$(git status --porcelain | wc -l | tr -d ' ')"
  else
    echo "git=absent"
  fi
  if [ -f package-lock.json ]; then
    if command -v sha256sum >/dev/null 2>&1; then
      echo "lock=$(sha256sum package-lock.json | awk '{print $1}')"
    else
      echo "lock=$(shasum -a 256 package-lock.json | awk '{print $1}')"
    fi
  elif [ -f requirements.txt ]; then
    echo "reqs=present"
  else
    echo "deps=unpinned"
  fi
  echo "python=$(python3 --version 2>/dev/null || echo missing)"
  echo "node=$(node --version 2>/dev/null || echo missing)"
} > "$out"

echo "record=$out"
python3 -m unittest discover -s tests -v | tee -a "$out"

Start by running the script in the directory the model edited, and keep the record in the review thread. Repeat the run on the free server only after the same revision is checked out, including claimed uncommitted files. Compare host, kernel, git, and dependency lines before you read the test tail as evidence. A mismatch is a reason to stop the comparison, not a reason to average two green results.

Suppose the remote run fails once on a missing module and then passes after an unplanned package install. The second record should then show a dependency line that the first record did not contain. If you paste only the second log, the reviewer inherits a command that fails on a clean checkout. The comparison is the debugging step, and the install belongs in the repository rather than only in server history.

# Proposed comparison after you rename the two records to stable filenames.
diff -u .env-records/run-local.txt .env-records/run-remote.txt || true

If git prints absent, the server session is not attached to the repository the reviewer can open. If deps prints unpinned, a later install can fetch a different tree and still show the same command. If dirty is not zero, the passing run may include files the pull request does not contain. Each of those lines corrects the myth that a green status string is already a self-explanatory record.

# Proposed, unexecuted helper: reject a log that never names its environment.
REQUIRED = ("utc=", "uname=", "git=", "pwd=")

def record_is_explainable(text: str) -> bool:
    lines = text.splitlines()
    return all(any(line.startswith(key) for line in lines) for key in REQUIRED)

The helper above is proposal code for a review check, and it has not been executed in this draft. It only asks whether identity lines exist, which is a weaker bar than trusting the values they carry. A record can contain an uname line and still describe a repository the pull request does not build. Use the helper to reject silent logs, and do not use it to declare that a release is ready.

Limits, and who should skip it

This workflow is a notebook rather than an attestation system, and it will not satisfy a signed-image release process. Do not place production secrets, customer data, or private keys on a free shared server for a convenient green run. The script trusts local commands such as git and sha256sum, so a compromised session can lie in its own record. Teams that already require provenance should stay with that build system, and a one-file sketch can skip the ceremony.

The digest command is not universal, and a macOS laptop may need a different flag before the lock line matches a Linux server. The date command can also reject the format flag, which makes the stamp line fail before any test starts. Those gaps are reasons to treat the script as a starting proposal, not as a finished portable tool. This draft does not report timings, pass rates, or token consumption, because the example was not executed.

Recent developer discussions keep separating a stronger model reply from a stronger software record, and that split fits here. A cleaner explanation from a free model can still leave the patch tied to one machine nobody else can name. The free server option is useful when you need a second shell to reproduce a failure away from a laptop. It becomes misleading when that second shell is allowed to replace the environment record itself.

If MonkeyCode currently offers you a free model session and a free server option, try the fingerprint habit on a small change first. Read the current product docs for limits before you reserve an afternoon, because this article does not freeze those limits. A useful session ends with two records and a diff, not with a claim that the suite is already done.

The mental model that survives this FAQ is modest, and it is meant to travel with the patch. A free draft and a free shell can shorten the path to a first visible failure, which is often the useful part. They cannot sign the machine, pin the dependency tree, or replace the reviewer who owns the failing case. Keep the green line, and keep the short record that tells the next reader where that line was produced.

A Smoke Curl Is Not a Closed Loop: A Myth-Busting FAQ

Sam Yang — Sun, 11 Oct 2026 21:12:57 +0000

A small platform team cloned a coding-agent repository on Thursday and started the host process before lunch. The health route returned a short success body, and a model call through the free access path also returned text. The channel treated those two green signals as a release note and moved the change toward review. Three days later the same sequence failed in a pull-request check because a tool field had been renamed.

The port still answered, which made the failure look like a flaky test rather than a broken tool trace. That confusion is common when a smoke curl is asked to stand in for a closed workflow loop. A curl proves that some process accepted bytes, much as a doorbell proves that a building has electricity. It does not prove who opened the door, which room was reached, or whether the same visitor would be recognized tomorrow.

MonkeyCode is relevant here only as one place where a free server option and free model access can host that early smoke path. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Only two product facts are used here: a free server option and free model access, with no model name or quota attached. Readers should confirm both facts in the current project docs, because access limits can change and this article does not freeze them.

Recent public discussion has noted that model quality can rise while everyday software practice stays loose. That observation fits this failure, because a better reply does not tighten an unpinned host, an unpinned schema, or an unpinned tree. The practical question is which claim each separate probe is allowed to support in a review. A useful workflow separates reachability, execution, and replay before anyone writes a summary sentence for the channel.

Myth: a listening port means the loop is closed

Developers repeat this claim because local demos reward the first successful status code more than the second differing one. The corrected model treats reachability as a gate, not as a verdict about tools, files, or repeated behavior. Record the process command, the bound port, and the status code, then stop the claim at that boundary. Anything past the boundary belongs to a different probe with its own fixture and its own failure name.

On Thursday the health body was enough to satisfy the channel, and that social fact is not technical evidence. A process can listen while its tool registry still points at yesterday's field names and yesterday's paths. The analogy is a shop with the lights on and the cash drawer disconnected from the inventory system. Customers can enter the shop, and the inventory books can still be wrong at the close of day.

Myth: a model paragraph means the tool actually ran

A fluent reply can describe a file change without the host ever having executed the described tool call. That gap resembles a meeting note that summarizes a decision nobody entered into the issue tracker. Evidence for execution is a structured tool result, a schema version, and a diff against the fixture repository. If any of those three artifacts is missing, the paragraph remains narration rather than an integration result.

Free model access does not change that standard, because availability of a model path is not identity of a schema. The same caution applies to the free server option, which can serve the process without freezing the registry inside it. Ask the run for a machine-readable tool result, and keep the prose reply in a separate field. Reviewers can then reject narration that lacks the structured sibling and the schema version it claims to match.

Myth: sending the same prompt again counts as a replay

A second prompt can hit a different tool schema, a different working tree, or a different unspoken default. Replay requires the same request body, the same repository revision, and the same recorded tool responses. Without those pins, the second run is a new experiment that happens to share some wording with the first. Store the pins beside the output so a later reader can see what was held constant and what was not.

Retry culture makes this myth sticky, because a green second attempt feels like confirmation rather than a new sample. Confirmation needs a recorded first sample and a diff of the pins, not a similar sentence in the terminal. If the schema version changed between attempts, label the second attempt as a migration check instead. If the revision changed, label it as a new build, even when the prompt text matches.

A corrected mental model

Think of the free server as a shared workbench and the free model path as a shared measuring tape. Both can be useful on the first afternoon, and neither becomes a calibrated instrument by being available. Calibration is the evidence file you write: revision, schema version, probe name, and a named outcome. When that file is absent, a green curl should stay in the lab notebook as a reachability note only.

The closed loop is the path from pinned tree, through a named tool call, to a stored result that a second machine can classify. Reachability opens the bench, the execution probe checks the measuring marks, and replay checks whether the marks still agree. Collapsing those three checks into one status code is how Thursday's note became Monday's review surprise. Naming the three checks separately is slower by a few minutes and cheaper than a reverted review thread.

An unexecuted classifier you can adapt

The following checker is a proposal, not a captured run from this account, and it invents no product metrics. It reads a local JSON fixture that you produce, then classifies each probe before any human summary is written. Adapt the field names to your host, and do not treat the sample paths as documented product endpoints. The script fails closed when a required pin is missing, which is the behavior a smoke curl usually lacks.

#!/usr/bin/env python3
"""Proposal only: classify a local evidence fixture. Not a recorded product run."""

import json
import sys

REQUIRED = ("git_revision", "schema_version", "probe", "outcome")
PROBES = {
    "reachability": {"allowed_outcomes": {"listening", "refused", "timeout"}},
    "execution": {"allowed_outcomes": {"executed_match", "executed_mismatch", "tool_missing"}},
    "replay": {"allowed_outcomes": {"pins_match", "pins_drift", "incomplete"}},
}

def classify(row: dict) -> dict:
    missing = [key for key in REQUIRED if not row.get(key)]
    probe = row.get("probe")
    outcome = row.get("outcome")
    if missing or probe not in PROBES:
        return {"verdict": "incomplete", "missing": missing or ["probe"], "claim_scope": "none"}
    allowed = PROBES[probe]["allowed_outcomes"]
    if outcome not in allowed:
        return {"verdict": "invalid_outcome", "probe": probe, "claim_scope": "none"}
    scope = {
        "reachability": "port_only",
        "execution": "tool_execution_only",
        "replay": "pinned_comparison_only",
    }[probe]
    passed = outcome in {"listening", "executed_match", "pins_match"}
    return {"verdict": "pass" if passed else "fail", "probe": probe, "claim_scope": scope}

def main() -> int:
    document = json.load(sys.stdin)
    rows = document.get("probes", [])
    report = [classify(row) for row in rows]
    scopes = {item["claim_scope"] for item in report if item["verdict"] == "pass"}
    needed = {"port_only", "tool_execution_only", "pinned_comparison_only"}
    closed = needed <= scopes
    json.dump({"probes": report, "closed_loop": closed}, sys.stdout, indent=2)
    sys.stdout.write("\n")
    return 0 if report and all(item["verdict"] == "pass" for item in report) else 1

if __name__ == "__main__":
    raise SystemExit(main())

A reader can generate the surrounding pins with ordinary commands, then fill the fixture by hand from observed responses. The commands below record identity of the tree and the clock, not the performance of any host or model. They are safe to run in a disposable clone, and they still do not prove a tool call happened. Paste their output into the fixture fields rather than into a chat summary that will be forgotten.

git rev-parse HEAD
git status --porcelain=v1
python3 --version
date -u +%Y-%m-%dT%H:%M:%SZ

{
  "git_revision": "replace-with-git-rev-parse",
  "schema_version": "replace-with-tool-schema-id",
  "probes": [
    {
      "git_revision": "replace-with-git-rev-parse",
      "schema_version": "replace-with-tool-schema-id",
      "probe": "reachability",
      "outcome": "listening"
    },
    {
      "git_revision": "replace-with-git-rev-parse",
      "schema_version": "replace-with-tool-schema-id",
      "probe": "execution",
      "outcome": "executed_mismatch"
    },
    {
      "git_revision": "replace-with-git-rev-parse",
      "schema_version": "replace-with-tool-schema-id",
      "probe": "replay",
      "outcome": "incomplete"
    }
  ]
}

python3 classify_probes.py < evidence.json

Read the machine-readable report before you accept a prose recap of that same afternoon in the team channel. A pass on reachability with an executed_mismatch is the Thursday shape: the door opened, and the inventory field did not match. An incomplete replay means you may not claim that the second attempt confirmed the first recorded sample. The closed_loop flag becomes true only when all three scopes have passed, which the sample fixture above should refuse.

What the Thursday notes should have contained

The useful note would have said the port listened, the schema id was unread, and no replay pins were stored. That sentence is less exciting than a release claim, and it is the one a later reviewer can trust. Add the git revision from the disposable branch and the tool schema identifier from the host response. Leave model marketing names out of the note unless the current docs and the response metadata agree.

If the execution probe returns executed_mismatch, stop and diff the fixture schema against the live registry. Do not paper over that diff with a second prompt that asks the model to explain the error. An explanation can be helpful after the mismatch is recorded, and it is not a substitute for the mismatch record. The corrected habit is to file the structured failure first and to write the narrative only after that file exists.

Limitations and who should skip this

This approach does not load-test the host, price a loop, or prove that a warm process kept state stable. It also does not verify authentication, secret redaction, or multi-tenant isolation on a shared free server. Teams that need those guarantees should use their own audited environment and their own written acceptance tests. People debugging a one-off prompt with no repository fixture will get little value from the extra pins.

Skip this workflow if you cannot record a repository revision, because the classifier will refuse to pass. Skip it if your question is capacity or cost, since those questions need different instruments and different fixtures. Skip it if you need a vendor benchmark, because this article records no timings and names no model. The sample outcomes are labels for a local file, not measurements collected from any public or shared run.

If the free server option is already available to you, point the three probes at a disposable fixture branch before you announce a result. Read the current docs for access limits before you quote any allowance in a design review. That habit keeps a Thursday smoke signal from turning into an accidental release claim on the following Monday.

Same Command, Different Locale: An Environment FAQ

Sam Yang — Sat, 10 Oct 2026 17:42:50 +0000

A developer cited a free execution host because a date assertion returned zero just after midnight. The pull request quoted the command string, and that command string matched the continuous integration line exactly. A teammate ran that same line on a laptop and watched strftime emit a month token the assertion did not expect. The host had not lied about the exit code, yet nobody had recorded the locale in which the string was born.

That failure is easy to misfile as a model mistake, because automated help drafted the patch and the host was only a convenience. Free model access can cheapen the draft, and a free server option can cheapen the first execution. Neither convenience records the room in which the command actually ran for the reviewer who arrives later. This FAQ separates five repeated review claims from the smaller set of facts a fingerprint can honestly support.

Disclosure: This article was prepared as part of MonkeyCode's product outreach. In the workflow below, MonkeyCode is only the place where a free model call and a free server run can sit side by side. The fingerprint, the hashes, and the decision to cite a result stay in the repository you already control. If those two availability options are absent on your account, the same fingerprint method still applies to any host you already use.

Claims that outlive the panel

Review threads still treat a copied command as if it carried its environment along with the text. They treat locale, timezone, and umask as decorative settings that serious tests are expected to ignore. They assume the free host shell is the same shell the pipeline will invoke on the next build. They also assume that naming the model, without naming LC_TIME, would have explained a mismatched month token.

A fifth claim is quieter and just as costly for the person who has to replay the branch. People expect the agent to notice locale drift on its own because a free model was available to read the failure. Availability of a draft is not attention, and attention is not a recorded measurement you can attach to a commit. Taken together, these claims confuse a verb you can copy with a room you forgot to describe to the next reviewer.

The room around the verb

A command line is a verb, and the process environment is the room where that verb is spoken. Two rooms can hear the same verb and still produce different strings, different file modes, and different calendar labels. You can learn from the host run, but you cannot cite it until the room is written down beside the verb. The corrected model is that equality of command text is a hypothesis, not a result the review can close on.

The midnight mismatch came from LC_TIME, not from a mysterious change in the assertion logic itself. One room used the C locale and emitted Oct for October, while another room used a regional locale and emitted a different abbreviation. Umask can create a cousin of the same bug when a later test checks file modes the process just created. Timezone can shift a calendar date across midnight even when both clocks are honest and synchronized.

Locale is part of the contract whenever a test formats dates, sorts text, or parses numbers that contain grouping characters. Calling those settings cosmetic is how a green host becomes a red laptop without any edit to the product code. Pin the locale inside the test when the formatted string is part of the contract you intend to keep. Record that same locale beside the host evidence, or the pin remains a wish on the machine that happened to pass.

The illustration below is unexecuted, and it shows an assertion that secretly depends on the LC_TIME category. It is not a measured result, and it is not evidence about any particular host image, quota, or piece of hardware. Use it to see why a zero on one machine cannot travel alone into a review comment. Adapt the expected token only after you decide which locale the contract actually promises to callers.

# Unexecuted illustration: this assertion encodes LC_TIME.
# A pass on one host is not a pass on every clone.
# setlocale raises if the named locale is not installed. Do not catch that and pass.
import locale
from datetime import datetime, timezone

def format_stamp(moment: datetime) -> str:
    return moment.strftime("%b %d, %Y")

def test_stamp_under_c_locale() -> None:
    locale.setlocale(locale.LC_TIME, "C")
    moment = datetime(2026, 10, 10, tzinfo=timezone.utc)
    assert format_stamp(moment) == "Oct 10, 2026"

If the contract truly wants Oct, the test should set LC_TIME itself and the receipt should show that the setting happened. If the contract wants a locale-independent numeric date, change the format string and discard the host pass until the new assertion is replayed. Free model access can propose either repair, but the proposal stays a draft until the fingerprint and the local replay agree. A model that suggests strftime without mentioning locale has not finished the diagnosis you still need.

What the receipt may decide

The fingerprint script below is an unexecuted proposal for the receipt, not a benchmark and not a hardware description. It writes an allowlist on purpose, because a full environment dump often contains tokens you must not commit. Run it on the host and again on the clone you will cite, then store both files with their digests. If a field is unset, write unset rather than inventing a value the machine never actually had.

# Unexecuted proposal: allowlisted fingerprint. Do not record full env output.
set -eu
receipt_dir="reviews/fingerprints/$(date -u +%Y%m%dT%H%M%SZ)"
mkdir -p "$receipt_dir"
line() { printf '%s=%s\n' "$1" "$2"; }
{
  line utc_now "$(date -u +%Y-%m-%dT%H:%M:%SZ)"
  line timezone "${TZ:-unset}"
  line lang "${LANG:-unset}"
  line lc_all "${LC_ALL:-unset}"
  line lc_time "${LC_TIME:-unset}"
  line umask "$(umask)"
  line shell "${SHELL:-unset}"
  if command -v bash >/dev/null 2>&1; then
    line bash "$(bash --version | head -n 1)"
  else
    line bash missing
  fi
  line uname "$(uname -srm)"
  if command -v git >/dev/null 2>&1; then line git "$(git --version)"; else line git missing; fi
  if command -v python3 >/dev/null 2>&1; then line python3 "$(python3 --version 2>&1)"; else line python3 missing; fi
  if command -v node >/dev/null 2>&1; then line node "$(node --version)"; else line node missing; fi
} > "$receipt_dir/fingerprint.txt"
if command -v shasum >/dev/null 2>&1; then
  shasum -a 256 "$receipt_dir/fingerprint.txt" > "$receipt_dir/fingerprint.sha256"
else
  sha256sum "$receipt_dir/fingerprint.txt" > "$receipt_dir/fingerprint.sha256"
fi
echo "wrote $receipt_dir"

On systems that lack shasum, the script falls back to sha256sum, whose digest line format is not identical. MonkeyCode can hold the model turn and the server turn together when those free options are enabled for the account. The session still does not archive itself, so copy the fingerprint and its digest onto the branch before review. A panel that you cannot reopen tomorrow is a memory, and a memory is not a locale you can diff.

Recording a model label would not have repaired the midnight mismatch by itself, because the token came from strftime. The label can be useful context for a later reader, yet the mismatched token was produced under a locale rather than by that name. The corrected model keeps the assistant in the provenance note and keeps the locale fields in the fingerprint file. If you observed a label, write it as an untrusted string, and do not infer quotas, hardware, or lasting availability from it.

The comparison below is another unexecuted proposal, limited to fields that often change visible strings or file modes. A mismatch on language, LC_ALL, LC_TIME, timezone, or umask means the host exit code cannot be cited for a locale-sensitive test. A mismatch limited to the kernel string or a toolchain version should be noted, then judged against the dependency the test actually uses. No mismatch at all still fails to prove network or filesystem identity, so the local replay remains necessary before you merge.

# Unexecuted proposal: compare allowlisted fields. Exit 3 on a cited-field mismatch.
set -eu
host_fp="${1:?host fingerprint}"
local_fp="${2:?local fingerprint}"
status=0
for key in lang lc_all lc_time timezone umask; do
  h=$(awk -F= -v k="$key" '$1==k {print substr($0, index($0,"=")+1)}' "$host_fp")
  l=$(awk -F= -v k="$key" '$1==k {print substr($0, index($0,"=")+1)}' "$local_fp")
  if [ "$h" != "$l" ]; then
    echo "mismatch:$key host=${h:-missing} local=${l:-missing}" >&2
    status=3
  fi
done
if [ "$status" -eq 0 ]; then
  echo "allowlisted fields match"
fi
exit "$status"

The comparison script exits with status three when a cited field differs, and with status zero when those fields match. An absent key prints empty, so a truncated file can look clean unless you also require every key to be present. It also ignores extra keys, which is intentional, because an allowlist that grows without review becomes another place to leak secrets. Treat a missing fingerprint file as a failed comparison, not as a quiet pass that the host can hide behind.

Who should skip the method

Do not run this workflow on a free shared host with production data, customer uploads, or tests that print secrets. Do not treat the fingerprint as a software bill of materials, a security assessment, or a performance measurement of any kind. Do not use it when you cannot copy files off the host, because a screenshot of locale settings is still just a screenshot. This article records no timings, no hardware sizes, and no quota figures, so none of those should be inferred from the method.

The method is also a poor fit for teams that need a sealed multi-tenant attestation rather than a review note. A fingerprint describes what a process could see, and it does not prove that other tenants were absent from the machine. If your release bar requires that stronger isolation, use the environment your security review already approved, and keep this FAQ out of that path. People who only need a quick visual check, and who will not replay the command, will not get value from the extra files.

Hold the command, the room, and the citation apart the way a lab holds a procedure apart from the bench notes. The model layer may propose a locale pin more cheaply when free model access is available to the account. The host layer may run the fingerprint and the test when a free server option removes the need for a local install. Only the committed fingerprint, the command text, the exit code, and a local replay are allowed to support the citation.

When a teammate asks whether the date test passed, answer with the fingerprint path and the replay command. If the replay fails, the host was a hint about one room, and the clone remains the authority you should cite. If a free model session and a free server are already available in your MonkeyCode account, capture one fingerprint beside the next date assertion. Compare that file with a clean clone before you cite the host in the pull request.

A Free Session Is Not a Sealed Run: A Myth-Busting FAQ

Sam Yang — Thu, 08 Oct 2026 09:41:27 +0000

On a Wednesday stand-up, two engineers tried to decide which prompt draft deserved to stay in the repository. The Monday transcript came from a free hosted workspace, and the Wednesday transcript reused that prompt after a quiet model rotation. Neither file stored a model identifier, a lockfile digest, or the server image label that the host had printed. Without those fields, the passing suites could not show whether the prompt changed the outcome or the host changed the treatment.

That scene is common once a hosted agent starts to feel as convenient as a local terminal. A free session removes setup friction, yet it does not freeze the treatment you think you measured. The corrected picture is closer to a shared kitchen than to a sealed laboratory bench with labeled jars. You can cook a meal there, but you should not publish a recipe trial unless you photographed the ingredients.

Is an available host already a controlled environment?

Developers often repeat the claim that an available host is already a controlled environment for agent work. The claim sounds practical because the workspace boots, the tests run, and the diff appears in the same browser tab. Availability only records that a provider accepted the job during that particular hour of the week. It does not tell you which model build answered, which base image mounted, or which package index the installer hit.

A second familiar claim says that two free runs with the same prompt already count as an ablation. An ablation needs one changed factor and a written record of the factors you held still. A free catalog can rotate aliases, context windows, or safety filters without changing the prompt text you pasted. If the card is blank, the second passing result is a new anecdote, not a contrast you can defend.

A third claim treats the hosted workspace path as a private laboratory notebook that will still exist next week. Shared workspaces are built for turnover, so disks, caches, and temporary credentials can disappear between ordinary sessions. A notebook that evaporates cannot support a later dispute about which files the agent actually saw. The mental model that survives is plain: a free session stays a convenience sample until an environment card says otherwise.

What should the card record before anyone argues?

An environment card is a small file you commit beside the transcript, rather than a badge on a dashboard. It should name the git revision, the dirty-tree flag, the lockfile digest, the UTC timestamp, and the model identifier from that session. It should also name the server label the provider printed, even when that label is only a short alias. Proposed cards are not measurements of quality, and this draft does not report any timing or accuracy numbers.

You can sketch the capture with a shell fragment that you run locally before the agent starts editing. The fragment below is a proposal, and it has not been executed as a benchmark in this article. Replace every placeholder with the identifier the live session returns, and never hard-code a catalog name from memory. If a required lockfile is missing, leave the digest as none rather than inventing a hash the tree cannot support.

#!/usr/bin/env bash
set -euo pipefail

# Proposed helper. Not executed as a benchmark in this article.
# Usage:
#   MODEL_ID="$(cat session_model.txt)" \
#   SERVER_LABEL="$(cat session_server.txt)" \
#   ./envcard.sh

: "${MODEL_ID:?set MODEL_ID from the live session, not from memory}"
: "${SERVER_LABEL:?set SERVER_LABEL from the live session, not from memory}"

rev="$(git rev-parse HEAD)"
dirty="clean"
if ! git diff --quiet || ! git diff --cached --quiet; then
  dirty="dirty"
fi

lock="$(python3 - <<'PY'
import hashlib
import pathlib
import sys
for name in ("package-lock.json", "uv.lock", "Cargo.lock"):
    path = pathlib.Path(name)
    if path.is_file():
        print(hashlib.sha256(path.read_bytes()).hexdigest())
        sys.exit(0)
print("none")
PY
)"

mkdir -p .experiment
stamp="$(date -u +%Y-%m-%dT%H:%M:%SZ)"
out=".experiment/envcard-${stamp}.json"
cat > "${out}" <<EOF
{
  "git_rev": "${rev}",
  "worktree": "${dirty}",
  "lock_sha256": "${lock}",
  "model_id": "${MODEL_ID}",
  "server_label": "${SERVER_LABEL}",
  "recorded_at": "${stamp}",
  "note": "convenience sample until cards match"
}
EOF
printf 'wrote %s\n' "${out}"

The helper assumes it is already inside a git checkout with Python 3 available on the path. It hashes the first lockfile it finds among common JavaScript, Python, and Rust names, and it writes none when all are absent. It does not contact a model endpoint, and it will not invent a server label if you forgot to export one. Run it once per session, then store the JSON next to the transcript instead of pasting secrets into the card.

A second fragment can refuse the comparison when treatment fields on two local cards differ from each other. The checker below is also a proposal, and it never calls a network or claims a provider result. It only reads two JSON cards from disk and exits non-zero when any sealed field diverges. Save it as compare_cards.py beside the shell helper, and pass the two card paths as ordinary arguments.

#!/usr/bin/env python3
"""Proposed card compare. Unexecuted as a study. No network calls."""
import json
import sys

FIELDS = ("git_rev", "worktree", "lock_sha256", "model_id", "server_label")

def load(path):
    with open(path, encoding="utf-8") as handle:
        return json.load(handle)

def main():
    if len(sys.argv) != 3:
        print("usage: compare_cards.py LEFT.json RIGHT.json")
        return 1
    left, right = load(sys.argv[1]), load(sys.argv[2])
    mismatches = [key for key in FIELDS if left.get(key) != right.get(key)]
    if mismatches:
        print("not a sealed contrast:", ", ".join(mismatches))
        return 2
    print("treatment fields match; still not a published benchmark")
    return 0

if __name__ == "__main__":
    sys.exit(main())

How should a reviewer read two cards?

Read the two commands as a gate in the review path, rather than as a numeric score of quality. The shell helper writes the fields the session declared, and the Python checker tells you whether a contrast is eligible. A match still does not prove that hidden weights stayed put, because an alias can point at a newer build. A mismatch is already enough to stop the victory lap and send the run back to the demo pile.

A matching card also does not prove that the prompt itself caused the new diff in the tree. Causation still needs a pre-declared edit and a check that no other intended file changed underneath the agent. The checker ignores timestamps on purpose, since two honest runs should not be punished for happening on different days. What it refuses is a silent swap of revision, lockfile, model identifier, server label, or dirty worktree state.

The invocation is deliberately boring, which is the point of a gate you can rerun next month. Write the live identifiers to temporary files, export them for the helper, and only then start the prompt edit. Point the checker at the two JSON cards, and stop if it prints the names of fields that diverged. Delete the temporary identifier files when the review ends, because a card should not become a secret store.

# Proposed invocation. Do not paste real secrets into these files.
printf '%s\n' "$LIVE_MODEL_ID" > session_model.txt
printf '%s\n' "$LIVE_SERVER_LABEL" > session_server.txt
MODEL_ID="$(cat session_model.txt)" SERVER_LABEL="$(cat session_server.txt)" ./envcard.sh
python3 compare_cards.py .experiment/envcard-left.json .experiment/envcard-right.json

Where does a free hosted option fit the gate?

MonkeyCode, as the operator described it for this draft, is an open-source coding-agent project with free model access and a free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Those two availability claims help when a short workspace is enough and local setup is the real blocker. They are not a quota, a hardware specification, or a promise that the same build remains available next week. This article names no models, token ceilings, or machine sizes, because those details were not verified against a primary page. Read the current product page on the day you run, and copy only identifiers that the live session actually returns.

Use the free server as the printer for identifiers your environment card requires before any prompt edit. Start the session, copy the model string and server label into the environment, and run the capture script before editing. If the provider does not reveal a stable identifier, keep the run as a demo and do not pair it with an older transcript. The same rule applies when the free option is temporarily down: a missing host is a scheduling fact, not a quality result.

That workflow stays honest in contribution season, when a passing agent diff can look like a finished review. A reviewer can ask for the card beside the patch and reject contrasts whose model or lockfile fields moved. The hosted convenience remains, but the claim shrinks to the narrow statement the card can actually support. Contribution season rewards visible patches, yet a visible patch still inherits every unrecorded change in the host.

Who should leave this gate unused?

Skip the free-session comparison if you need numbers for a paper, a vendor scorecard, or a launch post. A convenience sample cannot carry those claims, even when both cards match, because this method does not pin hidden weights or network conditions. Skip the method also when policy forbids shared hosts, customer code, or secrets outside a controlled network. A capture script does not make a shared disk compliant with a policy that forbids outside hosts.

Skip the method when you cannot read the current product page on the same day you run the session. Availability language in a blog post ages faster than a lockfile, and an outdated image name will poison the card. Teams that already own a frozen image and a pinned model build gain no rigor by moving that trial onto a free host. They would trade a sealed bench for a shared kitchen where other jobs can change the counters overnight.

The practical close stays modest, because a convenient host is not a reason to widen the claim. If your reviews already demand an environment card, add a free hosted session only when the live identifiers fit that card without guesswork. Otherwise keep the free run as a demonstration, and let the sealed comparison wait for a workspace that can hold still. That restraint is the whole method: record the treatment, compare only matches, and leave unpublished numbers unpublished.