Cloneable Is Not Reproducible: Four Myths About Free AI Coding Tiers

At 9:40 on a Tuesday, two engineers cloned the same repository at the same commit, ran the same test command, and got different results. Nothing in the repository explained the difference, because the deciding variable lived in one laptop's shell profile and a cache directory the second machine had never populated. The pair had spent the previous week wiring an AI coding loop on free model access and a free server, and nobody had asked whether the loop was reproducible at all.

That scene is a composite illustration rather than a logged incident, and the scripts below exist so you can replace it with your own measurements. The distinction matters, because recent AI news has followed a familiar shape: model capability improves quickly while the surrounding practice of verifying and reproducing changes lags behind it. Free tiers widen the gap, since a convincing demo costs nothing and a reproducible workflow costs discipline.

MonkeyCode is an open-source project that advertises free model access and a free server option, and the operator-supplied brief for this article cites a free allowance of 10 million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Allowances, available models, and server limits change with time and load, so treat every figure in this paragraph as a claim to verify on the project page rather than as a constant you can design around.

The four myths below circulate widely in issue threads, code reviews, and internal documentation. Each one is cheap to believe and expensive to act on, and each one has a measurement that settles the argument.

Myth 1: If the repository clones, the environment reproduces

Cloning copies files, not the machine state that decides whether those files run. Dependency resolution, locale, timezone, toolchain patch versions, and untracked caches all change outcomes while leaving the git tree identical. This is why "works on my machine" survives every migration, and why a fresh server can fail a suite that passed locally minutes earlier. The corrected mental model is narrow and testable: reproducibility is a recorded set of inputs plus a deterministic entrypoint, and anything unrecorded is a coin flip.

The script below records those inputs without leaking secrets, because it stores environment variable names rather than their values. Note that a manifest is not a lock: it tells you when two machines differ, not how to make them identical, and secret values still have to arrive through your existing channel.

#!/usr/bin/env python3
"""env_manifest.py - record the inputs that decide whether a fresh clone behaves."""
import hashlib, json, os, platform, subprocess, sys, time


def shell(cmd: str, timeout: int = 20) -> str:
    try:
        done = subprocess.run(cmd, shell=True, capture_output=True,
                              text=True, timeout=timeout)
        return done.stdout.strip()
    except Exception as exc:
        return f"unavailable:{exc.__class__.__name__}"


def file_sha256(path: str) -> str:
    try:
        with open(path, "rb") as handle:
            return hashlib.sha256(handle.read()).hexdigest()[:16]
    except OSError:
        return "missing"


manifest = {
    "recorded_at_utc": time.strftime("%Y-%m-%dT%H:%M:%SZ", time.gmtime()),
    "git_rev": shell("git rev-parse HEAD"),
    "git_dirty": bool(shell("git status --porcelain")),
    "python": sys.version.split()[0],
    "platform": platform.platform(),
    "locale": os.environ.get("LANG", ""),
    "timezone": shell("date +%Z"),
    "lockfile_sha256": file_sha256("requirements.txt"),
    # Names only, never values: values leak credentials into build logs.
    "env_names": sorted(k for k in os.environ if k.startswith(("APP_", "CI_"))),
}
print(json.dumps(manifest, indent=2, sort_keys=True))

Run that on the laptop and on the free server, diff the two JSON files, and the conversation stops being about vibes. In practice the diff almost always shows a different Python patch version, a dirty working tree, or a missing lockfile hash. Two lines of output are usually enough to explain a week of "flaky" failures.

Myth 2: Free model access makes the model the bottleneck

The intuition here is reasonable and usually wrong. Teams assume the constraint is model latency or the token allowance, then discover that the real consumer is re-sent context, since every failed test round ships the entire conversation back to the endpoint. A loop that converges in two rounds and a loop that thrashes for nine can differ by an order of magnitude in tokens while producing the same single commit. The unit worth optimizing is therefore tokens per verified change, not tokens per request.

The ledger below is a template written against an OpenAI-compatible /v1/chat/completions endpoint; response field names vary between providers, so check the parsing against your own server before trusting the totals. It logs the exact model string the endpoint returns, because a stable model name in your config does not guarantee a stable model behind the endpoint.

#!/usr/bin/env python3
"""loop_ledger.py - log tokens against verified changes, not against requests."""
import json, os, subprocess, time, urllib.request

ENDPOINT = os.environ["LLM_ENDPOINT"].rstrip("/")
MODEL = os.environ["LLM_MODEL"]
GATE = os.environ.get("GATE_CMD", "pytest -q --maxfail=1")
LEDGER = "loop_ledger.jsonl"


def call(messages, **params):
    body = json.dumps({"model": MODEL, "messages": messages, **params}).encode()
    request = urllib.request.Request(
        f"{ENDPOINT}/chat/completions",
        data=body,
        headers={
            "Content-Type": "application/json",
            "Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
        },
    )
    with urllib.request.urlopen(request, timeout=180) as response:
        payload = json.load(response)
    usage = payload.get("usage") or {}
    # Assert instead of defaulting if silent zeroes would mislead your totals.
    if "prompt_tokens" not in usage:
        raise RuntimeError("endpoint returned no usage block")
    return payload["choices"][0]["message"]["content"], {
        "prompt_tokens": usage["prompt_tokens"],
        "completion_tokens": usage.get("completion_tokens", 0),
        "model_returned": payload.get("model", "unknown"),
    }


def gate():
    started = time.monotonic()
    done = subprocess.run(GATE, shell=True, capture_output=True, text=True)
    return done.returncode, round(time.monotonic() - started, 2)


def record(task, round_no, usage, exit_code, gate_seconds, wall_seconds):
    row = {"task": task, "round": round_no, "exit_code": exit_code,
           "gate_seconds": gate_seconds, "wall_seconds": wall_seconds, **usage}
    with open(LEDGER, "a", encoding="utf-8") as handle:
        handle.write(json.dumps(row) + "\n")

Two derived numbers carry most of the insight. Divide the summed prompt and completion tokens of rounds whose exit_code was zero by the count of tasks that reached green, and you get tokens per verified change. Divide the prompt tokens in a task's final round by the prompt tokens in its first round, and you get context amplification, which is the number that quietly eats a free allowance. Truncating or summarizing context attacks amplification directly, while a faster endpoint does nothing for it.

Myth 3: A free server is an environment

A running process is not an environment, however convenient the shell feels. Free servers tend to be ephemeral by design, so databases, caches, and installed system packages may disappear between sessions while your mental model still assumes they are yours. The rule that survives this is blunt: anything you cannot rebuild from the repository does not exist on the server, and anything that exists only on the server is a future incident. Snapshot discipline is what converts borrowed capacity into a usable runner.

#!/usr/bin/env bash
# bootstrap.sh - install, or rebuild from zero and prove the rebuild is clean.
set -euo pipefail

install() {
  python3 -m venv .venv
  . ./.venv/bin/activate
  pip install --require-hashes -r requirements.txt
}

verify() {
  git clean -xfd          # drop every untracked file, including caches
  install
  pytest -q --maxfail=1
}

case "${1:-install}" in
  install) install ;;
  verify)  verify ;;
  *) echo "usage: $0 [install|verify]" >&2; exit 2 ;;
 esac

Run verify in a scratch checkout, never where uncommitted experiments live, since git clean -xfd is deliberately destructive. A clean rebuild that passes is the only evidence that your environment lives in version control. The same command then works on a laptop, in CI, and on the free server, which is usually the moment a team stops debugging machines.

Myth 4: Faster tokens mean faster delivery

Token throughput is the least interesting number in the pipeline, because wall clock is dominated by test runtime, review latency, and queueing. A loop that emits a thousand tokens per second and needs nine correction rounds still lands behind one that emits slowly and converges once. The honest metric is elapsed time from task opened to change merged, measured on the same tasks over time rather than on a single lucky afternoon. Once you accept that framing, the choice between free and owned capacity becomes a question about blast radius.

Situation	Free model access plus free server	Owned or paid environment
Prompt and script prototyping	Fits: disposable work, low blast radius	Unnecessary
Generating tests for existing modules	Fits when a human reviews the assertions	Same, provided review exists
Mechanical refactors behind a strong gate	Fits: the gate is the safety net	Required when no gate exists
Latency-sensitive user paths	Does not fit: shared capacity, variable timing	Required
Regulated or customer-identifying data	Does not fit without a data-handling review	Required
Unattended multi-hour autonomy	Does not fit without an external stop condition	Required

The corrected mental model

Three ideas replace the four myths. Record your inputs, because a reproduction you cannot diff is a rumour rather than a fact. Measure tokens per verified change, because convergence is the variable that free allowances actually react to. Treat borrowed capacity as a runner you can throw away, because state that cannot be rebuilt from the repository is state you do not own.

None of this requires paid infrastructure, and all of it survives a change of provider. That property matters more than any single endpoint, since free availability is exactly the part of your stack that can change without notice. If your ledger and your manifest live in the repository, a migration costs an afternoon instead of a rewrite.

Limitations and who should skip this

This workflow adds bookkeeping, and small solo prototypes rarely earn it back. Teams handling regulated or customer-identifying data should resolve the data-handling question before optimizing anything else, and latency-bound user paths generally do not belong on shared free capacity at all. The ledger also only counts what the endpoint reports: if usage fields are missing, you will see zeroes and a falsely cheap loop, which is why the template raises instead of defaulting. Finally, skip the whole approach if you cannot commit to a verification gate, because every myth above assumes a test command stands between the model and main.

If you want to try the loop on a repository of your own, MonkeyCode's free model access and free server option are a low-cost place to run the first few hundred rounds. Clone the project, run bootstrap.sh verify, point loop_ledger.py at your endpoint, and let the ledger tell you whether the free tier is a bottleneck or a bargain.