The Free Allowance Is Not the Budget: Myth-Busting the Real Cost of AI Coding Loops

A reviewer once sent back a pull request with three different fixes for the same flaky parser, all of them green in CI. The agent had been free to run, so nobody had counted anything, and the repo quietly paid the difference in review time. That mismatch between what the meter shows and what the work costs is the subject of this article. It is also the myth most developers repeat without checking: that a free token allowance makes AI-assisted coding free.

The corrected mental model is narrower and more useful. A token allowance caps one input to one step, while real cost accumulates across loop iterations, rework, and environment drift that no dashboard displays by default. If you want to argue about tiers, you first need a harness that measures the loop rather than the price tag. What follows is that harness, five claims checked against it, and the situations where the whole approach is the wrong tool.

Start with a measurement, not an opinion

The harness runs the same task against the same pinned commit several times and records what actually happened, including failures, which are usually the interesting rows. It talks to any OpenAI-compatible endpoint, so the tier you are evaluating stays a configuration detail rather than an assumption baked into the script. Platforms that offer free model access, including MonkeyCode, just become another base URL you can point it at.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Create a scratch copy of the repository and export three variables before running anything. The model identifier is deliberately left as a placeholder, because the right value depends on what your provider currently serves and you should read that from their own documentation.

git clone <your-repo> sandbox && cd sandbox
export LLM_BASE_URL="https://<provider-host>/v1"   # any OpenAI-compatible endpoint
export LLM_API_KEY="<your-key>"
export LLM_MODEL="<model-id-you-are-evaluating>"

python3 run_matrix.py \
  --repo ./sandbox \
  --commit "$(git -C sandbox rev-parse HEAD)" \
  --prompt prompts/fix-flaky-parser.md \
  --test "pytest -q tests/test_parser.py" \
  --repeats 5

The script below resets the worktree, asks the model once, tries to apply the returned patch, and records the diff shape alongside the exit code of your test command. Keeping temperature at zero reduces one source of noise, though it does not eliminate provider-side nondeterminism.

#!/usr/bin/env python3
"""run_matrix.py: run one task N times and record what the loop really cost."""
import argparse, csv, json, os, re, statistics, subprocess, time, urllib.request

def ask(prompt: str):
    body = json.dumps({
        "model": os.environ["LLM_MODEL"],
        "messages": [{"role": "user", "content": prompt}],
        "temperature": 0,
    }).encode()
    req = urllib.request.Request(
        os.environ["LLM_BASE_URL"].rstrip("/") + "/chat/completions",
        data=body,
        headers={"Authorization": f"Bearer {os.environ['LLM_API_KEY']}",
                 "Content-Type": "application/json"},
    )
    started = time.monotonic()
    with urllib.request.urlopen(req, timeout=600) as resp:
        payload = json.load(resp)
    usage = payload.get("usage", {}) or {}
    usage["seconds"] = round(time.monotonic() - started, 2)
    return payload["choices"][0]["message"]["content"], usage

def apply_patch(repo: str, text: str) -> int:
    blocks = re.findall(r"```

(?:diff|patch)\n(.*?)

```", text, re.S)
    if not blocks:
        return -1  # the model answered with prose instead of a patch
    done = subprocess.run(["git", "-C", repo, "apply", "--whitespace=nowarn", "-"],
                          input=blocks[-1], text=True, capture_output=True)
    return 0 if done.returncode == 0 else -2

def main() -> None:
    ap = argparse.ArgumentParser()
    ap.add_argument("--repo", required=True)
    ap.add_argument("--commit", required=True)
    ap.add_argument("--prompt", required=True)
    ap.add_argument("--test", required=True)
    ap.add_argument("--repeats", type=int, default=3)
    ap.add_argument("--out", default="loop-cost.csv")
    args = ap.parse_args()

    prompt = open(args.prompt).read()
    rows = []
    for run in range(1, args.repeats + 1):
        subprocess.run(["git", "-C", args.repo, "checkout", "--force", args.commit], check=True)
        subprocess.run(["git", "-C", args.repo, "clean", "-fdx"], check=True)

        text, usage = ask(prompt)
        applied = apply_patch(args.repo, text)
        diff = subprocess.run(["git", "-C", args.repo, "diff", "--numstat"],
                              text=True, capture_output=True).stdout
        tests = subprocess.run(args.test, cwd=args.repo, shell=True, capture_output=True)

        rows.append({
            "run": run,
            "seconds": usage.get("seconds"),
            "prompt_tokens": usage.get("prompt_tokens"),
            "completion_tokens": usage.get("completion_tokens"),
            "apply_code": applied,
            "files_touched": len([line for line in diff.splitlines() if line.strip()]),
            "lines_changed": sum(int(n) for n in re.findall(r"^(\d+)\t", diff, re.M)),
            "test_exit": tests.returncode,
        })

    with open(args.out, "w", newline="") as fh:
        writer = csv.DictWriter(fh, fieldnames=list(rows[0]))
        writer.writeheader()
        writer.writerows(rows)
    print(open(args.out).read())

if __name__ == "__main__":
    main()

The CSV header alone tells you which questions become answerable: run,seconds,prompt_tokens,completion_tokens,apply_code,files_touched,lines_changed,test_exit. Treat the values you collect as yours and nobody else's, because they describe one repository, one prompt, and one test command. A quick summary keeps the discussion grounded in medians rather than a single lucky run.

python3 - <<'PY'
import csv, statistics as st
rows = list(csv.DictReader(open("loop-cost.csv")))
print("runs:", len(rows))
print("pass rate:", sum(r["test_exit"] == "0" for r in rows) / len(rows))
print("median seconds:", st.median(float(r["seconds"]) for r in rows))
print("median lines changed:", st.median(float(r["lines_changed"]) for r in rows))
PY

Five claims, checked against the harness

Claim one: a token allowance is a spending cap. An allowance bounds one dimension of one request, and agent loops multiply that dimension by the number of attempts plus every retry your wrapper adds. The corrected model is that tokens behave like a mileage limit on a single trip while your costs live in the number of trips. If you add automatic retries, log them, because two retries are three requests and the meter will not tell you that.

Claim two: cheaper tokens mean cheaper work. Loop count and human review dominate the bill long before per-token price does. A slow run that produces a two-line patch often costs less attention than a fast run that rewrites four files, even though both passed the test command. This is why the harness records files_touched and lines_changed next to timing, and why the pair is more informative than either number alone.

Claim three: one successful run proves the tier can do the task. A single green run is a coin flip with good marketing, and the interesting signal is variance across repeats. Five runs that pass with wildly different patch shapes tell you the task is underspecified, not that the model is unreliable. Fix the prompt before blaming the endpoint, since the harness resets the worktree and cannot reset an ambiguous instruction.

Claim four: if tests pass, the patch is done. Test outcomes are necessary evidence and never sufficient evidence, because your suite only measures what someone already thought to assert. The harness surfaces this cheaply: when apply_code is -1, the model answered with prose, which is a prompt problem, and when test_exit is zero on a large diff you should read the patch before celebrating.

Claim five: the run on a hosted server equals the run on my laptop. Environment identity is part of the experiment, and a container with a different toolchain version can change both the patch and the result. Record the kernel, language runtime, and lockfile hash with each batch, or your comparison measures two variables at once. Recent debates about whether agent wrappers amount to conditionals miss this point, since the interesting difference is rarely the wrapper.

Where the free options actually fit

A harness is only persuasive when you can afford repetitions, and repetitions are exactly what per-request pricing discourages during exploration. Free model access plus a free server option, as offered by MonkeyCode, is one way to run five or ten repeats of a throwaway task without turning a measurement into a purchase decision. Verify the current terms yourself, including what the allowance covers and how long a server persists, because capacity and conditions change and this article cannot promise them. The honest framing is that these options lower the cost of finding out, not the cost of shipping.

Situation	Free model access sufficient?	Free server sufficient?	Reason
One-off refactor on a side repo	Often yes	Usually unnecessary	Few iterations, low review stakes
Nightly regression harness	Often yes	Yes if jobs fit the session limits	Volume matters more than latency
Agent on every pull request	Depends on review policy	Only with artifact retention	A lost log makes a failed run unfalsifiable
Code under residency or contract rules	Usually no	Usually no	Sending the diff offsite may be prohibited
Multi-hour interactive session	Depends on interruption limits	Rarely	State loss mid-task defeats the exercise

Limitations, and who should skip this

The harness measures one task shape against one test command, so it is not a benchmark and should never be quoted as one. Token fields come from the provider's usage block, and endpoints that omit it will leave those columns empty rather than wrong. A five-run sample cannot separate model quality from prompt quality, and comparing runs across different days quietly mixes in version changes you did not record.

Skip this approach if your work is subject to data-residency or contractual constraints that forbid sending source code to a third-party endpoint, because no amount of measurement fixes a policy violation. Skip it as well if you need deterministic, bit-identical reproducibility for compliance evidence, since hosted inference does not offer that guarantee. If you are optimizing interactive latency rather than loop cost, a batch harness answers the wrong question entirely.

If you already have a repository and a test command, the cheapest next step is to run five repeats before arguing about tiers again, then read the median row rather than the best one.