On a Tuesday, a developer on a small platform team pulled a clean patch out of a free hosted coding tier and used it to open a demo PR. By Thursday, the same task against a self-hosted clone of the same open-source project produced a patch that imported a package nobody could install. The model had not changed, the prompt had not changed, and yet the result had; the seam between a repository and a running service had simply moved. That seam is what this FAQ is about, because most disagreements about free AI coding tiers are really disagreements about which seam people are standing on.
The numbers I ask you to collect below are yours, not mine. Where a figure appears in this article it is illustrative and labelled as such, and the probe is built so that your own ledger can contradict me within an hour. That is the only kind of evidence worth publishing about availability claims.
A repository is a recipe, not a kitchen
A repository tells you how the author cooked; a service tells you what the kitchen looks like at 2 a.m. on a Wednesday. When you clone an open-source coding agent you inherit the recipe, the tool definitions, and the tests, but you do not inherit the model endpoint, the sandbox image, the queue, or the meter. Those four things are the service, they are rarely in the README, and they are exactly what your CI will collide with.
The distinction is not pedantry, it is a budget line. A patch that arrives in four seconds from a hosted tier and ninety seconds from your laptop changes how many iterations a developer attempts before lunch. Same code, same task, completely different loop.
MonkeyCode is an open-source coding agent project that, as of this writing, advertises free model access and a free server option, with a free token allowance the operator states as ten million tokens. Disclosure: This article was prepared as part of MonkeyCode's product outreach. Those availability claims are operator-supplied and time-sensitive, so treat them as hypotheses to test in your first ten minutes rather than as constants you plan capacity against.
What makes a project like that a useful test subject is not the price, it is that the free path and the self-host path sit side by side. That layout forces you to notice where they diverge instead of discovering it during a sprint review.
The probe: three trials, one ledger
Before arguing about patch quality, record what actually happens when your agent runs. The script below executes any command you hand it three times inside a throwaway clone, resets the working tree between trials, and appends one JSON line per trial with wall-clock time, exit code, touched files, and a hash of the produced patch. It is deliberately dumb: it does not know your provider's billing unit, it does not parse token usage, and it will record a timeout as a perfectly valid result.
#!/usr/bin/env python3
'''parity_probe.py - record what a coding agent did, trial by trial.
Run inside a throwaway clone. The probe resets the tree between trials,
so anything uncommitted will be lost.
'''
import hashlib
import json
import os
import subprocess
import time
from datetime import datetime, timezone
from pathlib import Path
WORK = Path(os.environ.get('WORK', './sandbox')).resolve()
CMD = os.environ['PROBE_CMD']
TRIALS = int(os.environ.get('TRIALS', '3'))
LEDGER = Path(os.environ.get('LEDGER', 'ledger.jsonl'))
TIMEOUT = int(os.environ.get('TIMEOUT', '900'))
def sh(cmd, timeout=120):
started = time.monotonic()
proc = subprocess.run(
cmd, cwd=WORK, shell=isinstance(cmd, str),
capture_output=True, text=True, timeout=timeout,
)
return proc, round(time.monotonic() - started, 2)
def snapshot():
_, head = sh('git rev-parse HEAD', 30)
_, status = sh('git status --porcelain', 30)
_, diff = sh('git diff --binary', 60)
return {
'head': head.stdout.strip()[:12],
'files': sorted({line[3:] for line in status.stdout.splitlines()}),
'patch_sha': hashlib.sha256(diff.stdout.encode()).hexdigest()[:12],
}
def reset():
sh('git checkout -- .', 120)
sh('git clean -fdq', 120)
def main():
LEDGER.write_text('')
for trial in range(1, TRIALS + 1):
reset()
try:
proc, seconds = sh(CMD, TIMEOUT)
exit_code, stderr = proc.returncode, proc.stderr[-400:]
except subprocess.TimeoutExpired:
seconds, exit_code, stderr = TIMEOUT, 'timeout', 'probe timeout'
record = {
'trial': trial,
'at': datetime.now(timezone.utc).isoformat(timespec='seconds'),
'seconds': seconds,
'exit': exit_code,
'stderr_tail': stderr,
**snapshot(),
}
with LEDGER.open('a') as fh:
fh.write(json.dumps(record) + '\n')
print(f'trial {trial}: exit={exit_code} {seconds}s '
f'files={len(record[chr(39)+chr(39)] if False else record["files"])}')
if __name__ == '__main__':
main()
Fix that print line to files={len(record['files'])} in your copy, then wire it to whatever command drives your agent. A typical session looks like this, and it costs you three runs of your own task rather than a benchmark you did not design:
python3 -m venv .venv && . .venv/bin/activate
git clone --depth 1 git@github.com:your-org/your-repo.git sandbox
cd sandbox && git checkout -b probe-base && git commit -am 'probe baseline'
cd ..
WORK=./sandbox TRIALS=3 TIMEOUT=900 \
PROBE_CMD='your-agent run --task tasks/add-validation.md' \
python3 parity_probe.py
Then read the ledger as a table instead of a vibe:
jq -r '[.trial, .seconds, .exit, .patch_sha, (.files|length)] | @tsv' ledger.jsonl
If the three patch hashes differ, you have measured variance on your own task. If they match but the file counts differ wildly, you have measured a different problem: the agent is reaching the same answer through unstable routes, and your reviewer is paying for that instability.
Four myths, corrected
Myth one: open source means the hosted tier and my clone are interchangeable
They are three separate contracts pretending to be one: the source you can read, the endpoint you are pointed at, and the runner that executes tool calls. Version drift in any of the three produces a patch that looks authored by a different engineer. The corrected model is to version all three explicitly, then record them in the same ledger entry the probe already writes.
Myth two: the free tokens are the free tier
The token allowance is the visible meter, and visible meters are excellent at hiding the invisible ones. On a shared free server the scarce resource is usually when you get to run and how often your job is retried, not whether the tokens exist. Think of free parking downtown: the space is free, the tow truck is not, and the tow truck is what your retro will discuss.
Myth three: the same model and prompt produce the same patch
Sampling settings, tool-schema versions, context assembly order, and even the order files appear in a directory listing all nudge the output. This is not a scandal, it is how probabilistic systems behave under a deterministic-looking interface. The corrected habit is to log the knobs alongside the diff, so that a surprise on Thursday can be traced to a changed default rather than to a colleague's imagination.
Myth four: one clean run proves the tier works for your team
A single green run is a sample of one, and samples of one are how teams end up with a demo that cannot survive a second developer. Three trials give you a range, a range gives you a review budget, and a review budget is the number that actually decides adoption. The measured wall clock matters less than the honest answer to whether anyone will read the diff.
Choosing where the loop runs
| Placement | You control | You inherit | Expect this failure | Skip it when |
|---|---|---|---|---|
| Self-hosted clone | endpoint, runner image, cache | upgrade churn, on-call, GPU or CPU bill | your config drifts from upstream within weeks | you cannot staff the maintenance |
| Hosted free tier | repo, task text, ledger | queue, capacity, tomorrow's defaults | a fast Tuesday and a slow Thursday | you need a contractual uptime figure |
| Local small model | everything, offline | RAM, heat, patience | plausible patch, wrong import | the task needs more context than your box holds |
The table is a decision aid, not a ranking; each row is somebody's correct answer on a different day. The free tier is the cheapest place to falsify the portability claim, which is precisely why the probe belongs there first.
Limitations and who should not bother
The probe does not measure your code review time, which is usually the largest cost in the loop and the hardest to attribute. Renamed files collapse into a single path in the porcelain parsing, so treat the file list as a hint rather than an inventory. Three trials are a range, not a distribution, and the probe itself consumes tokens and queue time while producing nothing shippable.
Skip this approach if your workload requires data residency guarantees you cannot verify, if you need an uptime commitment attached to the agent loop, or if your repository cannot survive a git clean -fd in a throwaway clone. It is also the wrong tool for anyone who will not read the ledger, because an unread ledger is just a slower way to guess.
If you want to test the portability claim rather than accept it, start where the setup cost is lowest and the record-keeping is identical. That is the honest role MonkeyCode's free model access and free server option can play in this experiment: they remove two steps the probe would otherwise have to document, finding an endpoint and finding a machine. The ledger is what stops you from mistaking a short walk to the first patch for a short project.
Top comments (1)
A hosted service stays portable only while it remains a thin layer over a repo you can point somewhere else. An index, a cache or a fine-tune that lives only in their account moves part of the service out of the repo, so an exit becomes a migration. You have to trust that everything the agent learns about your code stays in files you already hold. That trust is a promise from the vendor rather than a property of the design. Worth pinning down exactly which artefacts survive a move and which don't before treating the tier as free.