Thursday, 4:40pm. A platform lead drops a screenshot in our channel: free tokens, a free server option, and one line of text — "why are we paying for anything?"
Fair question. Also the wrong unit. We were never paying for tokens. We were paying for the parts tokens do not include: the retry that ate forty minutes, the review that caught a plausible-but-wrong migration, the maintenance window on a box nobody wanted to own.
So let's answer it properly. Not with a feature grid, but with a number you can defend in a budget review.
Define the unit before you define the winner
Price every lane in cost per merged PR. Not cost per seat, not cost per token, not "free".
$$
C_{lane} = \frac{\text{model spend} + \text{retry minutes} \times H + \text{review minutes} \times H + \text{blocked wait} \times H + \text{amortized ops}}{\text{merged PRs}}
$$
Where H is your loaded hourly engineer cost divided by 60. Yes, review minutes belong in the numerator. A free lane that doubles review time is not free; it moved the invoice to a person who does not see it.
Two things to exclude on purpose:
- Sunk setup cost. Amortize it, do not argue about it. Past hours are not evidence.
- "Developer happiness" unless you have a proxy. A proxy is something like voluntary lane usage after week two.
A scorecard like this is a conversation tool, not objective truth. It exists so two people can disagree about the same numbers instead of different vibes.
The ledger: twelve fields, one line per task
You cannot compare lanes you did not measure. Append-only JSONL, committed to the repo, reviewed in a PR like any other artifact.
{"date":"2026-10-15","lane":"free","task":"TICKET-412","tokens_in":8200,"tokens_out":1900,"wait_s":37,"retries":1,"review_min":22,"merged":true}
Logging one task without a spreadsheet:
start=$(date +%s)
# ... run your agent or assistant on the ticket ...
end=$(date +%s)
printf '{"date":"%s","lane":"%s","task":"%s","tokens_in":%s,"tokens_out":%s,"wait_s":%s,"retries":%s,"review_min":%s,"merged":%s}\n' \
"$(date -I)" "free" "TICKET-412" 8200 1900 $((end-start)) 1 22 true >> ledger.jsonl
If your tool does not report token counts, write null and record the proxy you used. A labelled proxy beats a fabricated integer every time.
Why JSONL and not a dashboard
Because a dashboard someone else owns becomes a number nobody trusts. A file in the repo survives re-quotes, vendor changes, and the person who set it up leaving.
The script that ends the argument
import json, collections
H = 85 / 60 # loaded engineer cost per minute; replace with yours
RETRY_MIN = 12 # your measured rework per failed attempt; replace
def summarize(path):
agg = collections.defaultdict(lambda: dict(prs=0, tokens=0, wait=0.0, retries=0, review=0.0))
with open(path) as f:
for line in f:
if not line.strip():
continue
r = json.loads(line)
a = agg[r["lane"]]
a["prs"] += 1 if r["merged"] else 0
a["tokens"] += (r["tokens_in"] or 0) + (r["tokens_out"] or 0)
a["wait"] += r["wait_s"] / 60
a["retries"] += r["retries"]
a["review"] += r["review_min"]
out = {}
for lane, a in agg.items():
if a["prs"] == 0:
continue
minutes = a["review"] + a["wait"] + a["retries"] * RETRY_MIN
out[lane] = {"prs": a["prs"], "cost_per_pr": round(minutes * H / a["prs"], 2)}
return out
def break_even(free_per_pr, hosted_var_per_pr, fixed_month, maint_hours, loaded_hourly):
monthly = fixed_month + maint_hours * loaded_hourly
saving = free_per_pr - hosted_var_per_pr
return None if saving <= 0 else round(monthly / saving)
Run it weekly. If a lane has fewer than twenty merged PRs, stop and collect more data — small-n comparisons in this space are mostly noise wearing a decimal point.
Break-even: the only number that survives a re-quote
A self-hosted or paid lane is worth it only above a volume threshold:
N* = (fixed monthly + maintenance hours × loaded hourly) / (free cost per PR − hosted variable cost per PR)
Worked example. All figures below are illustrative inputs, not vendor quotes — plug in your own.
- Fixed monthly hosting/seat cost: $180
- Setup: 30 hours, amortized over 12 months → 2.5 h/month
- Ongoing maintenance: 6 h/month at $85/h
- Free lane: $4.10 per merged PR (review-heavy, retry-heavy)
- Hosted lane variable: $0.90 per merged PR
Monthly fixed-equivalent = 180 + (6 + 2.5) × 85 = $902.50. Saving per PR = $3.20. So N* = 282 merged PRs per month.
Sensitivity, same base except where noted:
| Change | N* |
|---|---|
| Base | 282 |
| Maintenance drops to 2 h/month | 176 |
| Maintenance rises to 12 h/month | 441 |
| Free lane gets cheaper ($1.80/PR) | 1,003 |
| Hosted variable rises to $1.60/PR | 361 |
Read the second row, then the fourth. The decision is far more sensitive to your free-lane cost and your maintenance hours than to list price. That is the whole point of measuring before switching.
Runway: how long a free allowance actually lasts
An allowance is a pilot budget, not a P&L line. The project operator states a free allowance of 10 million tokens and a free server option; verify the current terms yourself before you commit, because allowances move and screenshots go stale.
months_of_runway = allowance / (median_tokens_per_task × tasks_per_month × people)
Illustrative arithmetic with your own median, not a promise: 9,000 tokens per task, 25 tasks per developer per week, 4 developers → 430 tasks/month → 3.87M tokens/month → about 2.6 months. That is enough to fill a ledger. It is not enough to skip the ledger.
MonkeyCode is the open-source project in that screenshot. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The free model access and free server option are what make it a usable zero-cost control arm in the experiment below — that is the honest role, and it is a real one.
Decision table: fit, not features
| Your constraint | Free lane fits when… | Self-hosted fits when… | Paid seat fits when… |
|---|---|---|---|
| Volume | below N*, i.e. you are still learning |
above N* for two consecutive months |
unpredictable, spiky |
| Review capacity | you have slack reviewers | you have a platform owner | vendor review tooling is the point |
| Data sensitivity | work is public or synthetic | data cannot leave your network | contract terms beat your own ops |
| Ops appetite | nobody wants a pager | you already run services | you want no pager at all |
| Reproducibility | you can tolerate drift | you need pinned versions | you need a paper trail, not control |
Three gates, one owner, one expiry
Every pilot needs an end date or it becomes architecture by accident.
- Gate 1 — Instrument (days 0–3). Owner: the lead running the pilot. Exit: twenty merged PRs per lane in the ledger, or the comparison is void.
-
Gate 2 — Compare (day 14). Exit:
N*computed and compared to your actual monthly volume. Decision rule: below 0.5 ×N*, stay free; above 2 ×N*, re-quote hosted; between, extend once, with a written reason. - Gate 3 — Re-verify and exit (day 30). Re-check the allowance and server terms on the date you re-check them, then publish the ledger and pick a lane. Expiry: 30 days from Gate 1. No silent renewals.
If you want a zero-cost control arm while you run this, the free tier and free server option are a reasonable place to start logging — check the repo README for current terms first, then let the ledger argue.
Where this model breaks
-
Thin data. Under twenty PRs per lane,
N*is theatre. - Unlogged review debt. If reviewers do not record minutes, the free lane always looks cheaper than it is.
- Allowance drift. A quota change mid-pilot invalidates your runway line, not your cost-per-PR line.
- Free server expectations you never wrote down. Where the compute lives is not the same as who owns the outage. Ask before you depend on it.
-
One team's numbers.
N*does not transfer between teams with different review cultures.
Who should not use this approach
Teams that cannot commit to logging for two weeks. Teams whose work is confidential and whose free-lane data path is unverified. Teams under roughly 50 merged PRs a month where a single bad week swings the average. And anyone hoping the ledger will confirm a decision they already made.
The reversal question
Before you sign anything, write down which single variable would flip your choice: maintenance hours, review minutes, volume, or the allowance itself. Then go measure that one first.
Which number in your current setup have you never actually measured — and what would it cost you to find out this week?
Top comments (0)