I still start these reviews the same way. Not with a model leaderboard. With the merge queue.
A twelve-person squad can mint more unreviewed AI answers in one week than staff engineers can reconcile in a month. That ratio can reverse a “make the free path the default” decision overnight. A viral thread claiming AI already codes better than most developers does not pay the review tax. You do.
Picture Tuesday standup. The EM drops a screenshot. Three people solved the same auth-boundary question in three private chats. Three answers. Three slightly different error contracts. Zero shared constraints in the repo. Cheap generation. Expensive divergence. Sound familiar?
The wrong question
“Is the model good enough?” is a product-marketing question. It is not an adoption question.
The adoption question is uglier. Should this free coding path stay a personal scratchpad, or should you promote it into shared context the team is allowed to trust?
Those are different products wearing the same chat UI. Mix them up and you get shadow standards, unowned incidents, and a prompt folklore nobody can exit.
I use four gates. They are a conversation tool, not objective truth. If one variable below would flip your call, write it down before anyone “standardizes” a server.
Scratchpad vs shared context
A scratchpad is private, disposable, and easy to abandon. A shared context path is a workflow. It has owners, reuse, and a blast radius when it is wrong.
| Mode | What you are actually buying | Failure mode |
|---|---|---|
| Personal scratchpad | Speed on a single task | Divergent patterns, duplicated wrong answers |
| Shared context | Reused constraints, fewer repeated questions | Unowned “the agent said so” merges |
| Paid team workspace | Vendor workflow, seats, support path | Seat proxies hiding unused licenses |
| Self-host | Control of data path and teardown | You inherited an internal SaaS |
Free model access and a free server option sit in the first two rows until you promote them. Promotion is the decision. Not the signup.
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is one open-source coding product that currently offers free model access and a free server option. I am not treating those as a quality ranking, a quota promise, or a permanent SKU. I am treating them as a cheap way to run the gates below before you pay, self-host, or freeze a standard.
Would you promote a wiki that nobody edits? Then do not promote a chat log either.
The 4-gate ownership test
Define the variables first. Then score. Then argue.
- Context reuse rate (CRR). Share of AI answers that get pasted, linked, or encoded into repo artifacts another human can find in 90 seconds. Unit: percent of AI-touched tasks in a 14-day window.
- Review hours per AI-touched merged PR (RH-PR). Staff time spent on first review plus re-review, including “please undo the clever bit.” Unit: hours / merged PR.
- Named revert authority (NRA). One human who can revoke the path, revert the pattern, and say no without a committee. Unit: named role, not a Slack channel.
- Folklore exit days (FED). Calendar days to recapture prompts, rules, and “we always do it this way” into files in git if the free path vanishes. Unit: days.
Score each gate 0, 1, or 2. Total 0–8. This is not a personality test. It is a promotion test.
Scoring rubric
CRR
- 0: reuse under 15%. Private chats die in the thread.
- 1: 15–40%. A few snippets land in PRs, still not searchable.
- 2: over 40%. Constraints live in repo docs, fixtures, or policy files.
RH-PR
- 0: review hours up more than 25% vs the prior 14 days on comparable PRs.
- 1: flat within ±25%.
- 2: down more than 25%, with the same review bar. No silent rubber-stamping.
NRA
- 0: “the team” owns it. Nobody can kill it on Friday.
- 1: an EM owns it, but only after a sync.
- 2: a named owner can revoke access the same day, in writing.
FED
- 0: exit would take more than 14 days or depends on one person’s head.
- 1: 8–14 days, with a documented prompt dump.
- 2: 7 days or fewer, because rules already live in the repo.
Hard promotion rule I use: do not promote to shared context unless CRR ≥ 1, NRA = 2, and total ≥ 5. A pretty model demo cannot override a missing owner. Should it?
Filled example (hypothetical squad, labeled as such)
Squad Helios. 12 engineers. 14-day window. Labeled unmeasured proposal, not a customer case.
They want to promote a free coding server from “try it” to “how we answer architecture questions.”
Observed in the worksheet (hypothetical):
- 40 AI-touched tasks
- 8 produced a repo artifact another person reused → CRR = 20% → score 1
- Mean RH-PR moved from 1.6h to 2.1h → +31% → score 0
- Owner is “platform, kind of” → NRA score 0
- Prompts live in four Notion pages and one engineer’s local snippets → FED ≈ 15 days → score 0
Total: 1 / 8. Keep it a scratchpad. Do not put it on the onboarding doc. Do not imply the answers are team canon.
Same squad, different 14 days, after they force one rule: every reused answer must land in /docs/ai-constraints/ or it does not count as shared.
- CRR 45% → 2
- RH-PR 1.4h vs 1.6h baseline → 2
- Staff engineer Maya is revert authority, written in the README → 2
- Constraints already in git, FED = 3 days → 2
Total: 8 / 8. Now you may discuss promotion. You still set an expiry. A scorecard is not a marriage certificate.
Sensitivity that should change your mind
Hold three variables still. Move one.
| Lever you change | If it moves this way | Decision flip |
|---|---|---|
| CRR | Falls under 15% | Demote to personal scratchpad |
| RH-PR | Rises over 25% | Freeze promotion, inspect review bar |
| NRA | Owner goes on PTO with no deputy | Suspend shared use the same day |
| FED | Exit estimate exceeds 14 days | Stop writing new shared prompts |
| Incident | One production miss tied to an unreviewed answer | Exit the shared path; scratchpad may remain |
Break-even I actually argue about: hours saved in repeated questions versus hours added in review.
Weekly question load Q. Minutes saved per reused answer S. Reuse rate CRR. Extra review minutes per AI-touched PR R. AI-touched PRs per week P.
Net hours ≈ (Q * CRR * S) / 60 - (P * R) / 60
If net hours go negative, the free path is a cost center even when the invoice is zero. Invoice-zero is not P&L-zero. You knew that, right?
Worked numbers for Helios, still hypothetical: Q = 30, S = 12, CRR = 0.20, P = 10, R = 30.
Net ≈ (30 * 0.20 * 12)/60 - (10 * 30)/60 = 1.2 - 5.0 = -3.8 hours/week.
That is the whole case against promotion. Not model quality. Not a tweet. Review tax larger than reuse.
Raise CRR to 0.50 and drop R to 10 minutes because constraints are in git: net ≈ (30 * 0.50 * 12)/60 - (10 * 10)/60 = 3.0 - 1.7 = +1.3 hours/week. Only then is shared context even in the conversation.
Measurement you can run without a vendor dashboard
Label this as unexecuted template. Do not pretend it is telemetry you already have.
# Last 14 days of merged PRs, titles only. Adjust the date.
git log --merges --since='2026-08-28' --pretty=format:'%h %ad %s' --date=short
Keep a boring CSV. If it is not boring, people will stop filling it.
date,task_id,ai_touched,reused_in_repo,review_hours,merged,owner
2026-09-01,PAY-118,yes,yes,1.2,yes,maya
2026-09-02,PAY-121,yes,no,2.4,yes,lee
2026-09-03,PAY-126,no,no,0.8,yes,maya
Proposed scorer. Hypothetical. Run it on your sheet, not on my claims.
# proposal only: score promotion gates from a local csv
import csv
from statistics import mean
BASELINE_RH = 1.6 # hours; set from YOUR prior window
rows = list(csv.DictReader(open("ai_context_gate.csv")))
ai = [r for r in rows if r["ai_touched"] == "yes"]
crr = sum(r["reused_in_repo"] == "yes" for r in ai) / max(len(ai), 1)
rh = mean(float(r["review_hours"]) for r in ai) if ai else BASELINE_RH
def bucket_crr(x):
return 2 if x > 0.40 else 1 if x >= 0.15 else 0
def bucket_rh(x, base):
delta = (x - base) / base
return 0 if delta > 0.25 else 2 if delta < -0.25 else 1
print({"crr": round(crr, 2), "crr_score": bucket_crr(crr),
"rh_pr": round(rh, 2), "rh_score": bucket_rh(rh, BASELINE_RH)})
NRA and FED still need a human. Scripts do not name owners. Scripts do not recapture folklore. If your “owner” is a distribution list, you already scored a zero.
Hard gates, owner, expiry, exit
Write these in the same README you were about to stuff with tool cheerleading.
- Owner: one staff engineer or EM. Deputy named. Not “platform.”
- Expiry: 14 days after promotion, auto-demote unless the four scores are re-recorded.
- Stop gates: CRR < 15%, RH-PR +25%, missing owner, or one production incident tied to an unreviewed answer.
- Exit: prompts and constraints already in git, or FED work starts the day you promote, not the day the free server hiccups.
- Non-goals: do not use seats, token headlines, or “it feels faster” as proxies.
If you cannot name the exit, you are not buying leverage. You are buying a habit.
Where free models and a free server actually fit
Use a free-model, free-server path when you are still in measurement mode. You need enough usage to fill the CSV. You do not yet need a seat contract or a self-hosted internal SaaS.
Stay on personal scratchpad if CRR is low and RH-PR is rising. That is most squads in week one. It is not a moral failure. It is a workflow mismatch.
Consider paid or self-host only after promotion passes twice in a row. Paid does not fix missing ownership. Self-host does not fix a review bar you were afraid to keep. Build-versus-buy is downstream of this test. Reverse that order and you will decorate a bad habit with invoices or with YAML.
MonkeyCode is a reasonable place to run the first 14-day measurement if a free model path plus a free server is what you need to get the worksheet. Run the gates first. If the score stays a 1, do not “standardize” anything. Just close the tab.
Who should not use this approach
Do not promote a third-party free server into shared context if you cannot put that code or that prompt traffic on someone else’s machine. Regulated data, unpaid legal review, or “we will anonymize later” is a stop, not a yellow flag.
Do not use the scorecard if you will treat a 6 as a mandate. It is a conversation tool. Two senior engineers can weight RH-PR differently. Good. Put the disagreement on the page.
Do not use it to launder a quality claim. I am not saying any model is better than your developers. I am saying unowned generation does not become knowledge because the chat was fluent.
Skip the whole promotion idea if the squad already has a working design-review ritual and AI is only used for local boilerplate. You do not need a shared context program to rename variables.
Limitations
I did not benchmark models. I did not invent a quota, a GPU SKU, or a durability promise. Token headlines change behavior; they are still a bad unit for this decision. Review hours and reuse are slower to collect and harder to fake. That is the point.
Fourteen days is an arbitrary window. If your merge cadence is weekly, extend the window or you will overfit two PRs. If your cadence is many merges per day, 14 days may hide a bad week. Say the window out loud.
And if the only artifact you can produce is a vibe, you are not ready to promote. You are ready to keep a scratchpad.
So which variable would reverse your call? CRR, RH-PR, the named owner, or the exit clock? If you cannot pick one, you are not deciding. You are hoping the free path stays free and the review queue stays quiet. Hope is not a gate.
Top comments (0)