Eleven of twenty-four AI-assisted pull requests merged last week with zero review comments. Median time-to-approve: nine minutes. Tests green. That number should reverse a buy. It should also reverse a self-host.
Why would you add capacity to a lane that already stopped being engineering?
Picture Tuesday standup. Maya, an EM on a payments squad, is proud. Cycle time dropped. Then a rollback lands at 4 p.m. because an AI-authored retry loop double-charged a webhook. Nobody in review asked how idempotency worked. The tool was not the failure. The silent approve was.
Names here are composite. The numbers below are a worked example, not a customer case. I still use this filter because the incentive is the same every time: a free or paid lane that makes diffs look finished will get expanded for the wrong reason.
The unit is question rate, not cycle time
Cycle time is a bad proxy. It moves in the same direction as both real engineering and pretend engineering. Question rate does not.
Question rate is review comments plus blocking questions per AI-assisted PR, counted after the first bot noise is stripped. Silent-approve rate is the share of those PRs merged with zero human comments and a sub-15-minute approve. If those two signals invert while cycle time “improves,” you do not have an adoption win. You have a trust unit going to zero.
Would you buy more seats for a queue nobody interrogates? I would not. Would you stand up a GPU box for the same queue? Also no.
This gate is a conversation tool. It is not objective truth. Game it and it will lie. Use it anyway, because token stickers and seat counts lie faster.
Four options, one bottleneck
You are not choosing a model brand. You are choosing blast radius.
- Keep a bounded free lane. Capacity stays small on purpose. You measure trust before you spend.
- Buy a paid lane. You pay for context, SLA, or vendor features. You also scale whatever habit you already have.
- Self-host. You pay in ops time for data control. Same habit risk. Different invoice.
- Stop or shrink. The lane is producing silent approves plus rollbacks. Extra capacity is malpractice.
A free model-access path and a free server option only matter in option 1. They are a blast-radius cap, not a trophy. If your reviewers already went quiet, paid and self-hosted are the same mistake with better hardware.
Variable definitions
Write these on a wiki page before you argue in Slack.
-
AI_PR_WEEK: PRs in the last 7 days labeled AI-assisted by the author or your bot. -
Q: human review comments + blocking questions on those PRs. Strip linter bots. -
QR = Q / AI_PR_WEEK. -
QR_HUMAN: same ratio on PRs with no AI label, same week. -
QR_RATIO = QR / QR_HUMAN. -
SA: silent approves. Zero human comments, merge under 15 minutes, tests green. -
SA_RATE = SA / AI_PR_WEEK. -
RB_H: hours spent rolling back or hotfixing AI-assisted changes in the last 30 days. Include the meeting. -
TEST_DELTA: new or changed tests per AI-assisted PR. 0.0 means the model wrote product code and nobody added proof. -
EXPLAIN_OWNER: named human who can defend the change without re-reading the model transcript.
If you cannot fill those fields from git and review data, you do not have a buy/self-host decision. You have a vibe.
A 30-minute ledger, labeled unexecuted
Do not trust memory. Export one week.
# Unexecuted example. Adjust labels to match your process.
gh pr list --state merged --search "merged:>=2026-09-10" --limit 50 \
--json number,title,author,mergedAt,reviews,comments,labels \
> /tmp/pr-week.json
Then score it. This script is a proposal, not a benchmark I ran on your repo.
# Unexecuted example: question-rate ledger from a simplified CSV.
# columns: pr, ai, human_comments, merge_minutes, rollback_hours, tests_changed
import csv
from collections import Counter
rows = list(csv.DictReader(open("ai_review_week.csv")))
ai = [r for r in rows if r["ai"] == "1"]
human = [r for r in rows if r["ai"] == "0"]
def mean_comments(group):
return sum(int(r["human_comments"]) for r in group) / max(len(group), 1)
qr_ai = mean_comments(ai)
qr_human = mean_comments(human)
sa = sum(1 for r in ai if int(r["human_comments"]) == 0 and float(r["merge_minutes"]) < 15)
rb = sum(float(r["rollback_hours"]) for r in ai)
td = sum(int(r["tests_changed"]) for r in ai) / max(len(ai), 1)
print({
"ai_pr_week": len(ai),
"qr": round(qr_ai, 2),
"qr_ratio": round(qr_ai / qr_human, 2) if qr_human else None,
"sa_rate": round(sa / len(ai), 2) if ai else None,
"rb_h": rb,
"test_delta": round(td, 2),
})
If QR_RATIO is missing because you never labeled AI PRs, stop. Label for two weeks. Then decide. Expanding first is how you buy a mess.
Worked example: Northwind payments, 8 engineers
Hypothetical week. I am not claiming this is a real team.
| Field | Human PRs | AI-assisted PRs |
|---|---|---|
| Count | 10 | 24 |
Mean human comments (QR) |
1.80 | 0.42 |
| Silent approves | 1 | 11 |
SA_RATE |
10% | 46% |
TEST_DELTA |
1.4 | 0.3 |
RB_H (30-day, AI only) |
— | 14 |
QR_RATIO = 0.42 / 1.80 = 0.23.
Maya’s cycle time looks great. The ledger does not. Eleven silent approves is not “the model got better.” It is reviewers treating green CI as a personality.
Decision under this gate: do not expand. Do not buy. Do not self-host for “more speed.” Keep a bounded free lane only if you still need the tool for drafts, and freeze it to files the squad already owns. If RB_H stays above the rollback gate below, shrink further.
Hard gates (any one fails → no expand)
These are stop signs, not vibes.
-
SA_RATE> 30% on AI-assisted PRs. Reviewers went quiet. Extra capacity scales silence. -
QR_RATIO< 0.35. AI diffs get less than about one-third the interrogation of human diffs. -
RB_H> 8 hours / 30 days. The lane is already costing review-adjacent labor you will not see on a token invoice. -
TEST_DELTA< 0.5. The model is writing product code without proof. Buying context will not grow a test habit you refused to require. -
No
EXPLAIN_OWNER. If the author cannot defend the change without the chat log, it is not a PR. It is a paste.
If gate 1 and gate 3 both fire, the call is stop or shrink, not “try a better model.” A better model with the same review policy is a faster webhook double-charge.
Sensitivity: which number reverses the call?
Hold Northwind constant and move one field.
- If
SA_RATEdrops from 46% to 18% andQR_RATIOrises to 0.70, buy becomes discussable — but only if the bottleneck is context or vendor workflow, not GPU envy. - If
RB_Hfalls from 14 to 3 andTEST_DELTAhits 1.0, self-host becomes discussable — but only if data residency or air-gap is a real constraint, not a status purchase. - If
AI_PR_WEEKdoubles whileQRstays at 0.42, the decision gets worse. Volume is not evidence. - If human
QRalso collapses to 0.42, you do not have an AI problem. You have a review-culture problem. No lane, free or paid, will save you.
Ask the squad this out loud: which single threshold would reverse your expand vote? If nobody can name one, you are shopping.
Free vs paid vs self-host under a quiet review queue
Fit is not “which model is smart.” Fit is whether capacity is the constraint.
| If this is true | Keep bounded free | Buy | Self-host |
|---|---|---|---|
| Reviewers still ask hard questions | Maybe | Maybe | Maybe |
SA_RATE already > 30% |
Cap, don’t market it | No | No |
| Context across repos is the actual bottleneck | Weak fit | Stronger fit | Only if you will staff the box |
| Data cannot leave the building | Weak fit | Usually no | Only after trust gates pass |
You cannot name an EXPLAIN_OWNER
|
Shrink | No | No |
| You want a trophy for “AI adoption” | No | No | No |
Disclosure: This article was prepared as part of MonkeyCode's product outreach. MonkeyCode is an open-source project with free model access and a free server option. I am not using that as a bake-off win. I am using it as an example of a bounded lane: useful while you run the ledger, dangerous if you treat free capacity as a reason to skip gates. The article still works if you never touch that project. Substitute any free lane you already have.
Paid and self-hosted fail the same way free fails. They fail faster when the invoice or the rack implies you must “get value.” Incentives leak into review. Quiet reviewers plus a larger context window is how pretend-engineering gets a budget line.
Owner, expiry, archive
- Owner: the EM plus one staff reviewer who is not the vendor champion. The champion can collect data. They cannot grade their own expansion.
- Expiry: 14 days for the first ledger, 30 days after that. An expired exception is a no.
- Exit: if gate 1 and gate 3 both fire in a window, disable AI labels from merge, or cut the lane back to draft-only on non-production paths. Write the rollback in the same doc as the adoption note.
- Archive rule: keep the CSV. Do not keep screenshots of cycle-time dashboards as proof.
Put the expiry on a calendar. Verbal “we’ll watch it” is how silent approves become culture.
Who should not use this gate
- Teams with no review data and no will to label AI PRs. You cannot price trust from anecdotes.
- Security or compliance work that needs a legal review of any tool, free or not. A decision card is not a SOC report.
- Solo hackers optimizing for learning speed. Question rate is a team unit.
- Anyone hunting a promotional build-versus-buy conclusion. If you already picked a vendor, this ledger will feel hostile. Good.
Limitations
Small, boring diffs can be silent and still correct. A 4-line typo fix should not inflate SA_RATE the way a 400-line retry loop should. Split trivial PRs out, or you will punish good hygiene.
Comment count can be gamed. “LGTM” is not a question. Count blocking questions if your tool allows it. If it does not, sample 10 PRs by hand once a week. Hand samples beat vanity metrics.
I have not run these commands on your repository. There is no benchmark hiding in this post. There is a policy you can fill, argue with, and expire.
The question that actually matters
Not “would you try the tool?” Not “is free cheaper than a rack?”
If silent-approve rate stayed at 46%, what extra capacity would you still purchase? If the answer is none, keep the lane bounded or kill it. If the answer is “we’d buy anyway because leadership wants AI,” you do not have a model problem. You have a P&L that is rewarding pretend work.
Run the ledger on last week’s merges. Fill QR_RATIO, SA_RATE, and RB_H. Then pick keep-bounded, buy, self-host, or stop. If you need a cap while you measure, a free-model plus free-server lane is one way to keep capacity from becoming the decision. The gate still comes first.
Top comments (0)