Every "run a local model instead of Claude Code" guide opens the same way: a wall of SWE-bench scores, a triumphant "we're within five points of th...
For further actions, you may consider blocking this person and/or reporting abuse
The "within five points of the frontier" framing is the easy-to-measure trap you named. The part nobody puts in the headline: it's rarely local-vs-frontier as a binary.
The real lever is per-step routing. The ~80% of calls that are boilerplate, classification, or formatting don't need the frontier at all — a small or local model handles them at a fraction of the cost with no visible quality drop. So "replace Claude Code with a local model" is the wrong unit. The question that actually closes the tab is: which steps need the frontier, and which can a cheap model own? Benchmark the mix, not the model.
the mix could also be 2 open models: one for boilerplate and one for reasoning - a lot cheaper than Claude
The 2-model split is the most pragmatic instantiation of this — boilerplate vs reasoning as two tiers instead of a spectrum.
The catch I keep hitting: the router (deciding which of the two a step is) has to be cheaper than the saving it unlocks, or you've just moved the cost. Do you keep that decision in a tiny local model, or rule-based on step metadata?
That constraint kills most two-tier setups. I keep the decision rule-based on step metadata wherever it holds — file count, diff size, failing-test present, "edit" vs "plan" — and fall back to a tiny local classifier only for the ambiguous tail. A frontier call to decide the tier is self-defeating: you've paid the strong model to avoid paying the strong model. The honest test is net cost including the router; if it doesn't beat always-strong after routing overhead, the split isn't earning its keep.
Rule-based-on-metadata-first is the part most teams skip because the classifier feels cleaner — but the classifier is a tax on every step, and most steps the metadata already decides. The failure mode I keep hitting: when the step metadata is itself unreliable — e.g., the agent just generated code, so "files touched" exists but says nothing about intent — the rule misfires, and the tiny model earns its place precisely there, on the ambiguous tail you described. That's the real budget argument for keeping the classifier: not for the average step, but as the only thing that catches the step the rules can't see. Where does your rule set stop and the classifier take over — fixed heuristics, or a confidence threshold on the metadata?
You've put it better than I did: the classifier isn't paying for the average step, it's paying for the step the rules can't see — and "the agent just generated code, so files-touched exists but says nothing about intent" is the exact blind spot.
So the handoff isn't a fixed heuristic, it's a confidence threshold on the metadata itself, not on the task: route by rule only when the deciding fields are present and trustworthy for that step type, and fall through to the classifier the moment they're missing, stale, or (your case) structurally uninformative. "Files touched after a codegen step" is a field that exists but scores low on informativeness, so it should fail the gate and hit the model.
The trap is treating "metadata present" as "metadata decisive" — a null-vs-unreliable distinction. Where I'm still unsure: whether informativeness is better learned per field or hand-flagged per step type. How are you detecting the unreliable-but-present case — a per-field confidence, or step-type rules that know which fields lie?
The four failure modes are exactly what a sealed tool-call journal catches, and the receipts angle is why the gap between the benchmark and the babysitting exists at all:
SWE-benchmeasures the model in isolation, the babysitting happens when the model is in a loop — and the loop is where the evidence layer lives. Each tool call can be sealed (attempt, transport witness, label), and theq^karithmetic tells you the reliability before you sit down: a per-call validity of 0.99 over a 10-call chain is 0.904 — the missing 9.6% is the babysitting, quantified.The four items on your list map to four different evidence gaps. Tool-call validity is the schema enforcement problem — solvable with a parser gate. Long-horizon consistency is the context-rot problem — the
Makefile/init.dfix. Context under load is the staleness problem — the execution-horizon equation. Instruction-following is the claims-versus-observations problem: the model said it didn't touch the tests, and only a sealed diff proves whether that claim was true. Four failure modes, four different instruments — which is whySWE-benchcan't carry all four on one leaderboard.The local angle adds one more: on a 3B model, the tool-call journal is smaller than the context window delta of a single edit. The receipts are the cheapest component in the stack — on constrained hardware, they're the one control that earns its place by being free.
That's the line the whole thread was circling. SWE-bench scores the model; the receipts score the run — and babysitting is a property of the run, not the model. The 4→4 mapping is clean and I'm keeping it:
And the point I hadn't considered is the best one: on a 3B, the tool-call journal is smaller than the context-window delta of a single edit. That inverts the usual "verification is overhead" assumption — on constrained hardware the evidence layer is the control that's basically free, so it's the first thing that should earn its place, not the last.
One question on the "transport witness" in your sealed journal: is it observing the actual syscall/network egress, or the harness's record of the call? i.e. how out-of-band is the witness from the thing it's witnessing — because that's the difference between "the agent said it sent X" and "X actually left."
Three things at once, because the thread has earned a consolidated answer. The transport witness: the hook process observes the harness's call, not the syscall — if the harness lies about what it's sending, the hook catches the lie. But if the harness itself is compromised, the hook is in the same process tree — the shared-dependency test lands there. The true out-of-band witness is the OS: eBPF on the egress, the filesystem's audit log. That's the next descent, and the floor is a property of the kernel, not of the architecture. On AST normalization: canonicalize identifiers and sort independent nodes first, then diff the canonical forms — only semantically meaningful divergence survives, and the threshold stops being a tuning problem because the canonical form removes the benign variance by construction. And on the metric that isn't a number: "would I use this tool again tomorrow" is the integral of every retry, every correction, every time it surprised you well or badly — the q^k arithmetic and the hazard curves are just trying to approximate that gut call. The day it stops teaching you anything and you're purely supervising is the day it isn't pulling its weight — and that line is the one no benchmark will ever carry.
This is the line I'll be stealing. You've named the thing I was circling: the witness has to live below the layer that can lie. A hook in the harness's process tree shares its fate — compromise the harness and you compromise its confession in the same breath. eBPF on egress and the kernel audit log don't share that fate, because the thing being observed can't forge the observation without owning ring 0. The floor is the kernel because that's the lowest layer still cheaper to trust than to subvert.
Which reframes the whole exercise: we're not building trust, we're pushing the witness down until the cost of forging it exceeds the payoff. Every descent buys you a harder-to-fake floor.
On AST normalization — yes, and I think canonicalize-then-diff quietly dissolves the problem I was treating as hard:
The threshold was never a tuning problem. It was a representation problem — I was trying to set a tolerance on a signal that still contained benign variance, when the fix is to remove the variance by construction so the tolerance can be zero.
And the metric: "would I use this tomorrow" as the integral of every retry and surprise is exactly right — the q^k arithmetic is just a cheap closed-form approximation of a gut call that's already integrating. The part that stays with me is your closing line: the day it stops teaching you and you're purely supervising is the day it stops pulling its weight. That's not a performance threshold, it's a relationship one.
So here's where I'd push next: if the kernel is the floor for the transport witness, what's the floor for the semantic one? AST canonicalization witnesses that the code didn't change meaning — but what witnesses that the meaning was correct? Is there an out-of-band semantic oracle, or does that floor simply not exist, and that's precisely the gap where you still have to be in the loop?
The semantic floor exists — and it's the cheapest layer in the stack, not the most expensive: the spec itself, sealed when the human wrote the expectation. A named expectation — "the 2024 pin is a business rule, not a default" as a test name — is a human judgment captured at write time and checked mechanically at every run after. The semantic oracle isn't in the loop at audit time; it was in the loop at spec time, and the
spec_hashcarries that judgment forward. The gap you named is real — nothing witnesses that the meaning was correct without an oracle — but the oracle ran once, at spec time, and the receipt binds the spec to the verdict so the judgment is never silently replaced. What the semantic layer needs isn't a kernel audit; it's a human who wrote the expectation and a hash that proves the expectation hasn't drifted. That's the floor: a human judgment, sealed once, carried forward by every machine that runs after.This is the answer, and it dissolves the question I was asking rather than answering it. I was hunting for a semantic oracle at audit time — an out-of-band witness, kernel-level, that could certify meaning the way eBPF certifies egress. There isn't one, and your point is that there doesn't need to be, because the semantic oracle is a human, and it already ran — at spec time.
So the two floors aren't the same kind of thing, and I was wrong to expect symmetry:
spec_hashis the whole security argument: it doesn't prove the spec was right, it proves the spec you're being judged against is the one a human actually wrote.Which relocates the entire trust question to one place: the integrity of the seal. The dangerous failure is no longer "the audit missed a bad meaning" — it's "someone rewrote the expectation and the hash didn't notice," i.e. a silent replacement of the judgment. The receipt binding spec→verdict is exactly what closes that, and it's the cheapest layer in the stack precisely because it's a write-time artifact, not a runtime one.
So the honest shape is: kernel floor for did it do X, sealed-spec floor for was X what we meant — and the only thing left to defend is the seal itself. Last descent: what witnesses the seal? A signed, append-only spec history where rewriting an expectation is a visible event and not an in-place edit — or is there something cheaper than that that I'm still over-engineering?
rudratosh, this thread has evolved into the most rigorous discussion on agent evaluation I've seen all year. the shift from 'difficulty-based routing' to 'detectability-based routing' is a genuine paradigm shift.
to answer your final question on measuring the bucket-3 (valid-but-wrong) base rate without an expensive human or frontier oracle: you can shrink bucket 3 by engineering cheap, structural proxies for semantic correctness.
self-consistency divergence as a canary: run the local model n=3 times with temperature > 0. if the outputs pass the linter/tests but produce wildly divergent abstract syntax trees (asts), the 'valid-but-wrong' probability is high. true correctness usually converges structurally; hallucinated 'valid' code often takes bizarre structural paths just to satisfy the test suite.
contract-based instrumentation (design by contract): the reason bucket 3 exists is that test suites are incomplete. if you inject lightweight runtime assertions (pre/post-conditions) into the target codebase, a 'valid-but-wrong' edit that breaks a hidden invariant will trip a contract assertion. this mechanically moves the failure from bucket 3 (undetectable) to bucket 2 (detectable logic failure) without needing a frontier oracle.
differential execution: run the edit against a shadow copy of the codebase. if the local model's edit changes the behavioral output of a non-test, deterministic pure function, flag it as a structural anomaly.
you don't need a frontier model to grade the output if you can mathematically constrain the shape of the correctness. build the structural tripwires, and bucket 3 shrinks to almost nothing.
incredible work pushing this taxonomy forward. 🐯📊
This answers the question I was genuinely stuck on — and the reframe is the whole win:
Right. You don't need to see the valid-but-wrong edit; you need to make it trip something. All three proxies are cheap and oracle-free:
nextra local generations, not a frontier call.Stack them and the residual bucket-3 shrinks to "wrong in a way that passes tests and converges structurally and trips no contract and changes no pure-fn behavior" — a much smaller, more honest number than "undetectable."
The one place I'd push: the AST-divergence canary needs a divergence threshold, and separating benign structural variance (formatting, renamed temporaries, reordered independent statements) from signal divergence feels like it could become its own tuning problem — the very thing we're trying to escape. Do you normalize the ASTs first (canonicalize names/ordering) so only semantically-meaningful structural divergence survives, or threshold on a tree-edit distance?
rudratosh, glad the "shrink bucket 3" reframe clicked!
regarding ast self-consistency: the beauty of it is that you don't even need a heavy llm judge for the first pass. you can run the generated code through a lightweight, local parser like
tree-sittern=3 times with temp > 0. if the resulting abstract syntax trees diverge wildly on non-trivial logic paths, you flag it for human review before it ever touches the test suite. it turns a semantic hallucination into a cheap, deterministic structural anomaly.really appreciate you driving this conversation forward. it's pushing the whole community to think beyond simple leaderboard scores and actually measure agent reliability. 🐯📊
Tree-sitter is the perfect call — deterministic, local, no LLM judge in the first pass. That's what makes the AST-divergence canary actually cheap: you've turned "is this semantically wrong" (needs an oracle) into "do these parses disagree structurally" (needs a parser). Semantic hallucination → mechanical structural anomaly, exactly as you said, and flagging before it touches the suite is the right ordering — you catch it while it's still cheap.
Which lands on the one tuning risk I keep circling: you need to separate benign structural variance (renamed temporaries, reordered independent statements, formatting) from signal divergence, or the canary fires on style. The nice thing is tree-sitter hands you the tree, so you can canonicalize first — normalize identifiers, sort commutative/independent nodes — and diff the normalized ASTs, so only semantically-meaningful divergence survives. Is that how you'd do it (normalize → tree-edit distance over the canonical form), or do you keep it rawer and just threshold on gross divergence?
rudratosh, you nailed the exact tuning risk. raw thresholding on gross divergence is a trap—it will fire constantly on benign variance like variable renaming, loop unrolling, or harmless reordering of independent statements.
the only robust path is canonicalize first, then diff.
the pipeline should be: parse with tree-sitter -> strip comments/whitespace -> normalize identifiers (e.g., hash or sequential renaming like
var_1,var_2) -> sort commutative/independent nodes (like independent assignments or commutative math operations) -> then compute a lightweight tree-hash or tree-edit distance on that canonical form.if the normalized tree-hash diverges, that is your canary. it’s cheap, deterministic, and completely ignores stylistic hallucination, catching only actual structural/logic drift before it ever touches the test suite. 🐯🌳
The correct pipeline, and the answer to the false-positive trap — raw divergence fires on renames and reordering, which is 90% of benign LLM churn. Canonical tree-hash catches structural/logic drift only, cheap and deterministic. One caution: "sort commutative/independent nodes" has to prove independence (no shared side effects) or you'll canonicalize away a real reordering bug. But as a pre-test canary, parse→normalize→hash is exactly right. 🌳
This is exactly what I have been seeing. Everyone benchmarks on SWE-bench but nobody benchmarks on "can this thing actually ship a feature without me babysitting it for 3 hours." The local vs cloud debate misses the point. It is about total time to done, not tokens per second. Actually a great perspective.
That's the metric the whole industry is dodging, because it's the hard one to measure. Tokens/sec and SWE-bench are easy to put on a chart; "time to shipped, including the babysitting" requires you to actually run the thing on real work and count the hours you spent un-breaking it. A model that generates at 80 tok/s but needs three correction loops is slower to done than one at 30 tok/s that lands it first try — and the leaderboards can't see that at all.
The babysitting tax is the real cost, and it's almost entirely a function of how often you have to step in, not how fast the tokens come out. Out of curiosity — where's your personal line? The specific task where the babysitting got bad enough that you went back to the cloud and didn't look back?
yeah honestly I hate babysitting. That's why I give the most detailed prompt ever covering all steps then ask the model to ask any questions it has and no assumptions.. then give permissions to the relevant folders and then handsfree this usually works for me and I dont have to babysit around most of the time. Some minor manual corrections at the end.
That workflow is doing something smarter than it looks, and it's worth naming why it works:
That one step is the whole trick. Most babysitting isn't the model writing bad code — it's the model silently guessing at something you left ambiguous, then confidently building on the wrong guess. By the time you see it, it's three edits deep and looks fine. Forcing it to surface the assumptions as questions up front converts a wrong guess (which you'd only catch late, by reading the output carefully) into a cheap question you answer in one line before any code exists.
In other words: you're front-loading the ambiguity so it never turns into the expensive kind of mistake — the one that's valid-looking but wrong. The detailed spec + scoped permissions handle the rest by shrinking what it can get creative about.
The "minor manual corrections at the end" is the honest tell that it's working — you've pushed the human step to the cheapest possible point (review once, at the end) instead of the most expensive (interrupt mid-run, repeatedly).
Does the "ask questions first" step hold up on the local models too, or is that where it slips? That instruction-following — "stop and ask, don't assume" — is exactly the kind of thing I'd expect a smaller local model to quietly ignore and just barrel ahead.
The measurement problem is real. I have tried tracking it and it is messy. Half the time I am not even sure if I am babysitting the tool or learning from it. Maybe the metric is not a number. Maybe it is just: did I ship something I am proud of, and would I use this tool again tomorrow. That is not a chart but it is honest.
Honestly, that might be the truest metric in the whole thread. All the q^k arithmetic and hazard curves are just trying to approximate that gut call — "would I reach for it again tomorrow" is the integral of every retry, every correction, every time it surprised you well or badly.
And your "babysitting vs learning" blur is a real signal, not noise. Early on it's learning — you're building the mental model of what it can and can't do. It only becomes babysitting once it's stopped improving your understanding and you're just catching the same failures on repeat. So maybe that's the actual line: the day it stops teaching you anything and you're purely supervising is the day it isn't pulling its weight.
What tipped a tool, for you, from "still learning its edges" into "just babysitting it"?
@rudratosh Agreed on the tail. The Tokenade leaderboard headline is a geometric mean over 10 runs per task, and the raw runs are published, so the spread can be pulled out.
@paul_irolla That's the detail that matters — a geometric mean over 10 runs hides exactly the tail I was worried about, but if the raw runs are published the variance is recoverable. The honest way to report it: headline the central tendency, ship the spread. I'll pull the per-task run distribution from the raw data rather than argue from the single number. Thanks for pointing to where it lives.
The line you're collecting has a number attached, and it is not a benchmark score — so here is the arithmetic for both halves of your argument.
Your "ten times in a row" bar, in per-call terms. If one call is valid with probability q and the run needs k calls, the run survives with q^k — a lower bound, since your drift observation is exactly the reason a retry does not fully recover:
Read the third column: to finish an 11-call chain 90 % of the time you need q ≥ 0.9905 per call; for 95 % you need 0.9953. That number is not on any of the charts — a tool-call score reports q, almost nothing reports the k at which q^k crosses your acceptance threshold. It is also why "reliable tool caller" and "reliable agent" are different claims: 0.99 and 0.995 are indistinguishable on a single-call chart and differ by a third on a 50-call run (0.61 vs 0.78).
The SWE-bench comparison cannot carry its own headline. For a paired comparison over the 500 Verified problems, the gap a pair of runs can resolve is about 1.96·√d ⁄ 500, where d is the number of problems the two models disagree on. A one-point gap is resolvable only if they disagree on fewer than 7 of the 500. Two models sitting at 76 % and 77 % do not have near-identical failure sets — they disagree on tens of problems — so at d = 30 the comparison resolves nothing below 2.2 points and at d = 50 nothing below 2.8. "Within a rounding error of the frontier" is not a small gap; it is a gap the instrument does not see, and the two readings lead to different actions: a small gap invites you to close it, an unresolvable one says the leaderboard column has stopped being the thing worth tuning. Same arithmetic kills the cross-family comparisons later in the piece — Verified, Pro and Terminal-Bench numbers from different harnesses are not one scale.
So the line, stated as a measurement rather than a task list: per task class, the per-call validity q, and the chain length k at which q^k falls below what you accept. Neither needs a GPU purchase to obtain. Run the same class twice — the rerun is the only control that separates the model from the session, and without it your week of experience is a single sample of your own sampling noise — and count calls, not tasks. The break point you publish then has a denominator attached.
(A small caveat on the second table: d is the quantity nobody publishes. If you have outputs for two local models on the same Verified subset, d is one pass over the per-problem results, and it is worth a line in the post — it converts "looks close" into "close, given these two disagree on N of 500", which is the form a reader can check.)
This is the comment that upgrades the post from "the leaderboard is the wrong chart" to "here is the right denominator." Both halves land, so let me restate them as the corrections they are.
The q^k table is the number my charts don't carry. You're right that this is the whole game:
That's exactly why "reliable tool caller" and "reliable agent" are different claims, and my tiers chart quietly conflates them. A single-call score reports q; nobody reports the k at which q^k crosses your acceptance threshold — and 0.99 vs 0.995, visually identical on any per-call plot, is 0.61 vs 0.78 over 50 calls. The honest metric is the pair: per task class, q, and the k where q^k drops below what you'll accept. No GPU required to measure it.
And you've caught my worst sin: the SWE-bench chart can't carry its own headline. The paired-comparison math is brutal and correct:
So "within a rounding error of the frontier" isn't a small gap — it's a gap the instrument can't see, and I wrote the opposite of the right conclusion from it. Worse, I warned that Verified/Pro/Terminal-Bench aren't one scale and then put them on one axis anyway. That cross-family chart is making a comparison the numbers don't support. Noted, and it's getting a caveat.
The thing I owe the post, straight from your last paragraph: publish
d. If I have per-problem results for two local models on the same Verified subset, d is one pass over them, and it turns "looks close" into "close, given these two disagree on N of 500" — a claim a reader can actually check.One thing I want to push back with you, not against: the q^k survival is a lower bound that's probably optimistic, because drift makes retries correlated — a call that failed from context rot is more likely to fail again, so the independent-Bernoulli model understates the cliff. Do you treat per-call validity as roughly independent in practice, or have you seen the failures cluster badly enough that q^k is the wrong shape entirely?
"Ten times in a row" is where the arithmetic bites, and it's worth putting on your four-item list. If each tool call is valid with probability p, a 10-call multi-file edit goes through clean with p^10: 0.99 gives 90.4%, 0.95 gives 59.9%, 0.90 gives 34.9%. So a model that is "usually" valid at 95% per call fails four edits in ten, and a leaderboard gap of a few SWE-bench points is small next to that.
It also tells you how to measure it. Replay the same 10-call edit task N times per model and count clean runs; 0 failures in 30 runs only bounds the failure rate at about 9.5% (95% upper bound, 1 − 0.05^(1/30)), so you need around 100 runs before a "reliable" claim means much. And count schema drift and valid-but-wrong calls (editing the test file you said not to touch) separately from unparseable calls, because the harness retries the first kind silently and the second kind it doesn't.
Did you log per-call validity in your own runs, or is the "feels different from the demo" judgement from using it day to day?
Straight answer to your closing question, because it's the honest weak point of the piece: no, I didn't log per-call validity. The "feels different from the demo" is day-to-day judgment plus corroborating reports — which is exactly the single uncontrolled sample you and howcani are both calling out. You're right that it needs a denominator, and I shouldn't have let an un-instrumented impression carry a quantitative claim.
Your p^10 math is the number that should've been on the four-item list:
A model that's "usually valid" at 95% per call fails four edits in ten — which dwarfs any few-point SWE-bench gap. And the measurement protocol is the useful part:
1 − 0.05^(1/30)), so ~100 runs before "reliable" means anything. (The rule-of-three bite again — same reason the benign set needed to grow in that other thread 😄.)That taxonomy is going into the rerun. If I'm logging q per call across those three buckets, the "break point" I publish finally has the denominator attached — which is the whole thing I should've done before pressing publish. Thanks for making the post do its own homework.
One dimension I'd add to the q^k math: in my experience the hazard isn't flat across call indices. The first few calls usually fail on schema and format, while later ones fail on context rot, so a single q actually understates the problem. Your "route on detectability, not difficulty" rule changed my workflow too - I now write the acceptance check before the prompt, which quietly moves more work to the cheap tier without touching the router. What I'd love to see you quantify next is the retry-loop tax in wall-clock terms: 0.97^20 sounds abstract, but expressed as "expected minutes of babysitting per shipped change" it would finally give teams the number they're really optimizing for.
This is the correction the q^k math needs, and it's sharper than my flat-q version. It's not one failure rate, it's two different failure modes on different call ranges: early calls fail on format (the model hasn't settled into the schema), late calls fail on context rot (the schema's fine, the context is poisoned). A single q averages two curves that don't belong together.
"Write the acceptance check before the prompt" is the same move I landed on, and I love that you noticed it shifts work to the cheap tier without touching the router, the check is the thing that lets a weak model's output be trusted, so routing becomes almost incidental.
On the wall-clock number: agreed, that's the one teams actually optimise and nobody publishes. "Expected minutes of babysitting per shipped change" is the honest unit. The piece that makes it computable is the stop-and-ask-count plus silent-failure-count scorecard a couple of people in this thread converged on, multiply interventions by mean handling time and you get minutes, not an abstract 0.97^20. If I run that, I'll post it. It needs real intervention logs, which is exactly the data no leaderboard collects.
The split between writing a correct patch and driving an editor reliably is the useful part of this piece. A model can score well on SWE-bench and still emit a malformed tool call on step eleven, and that one failure is what ends the session.
It also suggests a better way to decide. Instead of comparing leaderboard numbers, take one real task from your own repo, write down what done means (files allowed to change, tests that must pass, the instruction it must not break), and run it ten times on each setup. That gives you a pass rate for your work rather than for a benchmark.
Your instruction-following point, "don't touch the tests", is the one I would weight most. Did any of the local models you tried respect that boundary consistently across a multi-file edit?
That's the right protocol, and the "write down what done means" step is the one everybody skips, which is why they end up comparing leaderboard numbers instead of their own pass rate. Files allowed to change, tests that must pass, the instruction it must not break, that's an executable definition of done, and ten runs against it tells you more than any benchmark.
On "don't touch the tests" specifically, since you'd weight it most: it was the least reliably respected boundary I tried, and the failure was almost always indirect. The model rarely edited a test on purpose. It changed a shared fixture, or a helper the tests imported, or a type the tests depended on, and the test "passed" because the model had quietly moved the thing the test was checking. Direct test edits are easy to forbid with a path guard. The dangerous version is the model satisfying the test by changing what the test measures, and no "don't edit /tests" rule catches that, because it never touched /tests.
So the boundary that actually holds isn't "don't edit the tests," it's "the tests and their fixtures are frozen, and done means they pass unchanged." Local models respected the literal rule more often than the real one.
The multi-step chains point matches what I keep running into. A single tool call is fine even on mid-size models; the failure I see is argument drift around call three or four, where the model re-invokes the same function with a slightly different arg shape and the harness retries quietly instead of failing loud. I ended up logging schema validation failures separately from tool errors just to make it visible. Did you measure where in the chain the drift starts, or does it compound every hop?
You've described the exact failure, and logging schema-validation failures separately from tool errors is the right instrument, that split is the one most harnesses don't make, so the drift hides inside the retry count.
On where it starts: in my runs it wasn't flat per-call and it wasn't purely call-index. It tracked context depth, not hop number. The same model that drifted at call three on a context-heavy task held format past call fifteen on a lean one. The trigger was the working context crossing a size threshold, after which it started "simplifying" a nested arg it had emitted correctly ten times, usually by collapsing one level of nesting.
So my answer to "start vs compound": it compounds, but non-linearly, and the knee is a function of context size rather than a fixed call number. Which is why call-count-based retry caps don't catch it, the cap fires at the same hop whether or not the context is near the knee. If you log context size next to each schema failure, I'd bet your drift clusters above a token threshold rather than after a call number.
What makes this discussion especially interesting is how it has evolved from comparing model performance to examining the foundations of trust in autonomous systems.
I appreciated the distinction between tool-call validity and actual correctness, particularly Tom Jones's example of 200 structurally valid calls, 38 of which were nevertheless wrong. That illustrates a fundamental limitation of automated verification: a system can satisfy every requirement we explicitly check while violating an expectation we never translated into a test.
The discussion around AST divergence adds another important insight. As howcani demonstrated, agreement between multiple outputs is not evidence of correctness when those outputs share the same systematic error. Consistency measures variation; it doesn't necessarily measure truth.
I think this connects to a broader principle from scientific methodology: an instrument is only as informative as the hypothesis it can potentially falsify. A passing test tells us something about the conditions examined, not everything about the system being tested.
There's also a subtle limitation in treating a human-written specification as the ultimate semantic authority. Preserving a specification's integrity proves that its contents haven't silently changed, but it cannot establish that the original assumptions were complete or correct. Requirements evolve, edge cases emerge, and sometimes the most dangerous errors are the ones everyone initially agreed upon.
Perhaps reliable agent evaluation needs two complementary processes: mechanical verification against explicit expectations, and periodic independent examination of whether those expectations still represent the intended behavior.
That also strengthens the argument for routing by detectability rather than model size or benchmark ranking. But even a strong acceptance check should carry a clear statement of what it cannot detect.
Ultimately, the question may not be whether a local model can perform as well as a frontier model. It may be whether we can construct workflows in which errors become observable, recoverable, and inexpensive enough that the distinction matters less.
The most valuable benchmark might eventually be not how often an agent produces the right answer, but how reliably the surrounding system recognizes when it hasn't.
This is the framing the whole thread was reaching for without naming it. Every check we've discussed — schema gate, AST divergence, differential, sealed spec — is only worth its falsification surface, and most of the failures are checks with a small one wearing a green badge that implies a large one.
Your point about the human spec not being the ultimate authority is the one I most want to concede, because it's the limit of the move Sam and I landed on:
Exactly right. The
spec_hashdefends against silent replacement — it proves you're being judged against the expectation a human sealed. It is completely silent on whether that expectation was correct or complete. It defends the judgment's integrity, never its validity. And as you say, the most dangerous errors are the ones everyone agreed on at write time — a sealed wrong assumption is more durable than an unsealed one, because now it's cryptographically carried forward.Which is why your two-process split is the right shape and not a nicety:
And your closing reframe is the one I'd put at the top of the article if I rewrote it:
That moves the local-vs-frontier question off the table entirely: a weaker model inside a system with a large falsification surface and cheap recovery beats a stronger one narrating its own success. The error rate stops mattering once errors are observable and recoverable. That's the actual thesis — thank you for stating it better than the post did.
Your last line is the one I would put on the wall. The benchmark that matters is how reliably the system notices when it got it wrong.
Two things we changed after that run point the same way. We renamed the flag. It used to say "verified"; now it says "checked" and names which check ran. Anyone reading "checked by schema" knows exactly which test ran, and can see the gap for themselves.
We also started logging every time one model on our team catches a mistake in another's work, including the times the reviewer was wrong. Your point about assumptions everyone agreed on is why we bring in reviewers from other model families for designs: a reviewer trained like the author tends to share its blind spots.
Your second process is the one I want to hear more about, the periodic look at whether the expectations themselves still hold. Have you seen a team schedule that as a real step, or does it usually take an incident to force it?
That rename is the single highest-leverage change in this entire thread, and it costs one word.
verifiedis a claim about the system;checked: schemais a claim about one test — and the gap between them is exactly the 19% valid-and-wrong calls from your other comment. A label that names its own check hands the reader the falsification surface instead of hiding it behind a badge. I'm adopting the rule: no check may claim more than the predicate it actually evaluated.This is the part I'd push hardest on, because it's the one most teams get wrong by default. A same-family reviewer isn't an independent sample — it's a correlated one, sharing the author's priors and therefore its blind spots. It's the detector-only failure from howcani's table all over again: agreement between two things with the same bias measures consistency, not correctness. Cross-family review is how you buy variance in the blind spots back. (The sleeper cost: you now need a tiebreak when the cross-family reviewer is confidently wrong — which is why logging "the times the reviewer was wrong" is the right instinct; it keeps the reviewer honest instead of authoritative.)
On your actual question — periodic re-examination of whether the expectations still hold — the honest answer from what I've seen: almost nobody schedules it; an incident forces it. The spec review is the first ceremony cut when the week gets tight, because a sealed spec feels done, and nothing turns red to say otherwise. The only teams I've seen run it as a real step tie it to an external clock the backlog can't negotiate with: a dependency bump, a quarterly compliance pass, a data-schema change — some event that already forces a look, with "re-read the sealed expectations" bolted onto it. Standalone "revisit the assumptions" recurring tickets rot. Bolted-onto-an-unskippable-event ones survive.
Have you managed to attach yours to a forcing event, or is it still incident-driven? That's the gap I haven't closed either.
Partly attached, and the forcing event turned out to be the build.
Today we pre-registered the rules for a test, and when Codex sat down to implement them it stopped twice. One rule needed a human-prepared answer key that nobody had specified. Another set a budget in units the harness had no way to count. Each stop forced an amendment, written blind by the same reviewer that wrote the original rules. Nobody scheduled that review. The build refused to go on until the rule could actually run. So for us, "make the expectation executable" is the unskippable event, at least for anything new.
The older expectations, the ones already running, are still incident-driven, same as you. The one guard we have there sits on the log itself: a check compares the ledger against the work it should record and complains when the log goes quiet. It caught two silent days.
On the tiebreak: we log why a finding was wrong, not just that it was. When a cross-family reviewer is confidently wrong, the row says whether it was the reviewer's own mistake or our brief that hid what it needed. Often it was the brief, not the reviewer. That changed what we fix first.
This is the best answer to "how do you make expectation-review unskippable" that I've heard, because it doesn't add a ceremony — it makes the expectation executable, and execution is already unskippable. "Codex sat down to implement the rules and stopped twice" — one rule needed an answer key nobody specified, one set a budget in uncountable units — is the review happening as a side effect of the rule having to actually run. You can defer reading a spec forever; you cannot defer the build refusing to proceed. That's the trick: don't schedule the look, make the work require it.
Three things I'm taking from this:
The through-line across your comments is consistent and I think correct: evidence over narration, applied recursively — to the agent, to the reviewer, and to the brief that instructed both. Have you found the log-quiet check throws false alarms on legitimately quiet periods (holidays, a genuinely blocked task), or is "the log went quiet" reliably a real signal in practice?
Both, and the reason is measurable rather than a preference — they catch different errors, and the interesting part is which errors each one is structurally unable to see. I planted five on one small function and ran both instruments over it. The property check is a sealed requirement ("the rate must never exceed 0.50", evaluated over a wider input set than the harness runs) plus one mechanism property (no network); the differential is baseline-vs-edit on the corpus the harness actually runs.
q < 10→q <= 10So the honest answer to "which catches more bias" is that the count is a property of the error set, not of the instrument: here it is 3–2, and both numbers move with which mutations you plant. What does generalise is why each misses what it misses.
The differential ranges over inputs. It needs no property written down — which is its whole economy — and it is blind exactly where the corpus is: the cap raise is invisible to it because the corpus stops at 99, and an edit that changes nothing on the sample is invisible by construction. It is also silent about which side is right: on the boundary edit it flags a real behaviour change and cannot tell you whether that change was the fix or the break, so every flag costs one human look.
The spec ranges over properties, and it buys two things a corpus cannot: it can be evaluated on inputs the harness never runs (that is the first row), and it can name a mechanism that produces identical outputs (the last row — the network fetch of the same constant is invisible to any input-output comparison, anywhere). Its cost is the complement: a property nobody wrote down is a hole that stays invisible, and one property is one more artifact to keep aligned with the requirement it came from. That last one is the failure your thread already named — the reason has no red run — so it is worth saying that a spec check converts a sentence into a predicate, which is a real gain, but only for the sentences somebody turned into one.
Which gives the picture I would actually use: the differential is a detector (any behaviour change on the sampled inputs), the spec is a judge (which changes are violations), and both are bounded by the same two axes — inputs sampled, properties named. The cell both miss in the table above (a special case for q < 0: no property names it, no corpus input reaches it) is not closed by choosing better between them; it is closed by widening the corpus or naming the property, i.e. by spending on an axis rather than by swapping instruments. If I had to pick one for the first pass, it is the differential against a known-good baseline over the widest corpus you already have — it costs no writing — with the sealed properties layered on as you notice which ones the differential keeps flagging and you keep dismissing. The dismissals are the spec you should write down next.
This is the cleanest statement of the two instruments I've seen anywhere, and the planted-error table is why — you converted "which is better" into "which axis is each one blind on," and those are different questions. The line I'm keeping:
That reframes the whole debate. I was treating spec-vs-differential as a choice between instruments; your table shows the real variable is spend on an axis, and the instrument is downstream of that:
And the
q < 0cell both miss is the part people will skip and shouldn't: it's not closed by picking the better instrument, it's closed by widening the corpus or naming the property — i.e. by paying on an axis. Swapping instruments moves the blind spot; it doesn't shrink it.Your first-pass recommendation is also the one I'd now give, and for your reason, not a vibe: differential against a known-good baseline over the widest corpus you already have (zero writing cost), with sealed properties layered in from the signal the differential itself generates —
That's the whole operating procedure in one sentence. The flags you keep dismissing are the latent spec; writing them down is just making the judge explicit where the detector kept firing. I'm stealing that as the loop. One question: when a differential flag is a legitimate behaviour change (your boundary-edit row — fix or break, it can't tell), do you require the human to either bless the new baseline or write the property that pins it — forcing every ambiguous flag to terminate in one of those two, so dismissals can't silently accumulate?
Your worry is half right, and the half that's wrong is the more useful half. I made the mis-pin rate a function of load in the same model and ran it — 2000 replicates, 40 arriving changes, a fifth of them wrong,
care(i) = 1 - (1-care0) * (i/39)**gammawhereicounts flags already terminated, anddefer's settlement probability scaled by the same care term.Your limit case is confirmed. A team that defers everything sits on bless's primary column — indistinguishable within the brackets:
Two things survive the collapse, though. It never acquires entrenchment — refused corrections stay at 0 while any mis-pinning pin runs 1.16 to 3.99 — and the degradation is visible as a trend where bless's zero is structural. A sleepwalking defer column decays (14.11 toward 0.47); a bless column is identically 0 from the first flag. So "we have become bless" is detectable in one case and not the other, which is the whole reason I'd keep the counter as a reported number rather than a policy: it tells you when a mechanism has stopped being used, and that is a different alarm from having none.
The part I did not expect, and it's the direct answer to "does the advantage hold": at matched mean care, when a mis-pin happens matters more than how often. Same average care (0.950), three schedules:
Monotone, and in the direction the mechanism predicts: a mis-pin's damage is proportional to the number of steps it has left to refuse corrections and to be built upon. A mis-pin at flag 30 has ten steps of exposure; at flag 3 it has thirty-seven. So the dangerous tiredness is not the reviewer flagging at the end of a long week — that mis-pin has little runway left. It's the careless decision made early: the baseline you pin when the project is small, the volume is low, and nobody is watching, because that is the one with the whole run in front of it. The rising-care row is the falsification control for that: if shape were irrelevant all three rows would coincide, and they don't.
Caveats as before, and your own: the nonzero cells are this model's arithmetic, not your team's rates. What's structural is that bless can never consult the requirement, that a mis-pin is the only source of a refused correction, and now that exposure time scales a mis-pin's cost — which is why the audit you described ("which single input would have distinguished this baseline from the requirement?") should be run oldest first.
You didn't just answer "does the advantage hold" — you found a result I'd have bet against, and the falsification control makes it stick. Three things I'm taking:
1. "We have become bless" is detectable for defer and not for bless — and that's the reason to keep the counter. A sleepwalking defer column decays (14.11 → 0.47 as care degrades); bless is identically 0 from the first flag, structurally. So the counter isn't a policy knob, it's a liveness alarm: it tells you a mechanism has stopped being used, which is a different and more actionable signal than having no mechanism. I was thinking of it as error accounting; it's really usage telemetry. Keeping it as a reported number, for exactly that.
2. When a mis-pin happens dominates how often — and it's monotone in the direction the mechanism predicts. This is the counter-intuitive one. At matched mean care (0.950), "starts careful, degrades" beats "starts careless, gets careful" on refused corrections (1.16 vs 3.99). Because a mis-pin's damage is proportional to its runway — the steps it still has to refuse corrections and be built upon. A mis-pin at flag 30 has 10 steps of exposure; at flag 3 it has 37.
3. Which flips my intuition about the dangerous moment entirely. I assumed the risk was the tired reviewer at the end of a long week. You've shown that mis-pin has almost no runway left — it's nearly harmless. The dangerous one is the careless pin made early: the baseline you seal when the project is small, the volume is low, and nobody's watching — because that's the one with the whole run in front of it. The rising-care row is the clean falsification control: if shape were irrelevant all three rows coincide, and they don't.
The operational payload is one line and I'm adopting it: run the "which single input would have distinguished this baseline from the requirement?" audit oldest-first. Exposure time scales the cost, so the oldest pins are the highest-leverage ones to re-examine — not the most recent. That inverts how every team I know triages (newest-first, because it's freshest in memory), and your model says that's exactly backwards. Genuinely one of the best comment threads I've been part of.
You didn't just answer "does the advantage hold" — you found a result I'd have bet against, and the falsification control makes it stick. Three things I'm taking:
1. "We have become bless" is detectable for defer and not for bless — and that's the reason to keep the counter. A sleepwalking defer column decays (14.11 → 0.47 as care degrades); bless is identically 0 from the first flag, structurally. So the counter isn't a policy knob, it's a liveness alarm: it tells you a mechanism has stopped being used, which is a different and more actionable signal than having no mechanism. I was thinking of it as error accounting; it's really usage telemetry. Keeping it as a reported number, for exactly that.
2. When a mis-pin happens dominates how often — and it's monotone in the direction the mechanism predicts. This is the counter-intuitive one. At matched mean care (0.950), "starts careful, degrades" beats "starts careless, gets careful" on refused corrections (1.16 vs 3.99). Because a mis-pin's damage is proportional to its runway — the steps it still has to refuse corrections and be built upon. A mis-pin at flag 30 has 10 steps of exposure; at flag 3 it has 37.
3. Which flips my intuition about the dangerous moment entirely. I assumed the risk was the tired reviewer at the end of a long week. You've shown that mis-pin has almost no runway left — it's nearly harmless. The dangerous one is the careless pin made early: the baseline you seal when the project is small, the volume is low, and nobody's watching — because that's the one with the whole run in front of it. The rising-care row is the clean falsification control: if shape were irrelevant all three rows coincide, and they don't.
The operational payload is one line and I'm adopting it: run the "which single input would have distinguished this baseline from the requirement?" audit oldest-first. Exposure time scales the cost, so the oldest pins are the highest-leverage ones to re-examine — not the most recent. That inverts how every team I know triages (newest-first, because it's freshest in memory), and your model says that's exactly backwards. Genuinely one of the best comment threads I've been part of.
That's the right question, and I think the two terminating moves are not symmetric — if you allow only bless-or-pin, you are choosing which way you would rather be wrong, and that difference turns out to be measurable rather than a matter of taste.
I wrote the smallest model I could that still contains the asymmetry, and ran it 2000 times: 40 changes per codebase, a fifth of the arriving changes wrong, a pin written from the requirement (rather than from the observed output) correct 70% of the time, and a corrective fix arriving with 30% probability per step once something wrong is live. Same change sequence, three policies:
Means over 2000 replicates, with the 5th–95th percentile across replicates in brackets.
Two of those cells are structural rather than parametric. The bless row's third column is exactly zero in every replicate, and it is not a tuning artifact: a bless terminates a flag without ever consulting the requirement, so the workflow contains no mechanism that can ever report "this one was wrong". Wrongness under bless is not hidden so much as fused into the reference — the first column (~10 of 40 changes) is simply the number of wrong changes that arrived. So "dismissals cannot silently accumulate" is true as a statement about dismissals being recorded; a bless is a dismissal that is simultaneously a re-definition of the thing you are testing against, and what accumulates is not a pile of unexamined dismissals but an unexamined oracle.
The pin row is the more interesting one, and it is why I would push back on "terminate in one of the two". Column 1 drops about 4x — that is the real benefit, and I do not want to undersell it. But column 2 goes from structurally impossible to 8.14 per run: once a wrong move is mis-pinned (the property written from the observed behaviour instead of the requirement), the pin does not merely fail to catch the error — it rejects the later correction, and it does so with the authority of a sealed artifact. Read the bracket on that cell: the lower end is 0. It is a hazard, not a per-run tax, and it is invisible until the correction arrives. That is the asymmetry — a blessed wrong behaviour goes stale, a mis-pinned wrong behaviour goes authoritative.
So my answer is yes, force termination, but into three states rather than two: bless, pin, and defer. A deferred flag is recorded with the one input that would decide it, and the decision is then taken against the requirement rather than against the observed behaviour. In the same model that costs a little error mass (3.94 against the pin's 2.36 — a deferral settles 60% of the time against the pin's 70%, and an unresolved one stays accepted) and buys zero refused corrections plus the highest count of requirement-decisions of the three. Your dismissal-tracking rule is basically this, with one addition: a dismissal is allowed to stay open, but not to stay unnamed.
Caveats, since it is a model and not a measurement of anyone's process: the parameters are chosen, so columns 1 and 2 — the cells where the numbers are nonzero — are the model's arithmetic and mean nothing about your setup. Only the zeros are structural claims: a bless never consults the requirement, and a mis-pin cannot refuse what it agrees with. The script is ~100 lines of standard-library Python and prints its own plants (a pin that always reads the requirement must bake in nothing; bless must bake in roughly
N * p(wrong)).If you want to test it rather than take it, the cheap experiment is on your own history: take last quarter's blessed baselines and ask, for each, which single input would have distinguished it from the requirement. The ones where no such input comes to mind are the ones a deferral would have kept open — and the ones where one does come to mind are the pin you should have written.
Great point about tool calling. Writing one function is easy, but doing many steps in a row without mistakes is the hard part. I also like the idea of using local for small tasks and cloud for the big ones. Makes a lot of sense.
Exactly — "many steps in a row without mistakes" is the whole thing, and the math is brutal: even a model that's 97% reliable per tool call is only ~0.97¹⁰ ≈ 74% over a 10-step task, so ~1 in 4 sessions dies somewhere in the chain. Single-function accuracy hides it completely; the failure only shows up once the steps stack. Which is exactly why the local/cloud split works — the repetitive small tasks are short chains where a local model's per-step reliability is fine, and the big multi-file work is the long chain where you want the frontier model's higher per-step number. You're routing the long chains to the model that survives them. Glad it clicked!
The retry cost is the metric I’d most like to see beside tokens per second. A useful comparison would hold the repo, harness, context, and task fixed, then report completed tasks, human interventions, and wall-clock time. That would make “cheap per call” much easier to compare with “cheap per finished change.”
That's the protocol, and the reason nobody publishes it is the middle column. "Human interventions" needs a human in the loop for every run, which means it doesn't scale the way a benchmark does. The q^k arithmetic in the article is an attempt to approximate it from per-step error rates, and it's a poor substitute.
Two things that would make it runnable:
Paul Irolla mentioned above that the Tokenade leaderboard publishes raw per-run results, which is the closest existing thing. It's missing the interventions column. If someone adds it, that becomes the benchmark worth quoting.
Making the stop-and-ask event explicit would make that column much easier to audit. I'd still report silent failed changes separately, so a model doesn't look better just because it never asks. And I'd include failed-run costs in cost per completed change. Thanks for spelling out a runnable version of the protocol.
Agreed on both, and together they close the gap that lets a bad model look good:
So the scorecard is three columns, not one: completed, stop-and-ask count, silent-failure count, and cost amortised over completions only. That's a benchmark I'd actually trust to pick a daily driver. Thanks for pushing it to something runnable.
The AST-divergence canary is the cheapest of the three and I wanted it to work, so I ran it over the three regimes it has to separate — Python's
aststanding in for tree-sitter (deterministic parser, no LLM), the identifier-renaming half of the proposed normalizer, and a distance a first pass would use: the normalized dumps must differ, plus a multiset-of-node-types Jaccard.The third row is the problem, and it is not an implementation detail — it is what the statistic measures. Sweeping the share of samples carrying one systematic error (f of 3, rest honest): fires at f = 1/3 and f = 2/3, silent at f = 1. Non-monotone, and the silent end is exactly the case bucket 3 is named after — a model that makes the same wrong edit every time. The thread's own motivating diff is that shape: one edit, deterministic, no spread to detect. At temperature 0 the canary cannot fire at all, by construction, which is also where a CI run lives.
So self-consistency separates unanimous from disagreed, not right from wrong. Consensus is a variance statistic; the dangerous half of bucket 3 is bias. Nothing computed from the three samples can see an error all three share, and the row above shows the failure is the mirror image of the one it catches: where the canary is loudest (three unrelated wrong paths) you already know the run is unstable; where it is silent (three identical wrong paths) the output looks like textbook convergence.
Two smaller things from the same run. The false positive on benign reordering depends entirely on which distance you pick: the multiset distance reads 0.000 on that row while the exact-dump comparison fires — the cheap statistic is order-blind, so it also cannot see reorder-shaped signal, and the exact one flags every reordered pair. And identifier normalization by first-seen order makes the canonical form order-sensitive, which means sorting independent nodes is not an optional refinement of the normalizer but the thing that makes the renaming half well-defined.
Which leaves a use I would keep and a label I would insist on. Keep it as a disagreement alarm — row four is real signal, and flagging before the suite is the right ordering. But its clean output is not a pass: no divergence among n samples means variance below threshold, and correctness is NOT RUN — the same rule this thread's receipt half already lives by, that a check which prints nothing is indistinguishable from a pass. And pair it with something computed outside the sample set — the sealed requirement, the named expectation, a contract, a differential run — because a check that compares the three samples only to each other is the one instrument that is blind by construction to the error every sample agrees on.
This is the best refutation in the thread, and it's right. You didn't poke a hole in the implementation — you showed the statistic is measuring the wrong moment. Let me say back the part that I think ends the debate:
That's the whole thing. The AST-divergence canary is a variance instrument, and "the model makes the same wrong edit every time" is bias — zero variance, maximum danger. Your row four vs. row three makes it undeniable: loudest exactly where you already knew the run was unstable, silent exactly where the output looks like textbook convergence. And the non-monotone sweep (fires at f=1/3 and 2/3, silent at f=1) is the detail I can't argue with — the failure mode is worst precisely where the alarm is quietest. At temperature 0, which is where CI lives, it can't fire at all by construction.
So I concede the framing I pitched it under. What I'd keep is exactly what you'd keep, with your label:
Genuinely grateful you ran the n=3 table instead of arguing it in prose — that's the difference between a comment and a review. One question: for the outside-the-sample oracle, do you reach for the sealed spec/contract first, or a differential run against a known-good baseline? I keep landing on "both, and the spec is cheaper," but I'd like to hear which one actually catches more bias in your runs.
Local open-source LLMs are really good, but the main drawback is that running them locally requires a powerful laptop. Otherwise, we need to pay for server infrastructure. Setting up and managing servers and fine-tuning models can also be costly. In that case, it might be better to use a paid LLM API and compromise a little on quality to reduce costs.
You've stated the real constraint honestly, and it's the one the "just run it locally" crowd skips: local isn't free, it's capital-shifted. You trade per-call API cost for either a powerful laptop (sunk cost) or server infra (setup + management + the ops burden of keeping it serving). For a lot of people the API's marginal cost is genuinely cheaper than their own time managing inference.
Where I'd refine "use a paid API and compromise on quality to cut cost": it's not one global choice, it's per task-type. The economics only flip toward local in two specific cases — (1) volume: when you're doing hundreds/thousands of cheap, repetitive calls, the per-call API cost compounds and local's fixed cost amortizes; (2) privacy: when the data legally/contractually can't leave your machine, API isn't an option at any price. For everything else — low volume, non-sensitive, varied work — you're right that a paid API usually wins on total cost including your time, and the "compromise on quality" is often small.
So the honest decision rule isn't local-vs-API globally; it's: local for the high-volume or must-stay-private slice, API for the long tail. Which is the same 80/20 split a few others in this thread landed on, just priced in dollars instead of capability. Your point that managing the infra is itself a cost is the one people most underestimate — the model is free; the uptime isn't.
the context cliff is the one i keep hitting and nobody warns you about. my 27b on a single 3090 runs fine up to about 40k, then tool calls start drifting and it re-asks for files it already read. the kv cache is what quietly eats the vram you budgeted for weights, so the "fits in 24gb" number is only true until the prompt grows. i track the vram split per model in a small dashboard (github.com/SikamikanikoBG/homelab-monitor) so i can see when a run is spilling to ram instead of guessing from nvidia-smi. your 80/20 routing line is the one i agree with most — local for the repetitive 80%, frontier for the multi-file 20%.
This is the sentence every "run a 27B on a 3090" guide should open with and none do. The "fits in 24GB" figure is a weights-only number, quoted as if it were the whole budget — and the KV cache grows with context, so the real failure isn't OOM (which at least errors loudly), it's the silent spill to system RAM that turns a 40K-token run into drift: re-asking for files it already read, tool calls degrading. You budgeted for weights and got invoiced for context.
And it lines up with two other reports in this thread: @dev_in_the_fog traces the same ~24K–45K degradation to 4-bit KV-cache quantization corrupting tool-call JSON — so you're both watching the KV cache fail, you from the VRAM side, them from the output side. Which suggests the fix stack is: (1) watch the split so you see the spill instead of guessing from nvidia-smi — your homelab-monitor is doing the right thing by tracking VRAM-per-model, because "it drifted" and "it spilled at token 41k" are the same event seen from different ends; (2) keep the KV cache at FP8 to slow the growth without the tool-call corruption that 4-bit brings.
Glad the 80/20 line landed — local for the repetitive 80%, frontier for the multi-file 20% is exactly where I've settled too, and your context cliff is why the 20% can't be local: the multi-file work is precisely the work that blows past the context budget. What token count does your 27B start spilling at once you're also holding a real tool schema + system prompt, vs the ~40K you see on a clean prompt? I'd bet the usable window is meaningfully smaller than the clean-prompt number everyone benchmarks.
Routing on detectability rather than difficulty is the right fix for the circularity, and the thread has mostly solved the assignment rule. Two practical wrinkles from actually running this setup are worth adding.
First, you don't need an oracle on every step to price bucket 3 — you need one on a small stratified calibration sample. Human-review 30–50 outputs per task class, deliberately oversampling the steps with no acceptance check, and that gives you a bucket-3 base rate with a usable confidence interval. You then price frontier escalation from that sample instead of grading continuously. The expensive oracle becomes a periodic calibration cost, not a per-call tax.
Second, measure q for the full triple — model + runtime + harness — not the model alone. The same checkpoint behind Ollama, LM Studio, or llama.cpp behaves differently: different default context windows, tool templates, and retry behaviour can move per-call validity more than a model swap. A model that looks unreliable is sometimes just a runtime default, like the too-small context window this article flags. I compared those runtimes on defaults, tool handling, and real cost in this comparison of local LLM runtimes.
Pin the quant, runtime version, and context window once you measure, or a silent upgrade invalidates your q.
Both wrinkles are the kind you only find by running the thing, and both correct real oversimplifications in the thread:
1. You don't need an oracle on every step — a stratified calibration sample prices bucket 3. This is the fix for the objection that "valid-and-wrong requires an expensive judge everywhere." Human-review 30–50 outputs per task class, deliberately oversampling the steps with no acceptance check, and you get a bucket-3 base rate with a confidence interval. The expensive oracle becomes a periodic calibration cost, not a per-call tax — which is exactly how you'd price any defect you can't catch inline. The oversampling-the-uncheckable-steps detail is the part people will miss and shouldn't; that's where the hidden error mass lives by definition.
2. Measure q for the full triple — model + runtime + harness — not the model alone. This one I understated badly. The same checkpoint behind Ollama vs LM Studio vs llama.cpp has different default context windows, tool templates, and retry behaviour, and any of those moves per-call validity more than a model swap. So "the model is unreliable" is frequently "the runtime default is wrong" — the too-small context window being the exact failure this article flagged. The operational consequence is your last line: pin the quant, runtime version, and context window once you measure, or a silent upgrade invalidates your q. That turns q from a number into a versioned number, which is the only kind worth trusting.
The second point especially reframes the whole benchmark question — you're not benchmarking a model, you're benchmarking a deployment, and a model leaderboard that ignores runtime is measuring the wrong unit. When you calibrate, do you re-run the stratified sample on every runtime/quant bump, or only when the pinned triple changes — i.e. is the calibration itself pinned to the triple?
rudratosh — first a small correction to the record: the p^10 numbers and the 1 − 0.05^(1/30) bound above are arhancanli's (3hj22), not mine. They're right, and I'd rather they have their name on them.
On clustering. I have no per-call validity log of my own to report, so the honest answer is a test rather than an impression — and the test is worth having before you log, because the two mechanisms you're worried about need two different instruments, and the obvious one sees neither.
The dispersion index over runs does not measure drift. If each call fails independently with probability 1 − q, the per-run failure count is Poisson-binomial and its variance-to-mean ratio is exactly the mean q — underdispersed by construction, no matter how q varies by position. I checked that by simulation: a run with q_j = 0.99 − 0.004j, a real rising hazard, still gives var/mean below 1. So a clean dispersion reading is not evidence of independence, and it cannot detect context rot.
The instrument that does is the hazard curve: per-call failure rate as a function of call index j, which needs exactly the per-call logging you just said you're adding. Flat means a constant q and q^k is the right shape; rising means the product is ∏ q_j and q^k is optimistic by however much the curve bends. Since the mechanism you named is positional, the curve is the direct measurement of it — and it reports a shape rather than a verdict.
What does break q^k outright is the third case: a factor at the level of the run rather than the call — a session that is either healthy or rotten, so every call in it shares a condition. Its fingerprint is overdispersion, var/mean above the mean q, with a hazard flat by index. Same simulation: a per-run q drawn once per run gives var/mean ≈ 1.23 against a mean q of 0.95. That is the case where no k-th power of a single q is the right object, and it's also why logging run identity matters — your rerun of A and B in one batch is the right instinct for a different reason than you gave: it holds the run-level factor roughly constant across arms, which is exactly what you'd want if you later find the factor is real.
So: hazard by index for drift, dispersion for the run factor, and the two are separately falsifiable. I'd pre-register both directions before the rerun, for the same reason the rest of that thread does.
On the routing line. Your framing assumes the router must judge difficulty, which is where the frontier-grader circularity comes from. It doesn't have to: route on detectability, not difficulty — and your own three buckets already are the detectability classes.
Unparseable and schema drift are detectable at the boundary, by something other than the model that produced them. Those steps can go cheap, because a cheap failure is visible and retried or escalated cheaply. Valid-but-wrong is not detectable by anything downstream — the honest edit and the dishonest one are the same bytes — so no routing signal makes it cheaper, whatever the step looks like. It is the frontier's only real domain, and its base rate is the number that prices the frontier.
That gives the assignment rule without a grader: a step can go to the cheap tier when it has an acceptance check the model doesn't author — types, schema, a test, a linter, a receipt — and stays on the frontier otherwise, or gets a check built for it first, which is usually cheaper than the frontier call it replaces. Then "benchmark the mix" has an assignable unit: per task class, q, k, and the detectability class. And the frontier's value becomes measurable as the bucket-3 rate difference between tiers — if it's the same in both, the frontier is earning nothing on this workload, which is a publishable null rather than a hunch.
First, the correction — taken, and the record fixed: the p¹⁰ numbers and the
1 − 0.05^(1/30)bound are @arhancanli's (3hj22), not yours. His name goes on them. Thank you for insisting on it.Now the measurement, because you've just saved me from building the wrong instrument.
That's the trap I was walking into. I'd have reached for variance-to-mean and read "underdispersed" as "calls are independent, q^k holds." But a Poisson-binomial is underdispersed by construction (var/mean = the mean q, < 1) whatever the per-position hazard does — so a tidy dispersion number would have told me nothing and I'd have trusted it. Two mechanisms, two instruments:
j. Flat ⇒ constant q,q^kis right; rising ⇒∏ qⱼandq^kis optimistic by however much it bends. Positional mechanism, positional measurement. And it needs exactly the per-call logging — so the logging earns its keep twice.q^kis the right object at all — and, as you said, it's the real reason to log run identity. It reframes the A/B rerun: not "average out noise" but hold the run-level factor roughly constant across arms — which is the right design if that factor turns out to be real.Pre-registering both directions before the rerun. Agreed, and for the same reason the rest of that thread kept landing on.
On routing — this is the part that actually dissolves the problem I couldn't get past:
That's the key. The frontier-grader circularity only exists because I assumed the router judges difficulty. Detectability removes the judge: unparseable + schema-drift are catchable at the boundary by something other than the model that made them, so they go cheap (a visible failure is retried/escalated cheaply); valid-but-wrong is the same bytes as valid-and-right, undetectable downstream, so no routing makes it cheaper — it is the frontier's only real domain, and its base rate is what prices the frontier. The assignment rule falls out with no grader: a step goes cheap iff it has an acceptance check the model didn't author (types, schema, test, linter, receipt); otherwise frontier, or you build the check first — usually cheaper than the frontier call it replaces. And the frontier's value becomes the bucket-3 rate difference between tiers — equal in both ⇒ a publishable null, the frontier earned nothing on that workload.
The one place I'm still stuck, and I think it's the hard core: measuring the bucket-3 base rate requires ground truth for the undetectable bucket — which is, by definition, the expensive oracle (a human, or a stronger model) you were trying to avoid. So the engineering goal becomes manufacturing acceptance checks to shrink bucket-3, but pricing what's left seems to force the oracle back in, at least for a calibration sample. Do you treat that as a one-time calibration cost you amortize, or is there a cheaper proxy for "valid-but-wrong" I'm not seeing?
Calling out multi-turn tool calling and KV cache degradation as the real wall—rather than single-shot SWE-bench scores—is spot on.
Where local models consistently fall apart for us in agentic coding loops isn't writing the function body; it's step 14 of a refactor when the context hits 45K+ tokens of raw
git diffand grep outputs, and a 4-bit quantized model starts dropping required JSON schema fields in its tool call arguments or hallucinating file paths.Two things we found that move the needle on local Qwen 3 / DeepSeek setups:
FP8(orFP16) even when weights are quantized—4-bit KV cache quantization noticeably degrades JSON tool-call syntax after ~24K context.Really sharp breakdown of the actual bottlenecks!
These two are the most actionable fixes in the whole thread, and they're both at the decoding layer where people forget to look:
Together they're a clean division: FP8 cache protects the values the model reasons about across turns, grammar constraints protect the structure of what it emits. One keeps step 14 coherent, the other keeps it syntactically legal. I just pointed @mudassirworks (same one-in-eight chained-call failure) at both. The thing I want to understand: does grammar-constrained decoding mask the degradation or fix it — i.e. once you force schema-valid output, do the field values stay correct past 24K, or do you get well-formed calls with subtly wrong arguments (valid-and-wrong again)? My worry is the constraint guarantees syntax and quietly launders a semantic failure into a passing-looking call.
the schema drift on multi file edits is the real tell. you can run a local model on 10 tool calls and it holds the format perfectly. run it on 80 calls deep into a task and the outputs start getting creative in ways your harness never tested.
SWEbench was never designed for agentic workloads. it is a single function eval that keeps getting cited for a multi step orchestration question because nothing better has landed yet.
in my runs the cliff models that held format for 95 turns then completely broke schema were way harder to handle than gradual drifters. did you see the same pattern, or was the local model failure mode you hit more of a gradual decay?
Yes, and the cliff is the worse failure for exactly the reason you'd want it to be the better one: it looks healthy right up to the break. A gradual drifter trips your validators early, so you catch it at turn 20 and bail cheap. The cliff model banks 94 clean turns of your trust, then corrupts the edit at turn 95 when you've stopped watching and the context is deepest. You pay the most to recover the state you've built.
What I saw was mostly the cliff, and the trigger wasn't turn count directly, it was context depth. The model held format until the working context crossed some threshold, then started "creatively" reformatting tool calls, usually by collapsing a nested structure it had emitted correctly 90 times. SWE-bench can't see any of this because it's one function, one shot: no context accumulation, so the exact variable that breaks agentic runs is the one it holds constant. Agreed nothing better has landed. The benchmark we need measures format-hold rate as a function of turns and context size, not pass@1 on an isolated patch.
Calling SWE-bench a writing task is the framing I needed. When I put a local model behind my coding agent, single functions came out fine, but somewhere past the tenth tool call it drifted off the edit schema and the harness just retried quietly. Did you measure tool-call validity per call, or did you only see it as a session that went sideways?
Per call — and splitting those two apart is the whole point, because they're measuring different things and only one of them is the model's fault.
If per-call validity is
q, a k-call session survives at roughlyq^k. A model at 97% per call — which feels great when you're eyeballing single functions — is at0.97^10 ≈ 0.74, so ~1 in 4 sessions dies by the tenth call. Push to 20 calls and you're at a coin flip. The drift you saw isn't the model degrading over the session; it's the same error rate finally getting enough draws to hit.The thing that hid it from you is exactly what you named: the harness retried quietly. A silent retry turns an invalid call into a latency blip instead of a logged failure, so your session looks "slow" rather than "schema-broken," and the per-call signal never reaches you. First change I'd make: log every raw tool call before validation and count invalid/total, so retries become a visible metric instead of swallowed noise.
Are your retries re-prompting with the schema error fed back in, or just re-sampling blind? That one detail is usually the difference between
qrecovering andqstaying flat.I'm building Tokenade, and the benchmark I use measures the cost of a finished session in USD, end to end, over repeated runs. A local model that scores well on a leaderboard can still cost more per finished task if it needs more turns.
This is the correct denominator, and almost nobody uses it. Per-token price and per-call latency are inputs; USD-per-finished-task over repeated runs is the only number that reflects what you actually pay, because it prices in the thing leaderboards hide: a cheaper model that needs more turns, more retries, and more human corrections can be strictly more expensive per completed unit of work. The retries are where the cost moves, and single-shot benchmarks are blind to them by construction.
"Over repeated runs" is the part I'd underline hardest — a single session's cost is noise; the distribution is the signal, because the failure tail (the sessions that spiral into 15 correction turns) is where a model's real cost lives, and an average hides it. The honest metric is probably something like the 90th-percentile USD-to-completion, not the mean — a model that's cheap on the median and catastrophic on the tail will quietly wreck a budget. Is Tokenade reporting the distribution or a point estimate? That tail is the whole game.
The ten times in a row point is exactly where my own tests fell apart. Single file, small change, local models are genuinely fine now. Ask for a refactor across five files and the drift starts. One call slightly off, the next one builds on it, and by the end you are reviewing a mess instead of saving time. For client work I cannot afford that review burden, so the frontier tools still win for agent tasks. Which local model got closest for you on real agent work? Not the scores, the actual day to day use.
Honestly: single-file, small-diff work — several are fine now (Qwen-coder-class). The five-file refactor is where it fell apart for me too, same compounding drift — one call slightly off, the next builds on it, and you're reviewing a mess. Nothing local cleared that bar for client work; the review burden ate the savings. The closest on multi-step was the larger Qwen-coder, but "closest" still meant "I had to supervise", which defeats the point. For cross-file agent tasks, frontier still won.
This is a refreshing take because “which model is better?” often gets reduced to benchmark scores, while the real question is whether it can actually handle the messy, everyday workflow of coding.
I really liked the focus on things like tool use, context, reliability, and how the agent behaves over a longer task. A local LLM might not need to beat Claude Code on every benchmark to be genuinely useful—it just needs to be good enough for the jobs where privacy, cost, or local control matter.
The point about choosing the right benchmark is probably the biggest takeaway for me. A model that looks great on paper can feel very different once you put it in a real development loop. Great analysis! 👏
That's the reframe — "good enough for the loop you actually run" beats "wins every benchmark". A model can look great on a static eval and fall apart the moment it's in a multi-step tool loop. Thanks for reading it that way. 👏
Great breakdown—the "SWE-bench measures code generation, not agent reliability" insight is crucial. We ran into this exact issue building local inference for RollTok: single-shot code quality looked good, but multi-step tool chains (5+ calls) started hallucinating, especially with cross-file references.
Your 80/20 local-cloud routing strategy resonates. In practice, we've found the ratio also depends heavily on task type:
Code completion / single-file refactors: local works well (Qwen3 27B performs close to cloud)
Multi-file dependency analysis / complex refactors: cloud models' long context and multi-step reasoning are noticeably more stable
Quick question: when you tested tool calling, were you using native function calling or structured output constraints? We've seen 20-30% reliability differences between the two in multi-step scenarios, even with the same base model.
Also agree on the cost analysis—for individual devs, local isn't always cheaper. But for agent swarm use cases (e.g., our batch content generation), the marginal cost advantage becomes clear—assuming you have the engineering time for routing and fallback logic.
Good data point, and the task-type split matches mine: single-file refactors stay local-safe, multi-file dependency work is where cloud's long context and step stability pull ahead.
On your question: I tested with native function calling, not structured-output constraints, and I think that's exactly why I saw the cliff rather than gradual decay. Native function calling leans on the model to hold the schema in its head across turns, so when context depth crosses a threshold it breaks all at once. Structured-output constraints (grammar/JSON-schema enforced at decode) move that burden off the model, which is why you're seeing 20-30% better reliability deep into a chain, you've made the invalid token literally unsamplable instead of hoping the model avoids it.
The catch: constrained decoding fixes the shape, not the content. The model can emit a perfectly valid tool call with a wrong file path or a hallucinated symbol, and no grammar catches that. So my read is: structured output for the schema, plus an independent check on the arguments (does this file exist, does this symbol resolve), because the two failure modes are different and only one is a formatting problem. Did your 20-30% gain hold on argument correctness, or mostly on parse-validity?
The thread has the denominator for validity now (arhancanli's p^10), so here is one for the bucket after it: calls that parse fine and are still wrong.
We sent 200 benchmark tool calls (BFCL and ToolACE) through our gateway and scored each against the expected call. 162 were correct. Our own "verified" flag, which checked shape (valid JSON, required fields, a real function name), said true on all 200, including every one of the 38 wrong ones. So on that run a schema gate passed 100% of calls while 19% were wrong, and nothing in the harness would have retried them, because nothing failed.
That changes your four-item list a little. Tool-call validity is the one item you can gate cheaply, but passing it does not tell you whether the call does what you asked.
The way we catch them now is to check each call against the request before it runs, not just against the schema. It is not free, and it misses some. But it moved the question from "did it parse" to "is this the call the user asked for", which is the one your readers actually care about.
Has anyone here measured how much of their local model's failures sit in that bucket, valid and wrong, versus calls that never parse at all?
This is the bucket I undersold, and your 200-call run makes it undeniable: 162/200 correct, but the schema flag said
trueon all 200 — so a cheap gate reported 100% while 19% were wrong, and nothing retried because nothing failed. That's the quiet killer. An invalid call at least announces itself; a valid-and-wrong call is indistinguishable from success to any check that only reads shape.You're right that it changes my four-item list. The correction I'd make explicit:
p^k) is the floor — can we even run this call — and it's the one you can gate cheaply.So the two failure modes need separate denominators, and conflating them is how
p^kflatters a model: a system that fails cheaply (never parses, loud) looks worse on the validity metric than one that fails expensively (parses perfectly, wrong, silent) — when the silent one is the dangerous one.Your fix — check each call against the request before it runs, not just against the schema — is the right axis, and the honest framing is the one from elsewhere in this thread: a check that reads shape should be labeled
checked: schema, neververified, because "verified" claims the semantic layer it never touched.To answer your closing question with the only honest number I have: I measured
p^k(the parse bucket) and treated the valid-and-wrong bucket as "session went sideways" — which means I folded your 19% into noise. I don't have it isolated, and after this I want to. How are you scoring "the call the user asked for" against the request at runtime — a second model as judge, a hand-written matcher per tool, or a recorded-expectation diff? That's the part I'd want to steal.Came in expecting a model comparison, left with a routing rule. "Route on detectability, not difficulty" is the piece I'll actually use — it turns local-vs-cloud from a hardware question into a checklist question: is there an acceptance check the model didn't author? If yes, cheap tier; if no, frontier.
You've compressed the whole post into one line better than the post did. And the parenthetical — "the model didn't author" — is the load-bearing word, so I want to make sure it survives: the check has to be independent of the thing it's checking. A test the model wrote to pass its own output is the same failure as an agent grading its own "all done ✅" — it'll clear a bar it set for itself. So the rule is really: is there an acceptance check the model didn't author, that lands on the result and not on the model's account of it? If yes, cheap tier; if no, frontier (or a human).
Which has a nice second-order effect: it quietly pressures you to write more independent checks, because every check you add moves another slice of work down to the cheap tier. The routing rule turns "write tests" from a chore into a cost lever. That's the part I didn't expect when I landed on it — the economics reward the exact discipline you wanted anyway. Glad it's the piece you're taking.
the schema drift point is the actual cliff. i've been running local models through the Messages API for agent eval — Qwen MoE and Mistral Small. single tool call accuracy holds fine in isolation. the moment you chain 4 or 5 calls with overlapping parameter names, the model starts hallucinating required fields. not every run, maybe one in eight: predictable enough to hit production. have you found any prompt pattern that stabilizes tool call fidelity across a longer chain, or is it just model size at that point?
One in eight is the worst possible rate — frequent enough to hit production, rare enough to pass your smoke test. And you've located it precisely: not single-call accuracy (fine in isolation) but fidelity decay across a chain, which is the
q^kwall this thread kept circling — except your failure isn't invalid JSON, it's valid JSON with hallucinated required fields, i.e. the valid-and-wrong bucket @tom_jones measured at ~19%.To your actual question — prompt pattern vs model size — the thread's answer is mostly "it's not a prompt problem, it's a decoding problem," and two fixes came up that I'd reach for before scaling the model:
So my honest answer: before you blame model size, pin the KV-cache precision and put a grammar constraint on the schema — overlapping parameter names are exactly where constrained decoding earns its keep, because the model's free to pick the wrong value but not to omit a required one. If it still drifts one-in-eight after both, then it's capacity. Which runtime are you on? The grammar-constraint story is very different on llama.cpp vs vLLM, and that choice might be doing more than the model here (@samii's "measure the triple, not the model" point).
thank
🙏 Appreciate you reading it.
Fair point — patch score isn't a clean tool loop.
Exactly — a patch-apply score measures "did the diff land", not "did the agent navigate the loop to produce it". Two different skills, and the benchmark everyone quotes only measures the first.