Send Claude Code's whole requests to a local Qwen3 and almost nothing goes through. One recent measurement on an RTX 4070 counted 100 real requests and 96 had to go to the frontier model. Then the same workload was split into single Claude Code steps and counted again: 97 of 200 steps were fine on the local model. Same box. Same requests. The thing that changed was the routing granularity.
Both numbers are from the companion material for a Kindle book by kenimo49 (Local LLM or Claude Code?), with MIT-licensed reference code on GitHub as local-step-router. I want to walk through what those two numbers imply for anyone who has a local LLM sitting idle and a Claude Code bill that is not.
Request-level routing doesn't carry the material
Here is the first number again. 100 real daily requests were labeled for whether a local LLM on a single RTX 4070 with 12 GB of VRAM could handle them. Qwen3-8B at Q4 fits that VRAM band with room for context (llmhardware.io on Qwen3 VRAM). Of the 100, 4 could stay local (and only because they contained confidential data that shouldn't leave the machine); the other 96 had to go to the frontier model. Zero were a clean fit for a local model on merit.
The reason is not the model quality. A one-line request like "fix the test that just failed" is a pointer, not material. The actual work lives in the repo, the stack trace, the last three commits. Hand that one line to a local model and there is nothing it can bind to. Hand it the same line after Claude Code has gathered the trace and opened the file, and a 4B judge can tell you whether the diff compiles.
If you have tried "swap Claude Code for a local model" and been disappointed, this is probably why. The failure is routing, not the model. Swapping the model when the actual problem is granularity is the AI-era version of buying a bigger car to beat traffic.
Step-level routing changes the math
Claude Code does not act as a single request. One "fix the failing test" turns into dozens of steps internally: WebFetch, Read, Grep, Edit, Bash, Write, Task. Each of those steps has a return shape the orchestrator knows how to consume.
Counted at that granularity, the companion book's measurement said 97 of 200 Claude Code steps were fine on the local model. 48% of the work. The split was not uniform. Some step types routed cleanly, others didn't. The local-step-router repo's router.py encodes the pattern as rules first and a probability judgment second.
The rule is the interesting part.
A one-line rule beat a 9B model's probability
The LP for the book quotes a result worth sitting with: a one-line rule, "WebFetch goes local", beat a 9B model's probability judgment on the same task. Not by a little. The rule was simpler to implement, deterministic, and routed more work correctly than asking a local model "is this step safe locally?" and reading the number back.
That is the pattern the companion code implements. router.py decides by step type first. judge.py only runs on the edge cases the rules can't settle. If you read Ollama's native /api/chat endpoint for the top-token logprob, you can get a confidence number out of a 4B model for a yes/no question in a single token. The threshold you draw on that number is where the second number in the title comes from.
# conceptual shape of the router (see local-step-router/router.py for real code)
def route(step):
if step.type == "WebFetch":
return "local"
if step.type in ("Read", "Grep") and step.size < THRESHOLD:
return "local"
# only the undecided cases call the model
confidence = judge_with_logprobs(step)
return "local" if confidence >= calibrated_threshold else "frontier"
Rules do the bulk. Probability judgments cover the rest.
The threshold isn't 0.5
The second place a lot of "local LLM as judge" setups go wrong is treating 0.5 as the obvious threshold. It is not. The LP reports a measured case where a local gatekeeper let a production DB password through with 0.69 confidence. The model's ordering was perfect (secrets scored higher than non-secrets on average), but the raw scale was off.
The right line is drawn from your own labeled data, not from 0.5. On the book's evaluation set, calibrating the threshold cut missed secrets from 20 to 2. Same model. Same inputs. Only the decision rule changed. If you have ever drawn the 0.5 line and had a judge wave something through anyway, that is what you were looking at.
The companion repo ships a threshold.py CLI for this. You give it labeled examples, it reports the highest threshold with zero misses on your set, and you can decide how much margin to leave. The math is unsurprising (it is the "pick the lowest FPR at TPR=1" move), but having it as a tool next to the router means the calibration gets redone as the data drifts, instead of baked in once and forgotten.
Three traps where logprobs just disappear
If you are going to read a probability out of a local model, there are three places the number quietly becomes meaningless, all documented in the book's Chapter 6 outline:
-
The OpenAI-compatible endpoint silently drops logprobs. If you point your code at Ollama's
v1/chat/completionsand the native API at/api/chat, the native one returns top-token probabilities. The OpenAI-compatible one has dropped them in some Ollama versions without raising an error. A 2026-Q4 Ollama release note is worth checking before trusting alogprobsfield over OpenAI-compat. - Thinking models don't put the answer in the first token. Qwen3 series models with thinking mode on will emit a reasoning trace first and the yes/no later. Reading the first token's probability gives you the probability of a reasoning-trace token, not the decision.
- A top-20 candidate list hides low-probability answers. If the character you were watching for isn't in the top-20, the probability reads as zero without an error. For binary judgments ("Y" vs "N") this is not a problem; for routing across many labels, it is.
The book spends a chapter on these because each one fails silently. The code runs, the number comes back, it is just wrong.
Rules decide the fixed parts. Probability covers the rest.
If I had to collapse the two numbers in the title into one takeaway: route at the step level, decide by rules where you can, and only use a local probability for the rest. The 96 vs 48% gap is the cost of routing at the wrong granularity. The "rule beat probability" result is the cost of using a model to decide things a one-line rule already settles.
The book's blueprint, the local-step-router code, and the threshold worksheet are companion material to the data above. For the other side of the same question (how many hours of use per day flip the economics from frontier API to local), the author's Claude Code Max vs API vs Qwen hour-of-day breakeven breakdown is the closest public companion. For the raw tok/s side, RTX 4070 + --cpu-moe flag notes measure the 2.8x speedup from one llama.cpp flag.
The pattern transfers. Count your own steps. Decide the fixed parts by type. Keep the local judge for the edge cases, and calibrate its threshold from your own data instead of 0.5.
The book: Local LLM or Claude Code? 96% of Requests Failed Locally, Half of the Steps Didn't. Measured routing between a local LLM on one RTX 4070 and a frontier model, with the blueprint and the companion code (MIT) above.

Top comments (0)