The same small banking repo with hidden traps, run through cloud models, local models on three machines, and cheap Chinese models on OpenRouter.
I have a project I wrote about before, open-daimon (a group AI chat for Telegram and a platform for extensions). On top of it I built my own harness that runs other harnesses in different roles. The public open-daimon repo is not updated, because for now I develop only locally, it is faster that way.
At some point I wanted to get into local models, and the reason is simple: in the cloud, the rules belong to someone else, and they change without asking.
In October it came up again that Anthropic's consumer terms say it plainly for residents of the EEA and Switzerland: do not use the services "for any commercial or business purposes". These terms cover Claude.ai and the Pro and Max plans. So a developer in Europe who writes work code in Claude Code on a Max subscription is formally breaking them. For work you need the API, Team or Enterprise, and that is a very different amount of money.
The second story is about what you are allowed to do with your own hardware. A Decrypt writer asked Claude Opus 5.5 to help him get root back on his own LG TV. The model's safeguards cut the request off and suggested switching to Opus 4.8. And the researcher known for finding the always-on microphone in Samsung TVs says OpenAI banned his account for "Cyber Abuse" after he analyzed firmware with ChatGPT. About $500 of API balance went with the account. OpenAI has not commented, so for now it is only his word.
The direction is clear: poking around in someone else's software will be forbidden more and more often, and the vendor decides what counts as abuse. Napster was shut down by rights holders. They can come for Claude and Codex too, and there is plenty to ban there.
The models themselves are not safe either. In October, "uncensored" versions started disappearing from Hugging Face. One was taken down for violating the content policy. Another, the abliterated build of Kimi K2.7 Code by huihui-ai (the same huihui-ai whose Qwen3.6 shows up later in this test), just returns an error, and nobody has explained why (write-up). There is no mass purge yet. But in 2020 GitHub took down youtube-dl over a single RIAA complaint and only put it back a month later. As long as everything sits in one place, that place can be shut down.
Since April, Anthropic has been verifying the identity of some users: passport and selfie through Persona. One of the triggers is logging in from an unsupported country, and those accounts get blocked. Money is no longer enough to get the best model: it also matters where you live and what passport you hold.
So I wanted a backup in advance: download the weights while they are still there, and check that they work locally at least somehow. The cloud is unlikely to close tomorrow. But if it does, I want a doomsday agent system at hand: one that runs without a subscription, without someone else's rules and without anyone's permission.
A gaming laptop with an RTX 5080 already had LM Studio with Qwen3.6-35B-A3B. LM Studio runs models right on your computer and serves them through the same API as the cloud. I connected it to OpenCode, an open-source terminal agent, like Claude Code but with any model. It was slow, and models on the MacBook were even slower, really slow.
The usual advice here is to give a local agent a very light harness. For example pi, whose system prompt is about 2k tokens.
I decided to test everything on the same task: the cloud, a MacBook, a gaming laptop, an NVIDIA DGX Spark, and cheap, mostly Chinese, models through OpenRouter.
The task
I built the task for my "canary", a script that checks whether Opus and Codex have gotten dumber today. I made it after Opus was noticeably dumb for me for a while. Maybe I was in an A/B test, maybe there were technical problems, but I figured it would be useful.
The agent gets the minibank repo: ~700 lines of Python, a multi-currency banking ledger. Accounts in different currencies, transfers with conversion, fees, monthly statements, a CSV export for accounting. Next to it sits DECISIONS.md with the team's decisions, like in any living project.
And a request, as if from a real manager:
Customers are complaining: the monthly statement total doesn't match the account balance, especially when there were transfers between currencies. Figure it out and fix it. And check whether the same thing breaks anywhere else.
Not a word about tests. Not a word about DECISIONS.md either.
What is broken
The main bug: the statement does not take the amount that actually landed on the account during a transfer. It recomputes it through the old converter at the current rate. The rate changed during the month, and the statement total drifted away from the balance.
Then the traps begin:
- The bait converter. The old converter's docstring says "use this function everywhere". Right next to it there is a new, correct one. If you just swap the old one for the new one, almost all tests go green. But the statement still recomputes the transfer, just in a different way. The right fix is not to recompute at all and take the posted amount.
- A corrected rate. If a rate is corrected after the fact, old statement lines must not change. This catches exactly those who recompute.
- Time zones. The statement month is counted in UTC, not in the server's time zone. In Los Angeles and Tokyo, a naive implementation shifts the month boundary.
- The fee. Each fee is its own line and is not subtracted from the transfer amount.
- The export. The accounting CSV is broken in exactly the same way, and its timestamps must be ISO UTC. This is the "check whether the same thing breaks anywhere else".
-
The forbidden module. The old converter must not be touched: a partner export lives on it. This is written in
DECISIONS.md. - The traitor test. One of the existing tests pins the wrong numbers. It has to be updated, not deleted, and the code must not be bent to fit it.
How the score works
The agent does not see what it is graded with. After the run I take whatever is left in the repo and run:
- 20 hidden acceptance tests, worth half of the score;
- 7 of them check the "subtle" conditions from the list above (UTC, the fee, the corrected rate, ISO time), worth another 15%;
- 9 plausible wrong fixes (like "just swapped the converter"): I plant them one at a time and count how many are caught by the tests the agent wrote on its own, unasked. That is a quarter of the score;
- rules (did not touch the forbidden module, existing tests pass, the traitor test is updated) make up the remaining 10%.
Some hidden tests and rules check that what worked still works. So an agent that did nothing at all would get 0.31. That is awkward to read, so below every score is rescaled from zero: 0 means nothing changed, 1 is perfect, and below zero means the agent broke more than it fixed.
The limit is 20 minutes from the request. Cloud models finish in 1.5–11 minutes, so there is headroom. If an agent runs out of time, I grade what it managed to do. I picked 20 minutes knowing that the dense model on the Spark needs 16.5. I did not go higher: who wants to wait longer than that for a fix to a small task?
How much to trust the numbers
One task, in Python, and mostly one or two runs per setup. On the MacBook I ran more, and there you can see that the same model with the same settings can score 0.21 or go below zero. So this test cannot tell similar local models apart. But the gap between the best cloud models and the fast local ones came out so big that no noise covers it.
The baseline: the cloud
| Model | Time | Score | Hidden tests | Subtle conditions | Wrong fixes caught by its own tests | Run cost at API prices |
|---|---|---|---|---|---|---|
| Codex (GPT-6.1-sol, medium) | 101 s | 1.00 | 20/20 | 7/7 | 9/9 | ≈ $0.14 |
| Claude Opus 5.5 | 91 s | 0.96 | 20/20 | 7/7 | 8/9 | $0.30 |
| Claude Fable 5.1 | 288 s | 0.96 | 20/20 | 7/7 | 8/9 | $1.79 |
Codex does not report its cost, so I calculated it from the run's tokens (280k input, 243k of them from cache, and 4k output) and GPT-6.1 Sol API prices: 2 dollars per million input tokens, 10 cents for cached ones and 10 dollars for output.
All three read DECISIONS.md, did not fall for the bait converter, found the export, updated the traitor test and wrote their own tests that catch almost all the wrong fixes. Opus did it in a minute and a half with 7 tool calls.
Opus and Codex fixed the code the same way. The only difference is in the tests each wrote on its own. If you merge the fee back into the transfer line in the statement, Codex's tests catch it and Opus's tests don't. Opus itself made the fee a separate line, it just did not pin that with a test. Fable has one hole too, but a different one: its tests won't notice if the export starts recomputing the transfer at the new rate again. One missed wrong fix out of nine is the whole difference between 1.00 and 0.96. In a second run, both Opus and Fable missed both of these and got 0.92. In a third run Opus scored 0.96 again, but missed a different wrong fix, the one with the month boundary. The holes are mostly the same; what changes is how many of them get closed this time. I ran Codex only once, so its 9 out of 9 may be luck too. My personal take, as of October 2026: in everyday work I still prefer Opus, even though on this task Codex came out both cheaper and slightly better.
I also ran Opus and Fable in my working setup: with my settings, hooks, MCP servers, and separately also with 178k tokens of someone else's long session before the task, like on a normal working day. Everywhere the same 20/20 and all subtle conditions. The only thing that moves is how many wrong fixes their own tests catch: from 7 to 9 out of 9. In the clean setup it moves too, from 7 to 8, so it is noise. The long context only affected the price: a run costs 2–6 times more.
So the bar is: one and a half to five minutes and an almost perfect solution.
MacBook Pro on Apple M5 with 32 GB of memory
This is the computer many developers already have, and the first thing you want to try.
First, raw speed, without the task. Generation is how many tokens per second the model writes. Prefill is how many it reads: before writing a single word, the model reads the whole context: the system prompt, tool descriptions, open files and the history of steps. An agent's context grows with every step, so slow prefill hurts it more than slow generation.
| Model | Generation | Prefill |
|---|---|---|
| Qwen3 Coder 30B A3B (MoE, 4-bit) | 52 tok/s | 743 tok/s |
| Gemma 4 12B (4-bit) | 14 tok/s | 275 tok/s |
| Qwen3.8 27B (dense, 4-bit) | 5.4 tok/s | 159 tok/s |
In a MoE model, out of 30 billion parameters only about 3 work on each token. Each layer has many experts (separate blocks with their own weights: as if the model kept dozens of narrow specialists and called a few suitable ones for each token). The other experts sit idle meanwhile. That is why it is fast. Hence the names: 30B A3B means 30 billion total, 3 active. The dense 27B honestly runs all 27 and gives five tokens per second on a laptop. "4-bit" is quantization: each parameter is stored in 4 bits instead of 16. The model becomes four times smaller and fits in the laptop's memory, but loses a bit of quality.
LM Studio loads some models with an 8192-token context. And OpenCode sends about 15k tokens of system prompt and tool descriptions on every step. The task does not fit in that context. There is no error, the agent just works badly.
So the first thing to do is set Context Length in LM Studio for each model separately. I set 65536 for all of them.
Results
The first run of each model in pi:
| Model | Time | Finished on its own | Score | Hidden | Subtle |
|---|---|---|---|---|---|
| Gemma 4 12B | 575 s | yes | 0.21 | 12/20 | 3/7 |
| Qwen3 Coder 30B A3B | 472 s | yes | 0 | 8/20 | 1/7 |
| Qwen3.8 27B | 1200 s (limit) | no | 0 | 8/20 | 1/7 |
Qwen3 Coder turned out to be the fastest and the most useless. 50 tool calls, a cheerful final report about a "minimal and precise fix". And exactly that trap: it swapped the old converter for the new one but kept recomputing the transfer. By the hidden tests, it was as if it had done nothing. It also left its own test_debug.py in the repo root.
Qwen3.8 27B read DECISIONS.md, the only local model on the laptop that did. And in 20 minutes it did not make a single edit, neither in pi nor in OpenCode.
On the laptop, Gemma 4 12B won. It found the main thing: it took the posted amount instead of recomputing, in both the statement and the export. It missed UTC, ISO time and the fee, wrote no tests and did not read DECISIONS.md. But it fixed the customer's basic bug.
The smallest of the three turned out to be the most useful. It finished in under 10 minutes and did not wander off like Qwen3 Coder. Although it got this result in only three of its ten runs.
I also played with sampling (the randomness settings for picking the next token) on Gemma and Qwen3 Coder. I tried LM Studio defaults (temperature 0.8, top_k 40, top_p 0.95, min_p 0.05, repetition penalty 1.1), the values recommended by the model authors (temperature 1.0 and top_k 64 for Gemma; temperature 0.7, top_k 20 and top_p 0.8 for Qwen3 Coder), and a temperature of 0.3. It did not help. With the same settings, Gemma scored 0.21 and −0.17 in two runs, so the spread from chance is bigger than from any settings. And the mistakes are the same: no run managed UTC, ISO time or the traitor test.
A gaming laptop with an RTX 5080 and a light harness
The gaming laptop: an RTX 5080 Laptop with 16 GB of VRAM and 32 GB of RAM. LM Studio serves the model over the network. It runs Qwen3.6-35B-A3B, also MoE, but bigger.
Generation speed before and after retuning LM Studio:
| Context | Before | After |
|---|---|---|
| short | 54 tok/s | not measured |
| 17k tokens | 21 tok/s | 102 tok/s |
| 42k tokens | 9.5 tok/s | 94 tok/s |
The drop from 54 to 9.5 tok/s on a long context looked suspicious, and I started digging. The model weighs 21.7 GB and there are 16 GB of VRAM, so part of the model lives in RAM either way. Almost all the weight of a MoE model is in the experts, the same blocks of which only a few switch on for each token. Attention (the part of every layer that, on each new token, looks through the whole context again to decide what matters in it) and the shared layers take little. It pays best to keep some of the experts in RAM: only a few of them switch on per token, about 3 billion parameters out of 35, and the CPU copes with them. But attention has to stay on the GPU: the longer the context, the more it has to look through. Parallel requests and context size turned out to be irrelevant: with one slot instead of four and a 32k context instead of 64k, the speed did not change. What decided everything was how LM Studio splits the model between the GPU and the CPU. The model had no saved layout, and LM Studio decided on its own how many layers to put on the GPU. I set three parameters:
- GPU Offload to the maximum: all 40 layers on the GPU;
- "Number of layers for which to force MoE weights onto CPU" to 40%: the experts of about 16 layers run on the CPU, and everything else fits in 16 GB;
- the KV cache in q8_0, to free up VRAM for the context.
Together with the KV cache, about 14.5 GB of VRAM is in use. Without these settings, judging by the speed, LM Studio moved whole layers to the CPU, attention included. The longer the context, the longer the CPU took to process it, hence the 9.5 tok/s. If you send the experts of all layers to the CPU, only 4 GB of VRAM is used, the GPU sits almost idle, and you get about 40 tok/s. It is fastest when VRAM is filled almost to the top.
A light harness: pi vs OpenCode
I installed pi and compared it with OpenCode on the same model. On the first step OpenCode sends the model about 15k tokens, and pi about 2,200. A local model reads about 1000 tokens per second, so OpenCode's extra 13k tokens cost it about 13 seconds. A cloud model reads that much almost instantly.
The minibank run, still before retuning:
| OpenCode | pi | |
|---|---|---|
| Score | 0.21 | 0.29 |
| Hidden tests | 12/20 | 12/20 |
| Subtle conditions | 3/7 | 3/7 |
| Wrong fixes caught by its own tests | 0/9 | 2/9 |
| Tool calls | 28 | 29 |
| Time | 20 minutes, limit | 10 minutes, finished on its own |
Pi finished twice as fast and ran the tests four times, and its tests caught two wrong edits. But both solved the same thing and missed the same conditions: UTC months, ISO time and the fee. With a 10-minute limit it was the same.
After retuning I ran minibank again:
| OpenCode | pi | |
|---|---|---|
| Score | 0.14 | 0.31 |
| Hidden tests | 10/20 | 12/20 |
| Subtle conditions | 2/7 | 3/7 |
| Wrong fixes caught by its own tests | 1/9 | 2/9 |
| Time | 20 minutes, limit | 285 seconds, finished on its own |
Pi became twice as fast: under five minutes instead of ten. The same solution with the same misses, only this time pi updated the traitor test. OpenCode, already updated to version 2.2, still did not fit in the limit: the model reasoned out 55k tokens in it versus 11k in pi, ran the tests ten times, never got to the end, and on top of that broke the corrected rate in the export. So the model got faster, but not smarter.
Separately, about speculative decoding, which this laptop also uses: an extra layer of the model itself (MTP, multi-token prediction) quickly guesses several tokens ahead, the main model checks them in one pass, and where the guess is right, it comes out faster. It is turned on by a separate flag when the model is loaded. If you forget the flag, the model quietly runs twice as slow.
DGX Spark
The Lenovo ThinkStation PGX is built on the NVIDIA DGX Spark: a desktop box with a GB10 chip and 128 GB of unified memory. Models run in vLLM, a model server built for many requests at once. The task went to it as an archive (without the hidden tests, only the repo, the request and the scripts) and was run through OpenCode. I graded it with the same script.
MoE: the same model, the same mistakes
Qwen3.6-35B-A3B, an Opus distill (that is, a model fine-tuned on Opus's answers), in 4-bit NVFP4. This is NVIDIA's quantization format that the Spark's chip computes in hardware. 35–50 tok/s.
340 seconds, finished on its own, 0.25. 12/20, 3/7.
It missed UTC months, ISO time and the fee. Exactly the same conditions as on the RTX 5080 laptop in OpenCode and in pi. Five runs, two different computers, two different servers, two different harnesses, and all five missed UTC months, ISO time and the fee.
Dense 27B: ten minutes is not enough
ThinkingCap-Qwen3.8-27B is dense and a reasoning model, in NVFP4 with speculative decoding of 3 tokens. 20–25 tok/s.
With a 10-minute limit: an empty diff, not a single change to the code, 0.
According to the log, for the first 5.5 minutes OpenCode's scout subagent was looking over the project. Until minute 8 the main agent read all the files and correctly named the main bug in its reasoning. At minute 8 it started one long generation, and the limit cut it off. Not a single edit.
The same run without a limit: 987 seconds, 0.92: all subtle conditions, the traitor test updated, its own tests catch 7 fixes out of 9. An Opus-level result.
MoE at 125B: just about fits in ten minutes
Another model on the Spark is Qwen3.8-Flash-Next. It is an abliterated version, MoE, but of a completely different size: 125 billion parameters, with about 6 working per token. NVFP4 quantization mixed with MXFP8, speculative decoding of 3 tokens, 22–32 tok/s. Its limit was 10 minutes, not 20.
The limit did cut it off, but during a final re-check: all the edits were already made by then, and it says it ran the visible tests in four time zones. I graded what was left in the repo.
0.96. 20 out of 20 hidden tests, all 7 subtle conditions, the traitor test updated, nothing forbidden touched. Seven of its own tests that catch 8 out of 9 wrong fixes. They missed only one: an export that recomputes the transfer at the new rate.
That is Opus level in a clean setup. Six to seven times slower than Opus, and it used up the whole ten minutes.
The price is memory. The driver holds about 96 GiB for the model, about 106 together with the vLLM processes, and the Spark is almost completely full:
| What | GiB |
|---|---|
| Weights on the GPU | 75.8 |
| MTP layer for guessing tokens | 1.5 |
| vLLM service buffers | ~5.6 |
| KV cache pool (fp8) | 13.4 (~899k tokens) |
| Total held by the NVIDIA driver (measured) | ~96.2 |
| vLLM processes on the host side | ~10 |
| PLE table (26.8 GiB on disk, read via mmap) | 13–16 in the page cache, the system can evict it |
| Free for the system while running | ~16 of 121 |
One run. After Gemma gave me both 0.21 and −0.17 with the same settings, I would not draw conclusions from a single run. But 20 out of 20 with all seven subtle conditions does not happen by accident either.
Worse than Flash-Next, but within the limit: Nemotron 3.5 Super
NVIDIA Nemotron 3.5 Super is another big MoE: 120 billion parameters, 12 active, NVFP4, a draft model guessing 7 tokens ahead. 24–42 tok/s. The driver holds about as much for it, ~96 GiB.
The same 10-minute limit, and it finished on its own within it: 533 seconds.
0.68. 18 out of 20 hidden tests, 6 out of 7 subtle conditions. It fixed the transfer, UTC months and the CSV format. But it left the fee inside the transfer line in the statement, although DECISIONS.md has exactly that as a separate rule. It did not update the traitor test. It wrote its own tests, but they catch only 4 wrong fixes out of 9.
So on the Spark: of the two big MoEs, Flash-Next leads, it found everything. Nemotron is weaker, but finished within the limit on its own. The driver holds about 96 GiB for each.
Cheap cloud: Chinese models
Between local hardware and Opus at $0.30 per run there is a whole market of cheaper models. Through OpenRouter I took one or two fresh models from each of eight major Chinese vendors, added Devstral for comparison, and ran them with OpenCode and the same limit. These are the regular versions, not abliterated.
The eight Chinese vendors: DeepSeek, Zhipu (GLM), Alibaba (Qwen), Moonshot (Kimi), MiniMax, Tencent, StepFun and Xiaomi. Devstral is French, from Mistral.
| Model | Time | Score | Hidden | Subtle | Wrong fixes caught by its own tests | Run cost |
|---|---|---|---|---|---|---|
| DeepSeek V4.1 Flash (Oct 17 snapshot) | 112 s | 1.00 | 20/20 | 7/7 | 9/9 | $0.008 |
| Xiaomi MiMo V2.6 Pro | 450 s | 1.00 | 20/20 | 7/7 | 9/9 | $0.06 |
| Tencent HY4 Preview | 333 s | 1.00 | 20/20 | 7/7 | 9/9 | $0.17 |
| Qwen3.8 Max Prime | 354 s | 1.00 | 20/20 | 7/7 | 9/9 | $0.73 |
| GLM-5.3 | 198 s | 0.98 | 20/20 | 7/7 | 9/9 | $0.10 |
| Qwen3.8 Max | 671 s | 0.92 | 20/20 | 7/7 | 7/9 | $0.36 |
| MiniMax M3.1 Flash | 194 s | 0.90 | 20/20 | 7/7 | 7/9 | $0.03 |
| StepFun Step-5 Preview | 164 s | 0.88 | 20/20 | 7/7 | 6/9 | $0.17 |
| Kimi K3.1 Pro | 370 s | 0.80 | 20/20 | 7/7 | 4/9 | $0.27 |
| Kimi K3 | 202 s | 0.72 | 18/20 | 6/7 | 5/9 | $0.16 |
| DeepSeek V4 Pro | 103 s | 0.47 | 15/20 | 4/7 | 3/9 | $0.11 |
| Devstral 2512 | 276 s | 0.28 | 14/20 | 3/7 | 0/9 | $0.16 |
Four models solved the task like Codex: all hidden tests, all subtle conditions, the traitor test updated, and their own tests catch all 9 wrong fixes. DeepSeek V4.1 Flash did it in two minutes for under a cent, 36 times cheaper than Opus, which scored 0.96 on the same task. The September version of the same Flash scored 0.88: its tests caught 6 fixes out of 9. The October snapshot seems to have closed the holes, although that run was still on the old version of OpenCode.
GLM-5.3 is right there: it lost 0.02 only on the traitor test, which it did not update. Three times cheaper than Opus.
MiniMax M3.1 Flash: 20 out of 20, all subtle conditions, 18 of its own tests that catch 7 fixes out of 9. And the same habit as the previous M3: it added a paragraph to DECISIONS.md, which must not be touched. Three cents a run.
Step-5 from StepFun also got 20 out of 20, in under three minutes, but its tests catch neither the swapped converter nor the recompute at the new rate.
Kimi disappointed. K3.1 Pro, the flagship priced at 2 dollars per million input tokens, passed all hidden tests but wrote only three tests of its own, and they catch 4 fixes out of 9. K3 missed the fee and did not update the traitor test. The previous Kimi K2.7 Code (the one whose uncensored build was removed from Hugging Face) scored the same 0.72. K2.7 and K3 have the same holes: the fee and the traitor test. K3.1 Pro closed them, but hardly writes tests, and $0.27 a run is almost the price of Opus.
DeepSeek V4 Pro is a strange case: a bigger and more expensive model than Flash, and half the result, with time zones and the fee missed. Maybe it is the dates: DeepSeek's Pro is from August, its Flash from October.
Devstral missed time zones, ISO time and the fee and did not write a single test of its own.
The cloud Qwen3.8 Max and the local 30–35B Qwens are one family, and their results sit at opposite ends of the table. Max passed every hidden test and every subtle condition and updated the traitor test; the only holes are in its own tests, 7 out of 9. It did think for 11 minutes, though, longer than Flash-Next on the Spark, and cost $0.36, more than Opus. Max has 2.4 trillion parameters, the local MoEs 35 billion. Max found almost everything, and they fixed only the obvious. Then again, the dense local 27B on the Spark scored the same 0.92, it just thought for sixteen minutes.
Qwen also has Max Prime, a faster and more expensive variant of the same Max. It solved the task perfectly, 1.00 in six minutes, but cost $0.73 a run, 2.4 times more than Opus.
One run per model is not enough, we have already seen the spread. But the gap between the top half of this table and the fast local models is too big.
Everything together
| Model | Where | Time | Score | Run cost |
|---|---|---|---|---|
| Codex GPT-6.1-sol | cloud | 101 s | 1.00 | ≈ $0.14 |
| DeepSeek V4.1 Flash | OpenRouter | 112 s | 1.00 | $0.008 |
| Xiaomi MiMo V2.6 Pro | OpenRouter | 450 s | 1.00 | $0.06 |
| Tencent HY4 Preview | OpenRouter | 333 s | 1.00 | $0.17 |
| Qwen3.8 Max Prime | OpenRouter | 354 s | 1.00 | $0.73 |
| GLM-5.3 | OpenRouter | 198 s | 0.98 | $0.10 |
| Opus 5.5 | cloud | 91 s | 0.96 | $0.30 |
| Fable 5.1 | cloud | 288 s | 0.96 | $1.79 |
| Qwen3.8-Flash-Next 125B (MoE) | DGX Spark | 600 s, 10-minute limit | 0.96 | local |
| Qwen3.8 Max | OpenRouter | 671 s | 0.92 | $0.36 |
| ThinkingCap-Qwen3.8-27B | DGX Spark | 987 s | 0.92 | local |
| MiniMax M3.1 Flash | OpenRouter | 194 s | 0.90 | $0.03 |
| StepFun Step-5 Preview | OpenRouter | 164 s | 0.88 | $0.17 |
| Kimi K3.1 Pro | OpenRouter | 370 s | 0.80 | $0.27 |
| Kimi K3 | OpenRouter | 202 s | 0.72 | $0.16 |
| Nemotron 3.5 Super 120B (MoE) | DGX Spark | 533 s | 0.68 | local |
| DeepSeek V4 Pro | OpenRouter | 103 s | 0.47 | $0.11 |
| Qwen3.6-35B-A3B + pi | RTX 5080 Laptop | 285 s | 0.29–0.31 (2 runs) | local |
| Devstral 2512 | OpenRouter | 276 s | 0.28 | $0.16 |
| Qwen3.6-35B-A3B | DGX Spark | 340 s | 0.25 | local |
| Gemma 4 12B | MacBook M5 | 575 s | 0.21 (over 10 runs: from −0.17 to 0.21) | local |
| Qwen3.6-35B-A3B + OpenCode | RTX 5080 Laptop | 1200 s, limit | 0.14–0.21 (2 runs) | local |
| Qwen3 Coder 30B A3B | MacBook M5 | 344 s | 0.14 (over 6 runs: 0–0.14) | local |
| Qwen3.8 27B | MacBook M5 | 1200 s, limit | 0 | local |
0 means nothing changed, below zero means the model broke more than it fixed.
So, can you or can't you?
On this task, you can. On a real project, it is unknown.
A smart local model already exists, but the hardware is expensive and the speed is slow. The dense 27B solved the task at Opus level, it just worked sixteen minutes instead of a minute and a half. The 125B MoE did it in ten, but took almost all 128 GB of the DGX Spark. Small models that fit on a laptop fix only the obvious. Speed is limited by hardware, quants and speculative decoding, and all of that gets cheaper. The smarts, it seems, are already there, at least on 700 lines.
The Chinese cloud is still cheaper and more practical: four models solved the task like Codex, DeepSeek V4.1 Flash for under a cent.
A local model is dumb sometimes too, and its spread, as you saw above, is big. But it is always the same model, and nobody can swap it without asking.
And separately about safety. Several of the local models in this test, including Qwen3.6 from huihui-ai and Flash-Next, are abliterated, meaning their refusals were cut out. Pi runs commands without confirmation by default, and its documentation says plainly that watching the transcript does not protect you. A model that can't refuse and an agent that doesn't ask: on a work computer, that pair is better run in a container.
P.S. I edited the text of this article at the end with Qwen3.6-35B-A3B. This model answers fast, heats the room great, and mostly does harm. With Opus I had managed to forget how far we have come from where we started, and that if you want to drop Opus and Codex, you have a long road ahead. The cover was also made by a local model: Qwen-Image 2.1 in a 4-bit quant, in ComfyUI on the same RTX 5080 gaming laptop, about two minutes per image.
Top comments (0)