DEV Community: david

A 7.5B model beat a 24B on my coding benchmark.

david — Sat, 01 Aug 2026 13:33:57 +0000

16 local model configurations, 56 hidden-test coding tasks, 36 full runs, one 16 GB card

Every number below was recounted from the committed SCORES-*.tsv files, not transcribed from notes. The raw data - 36 rows, one per run, each with its full failure list - is RESULTS-q56.csv.

gemma-4-e4b is 7.5B parameters in a 4.97 GiB file. It scored 42/56.

devstral-small-2-24b is 24B in 11.90 GiB. It scored 40/56. gpt-oss-20b scored 38. Both
gemma-4-12b variants - one of them at Q8_0, more than twice the precision - landed at 42.5 and 41.

Three runs each. The ranges do not overlap. On a small card, the 5 GiB model is not the
compromise option, and no public leaderboard would have told you that.

Everyone has opinions about which local model to code with, which quantisation to run, and what
KV cache setting costs you. Almost all of those opinions come from a public leaderboard, or from
one run of one config on somebody's box.

So I built a 56-task coding benchmark that grades with hidden tests, and ran 16 model
configurations through it on the same card at the same constants, with every held-constant
actually recorded. Then I ran several of them repeatedly - 36 full runs in total - to find
out how much of a single number is noise.

The ranking does not match any public leaderboard; in one case it inverts one outright. And the
noise result surprised me more than the ranking did: the error bar is a property of the model,
not of the benchmark. One model swings 8 points between identical consecutive runs while another
is bit-for-bit reproducible - and it is not the weaker one that swings.

The method, and why there is no LLM judge

56 medium-hard coding tasks run through a real agent loop - the model reads files, edits them,
runs tests, and iterates until it says it is done. Grading is hidden verification:

The fixture the model sees ships only happy-path visible tests.
At grade time, a verifier injects hidden reviewer tests and builds/runs them against whatever the model actually left on disk.
Pass/fail is the compiler and the test runner. No model judges another model's output.

That last point matters more than it sounds. The lesson this project inherited from its own
ancestors: LLM judges inflate. Deterministic ground truth may be lowered by a model, never
raised. So there is no judge here at all - just cargo test, pytest, and node.

Tasks are test-first gated. A task enters the corpus only if its verifier fails on the
untouched fixture and passes on fixture + reference solution. That rules out tasks that are
accidentally already passing, and tasks that are impossible.

The discriminating axis turned out not to be algorithmic difficulty. It is unstated correctness
traps - the degenerate input, the falsy-vs-absent distinction, the boundary the prompt implies
but never enumerates. Prompts state the goal the way a user would; the hidden tests grade the way
a reviewer would.

Corpus: 56 tasks - 15 python, 14 rust, 10 js, 9 ts, 8 shell. By kind: 18 implement-spec,
7 boundary, 6 bugfix, 6 api-misuse, 5 error-handling, 4 refactor, 4 perf, 4 multi-file,
2 concurrency.

Box: RTX 5060 Ti 16 GB, Windows 11, LM Studio. Held constant on every row: ctx 32768,
KV cache q8_0, numParallelSessions 1, one agent session at a time, identical agent binary.
Every row's actual values are recorded rather than assumed - see "the method lesson that cost the
most", which is the section I would most want another benchmarker to read.

The table

Full 56 tasks on every row. Weak scores are data, not failures - a benchmark that reports only
winners tells you nothing about the shape of the field.

model	params	quant	GiB	runs	median	range	n	wall-clock
google/gemma-4-26b-a4b-qat	26B-A4B	Q4_0	13.45	55, 54, 55	55	1	3	71–76 min
unsloth/gemma-4-26B-A4B-it	26B-A4B	UD-Q4_K_M	15.78	54	54	-	⚠️1	71 min
qwen3.6-35b-a3b-mtp@iq3_s	35B-A3B	IQ3_S 3.06bpw	12.67	49, 50, 50, 50, 48, 52	50	4	6	21–31 min
qwen3.6-35b-a3b@iq4_xs	35B-A3B	UD-IQ4_XS 4.25bpw	16.51	50	50	-	⚠️1	53 min
qwen3.6-35b-a3b-mtp (GPU-3)	35B-A3B	IQ4_XS 3.53bpw	14.59	49	49	-	⚠️1	43 min
qwen3.6-35b-a3b-mtp (GPU-4)	35B-A3B	IQ4_XS 3.97bpw	16.43	49	49	-	⚠️1	51 min
qwen3-coder-30b-a3b-instruct	30B-A3B	UD-Q4_K_XL	16.45	43	43	-	⚠️1	58 min
google/gemma-4-12b	12B	Q8_0	11.80	42, 43	42.5	1	2	44–53 min
google/gemma-4-e4b	7.5B	Q4_K_M	4.97	43, 42, 42	42	1	3	30–35 min
north-mini-code-1.0 (Cohere)	30B-A3B	UD-Q3_K_M	13.24	42	42	-	⚠️1	82 min
google/gemma-4-12b-qat	12B	Q4_0	6.50	41	41	-	⚠️1	57 min
devstral-small-2-24b-2512	24B	IQ4_XS	11.90	40, 39, 40	40	1	3	19–22 min
openai/gpt-oss-20b	20B-A3.6B	MXFP4	11.28	40, 38, 32	38	★8	3	10–12 min
qwen3.5-4b	4B	UD-Q8_K_XL	5.54	31, 33, 33	33	2	3	33–42 min
google/gemma-4-e2b	4.6B	Q4_K_M	3.19	31, 31, 31	31	0	3	23 min
unsloth/granite-4.1-8b	8B	Q8_0	8.70	25	25	-	⚠️1	22 min

Sizes are the measured GGUF file in GiB, not a vendor marketing figure and not lms ls
(which reports decimal GB and folds in the mmproj projector). Quant names are exact, because
"gemma-4-12b" at Q8_0 and at Q4_0 are two different experiments.

On the rows marked ⚠️1: a single run carries an error bar this benchmark can now put a number
on, and it is not small - see the noise section. Nothing load-bearing in this piece rests on an
n=1 row. Where one does (QAT at 12B), I say so and decline to make the claim.

One model was excluded rather than scored: nemotron-3-nano-omni-30b-a3b took 299 seconds on
the corpus's easiest task (vs 18 s for devstral) and projected to roughly 4.5 hours. At that
speed the per-task timeouts start failing tasks on the clock rather than on quality, which
produces an unreadable row. Its config was verified correct and its template healthy - the
verdict is speed alone. Recording why a model is absent beats leaving a hole.

Part 1 - what the models did

Result 1: size buys nothing - but it is a plateau with cliffs, not a flat line

The single most repeated claim in local-model threads is some version of "just run the biggest
model that fits". Here is what the gemma-4 family does, one architecture, one vendor, four sizes:

model	params	quant	GiB	median
gemma-4-e2b	4.6B	Q4_K_M	3.19	31
gemma-4-e4b	7.5B	Q4_K_M	4.97	42
gemma-4-12b	12B	Q8_0	11.80	42.5
gemma-4-26b-a4b-qat	26B-A4B	Q4_0	13.45	55

4.6B → 7.5B    +11.0
7.5B →  12B     +0.5      ← the plateau
 12B →  26B    +12.5

Scaling 7.5B → 12B buys half a point. That is the flat stretch, and it is flat despite the
12B carrying the better quantisation - Q8_0 against the 7.5B's Q4_K_M, which is the conservative
direction. Give the small model the same precision and the gap would likely go the other way.

But it is bounded on both sides by real cliffs. Below it, dropping to 4.6B costs 11 points.
Above it, the 26B MoE gains 12.5. "Size buys nothing" is true in a band, and the band is
narrower than the slogan.

Which is where the title comes from. gemma-4-e4b clears devstral-small-2-24b on under a
third of its parameters and 42% of its file size - and it does it across the whole family
boundary, not just against one unlucky model: it also beats gpt-oss-20b (38), north-mini-code
(42, n=1), and both 12B gemmas, one of which has twice its precision. Three runs each for the
replicated rows, non-overlapping ranges.

And active parameters do not explain it either. gemma-4-26b-a4b activates about 4B
parameters per token - roughly what the 4.6B e2b activates - and scores 55 against 31. Same
active compute, 24 points apart. Whatever is doing the work here, it is neither total size nor
active size on its own.

Result 2: public leaderboards do not rank this corpus

None of Result 1 is visible from a leaderboard, and that is not an accident - the ranking they
produce is different from mine at the top of the table too.

A vendor comparison table puts Gemma-4-31B roughly 21 points below Qwen3.6-35B on SWE-bench
Verified. On this corpus the gemma MoE beats the qwen champion 55 to 50.

LiveCodeBench v6 anti-correlates outright: gemma scores 77.1 there and 55/56 here; qwen scores
80.4 there and 50/56 here. Higher on the board, lower on my tasks - in both directions, on the two
models I care most about. And LiveCodeBench's own table for these models is 0-verified and
53-self-reported.

It is worse than "different benchmarks measure different things", because several of the obvious
alternatives are simply not usable any more: BigCodeBench has been frozen since 2025-04-14
(202 rows, zero 2026 models) and Aider's polyglot benchmark went dead 2025-10-04. Meanwhile
published work shows that scaffold choice alone swings the same weights about 2× on
Terminal-Bench - 24.6% on the official board against 51.5% vendor-claimed for the same model.

That last number is the actual mechanism, and it is why I do not think my ranking is "righter"
than LiveCodeBench's. A leaderboard measures a model answering in a box. What you experience is a
model driving your agent, with your tool schema, your context window, your file-edit format. The
scaffold is not a detail wrapped around the model; on Terminal-Bench's own numbers it is worth as
much as the model choice.

Use leaderboards to decide what to download. Never use them to predict what will happen inside
your own harness. The ranking above took 34 full runs to discover and could not have been
guessed from any public table.

Result 3: quantisation bits bought nothing, in the band people argue about

Four quantisations of one MoE family, same KV cache and context:

variant	bpw	GiB	score
MTP-GPU-2	3.06	12.67	50 (median of 6)
MTP-GPU-3	3.53	14.59	49
MTP-GPU-4	3.97	16.43	49
UD-IQ4_XS	4.25	16.51	50

Flat across the entire 3.06 → 4.25 bpw range, while file size varies by nearly 4 GiB and
end-to-end speed varies about 3×. The lowest bpw of the four is the best deal on this hardware,
and it is not close - it is the only one that stays fully VRAM-resident.

Caveat, stated plainly: this is one MoE family on one 16 GB card, and three of those four rows
are single runs. It is not "quantisation never matters", and I would not assume it transfers to
dense models, which are materially more quant-sensitive. The claim is narrower and better
supported: within the band people actually argue about, on a card this size, for this family, the
extra bits bought no measurable code quality and cost real speed.

The QAT claim I am not making

Quantisation-aware training looks free, and I have two pairs that appear to show it:

pair	higher precision	QAT	gap
gemma-4-26b-a4b	UD-Q4_K_M 15.78 GiB → 54 (n=1)	Q4_0 13.45 GiB → 55 (n=3)	Q4_K_M vs Q4_0
gemma-4-12b	Q8_0 11.80 GiB → 42.5 (n=2)	Q4_0 6.50 GiB → 41 (n=1)	★Q8_0 vs Q4_0

These are not the same experiment, and collapsing them into one "QAT is free" line is wrong in
both directions. At 26B it compares Q4_K_M against Q4_0 - both 4-bit-class, a near-like-for-like
swap, so "QAT matches it" is a modest claim. At 12B it compares Q8_0 against Q4_0, a 45%
smaller file at half the precision, which would be a genuinely strong result.

Except the 12B QAT arm is one run, against a non-QAT arm whose own two runs span 42–43. One
point of difference, inside both models' measured spread. So I am not claiming it. The
experiment that would settle it is two more runs of gemma-4-12b-qat, and until those exist the
honest statement is: QAT at 26B costs nothing measurable; QAT at 12B is untested.

I am flagging this at length because it is the shape of claim that gets repeated forever after one
run, and I nearly published it.

Part 2 - what the benchmark did

This is the half I did not expect to be interesting.

Result 4: the error bar is a property of the model, not of the benchmark

For a long time I carried a single global noise figure - "±3" - and applied it to every
comparison. That was always wrong, and it took replicating several models to see it.

Consecutive runs, identical config, identical engine, same box, nothing changed between them:

model	runs	median	range
gemma-4-e2b	31, 31, 31	31	0
gemma-4-e4b	43, 42, 42	42	1
devstral-small-2-24b	40, 39, 40	40	1
gemma-4-26b-a4b-qat	55, 54, 55	55	1
qwen3.5-4b	31, 33, 33	33	2
champion 3.06bpw	49, 50, 50, 50, 48, 52	50	4
openai/gpt-oss-20b	40, 38, 32	38	★8

gpt-oss-20b swings 8 points across three identical runs. Its published single-run number -
the 40 that would have gone in the table if I had run it once, as one normally does - is the
top of its range, not its centre. Single-run reporting flattered it by two points and nothing
in that run signalled it.

Meanwhile gemma-4-e2b ran three times and produced not just the same score but the identical
failure list - the same 25 tasks, the same error text. The only diffs between run 2 and run 3
were harness artifacts: cargo's parallel test ordering, an OS thread id, ±1 s timings.

Weakness does not cause the variance. The failure mode does.

The obvious explanation for gpt-oss's swing is "it is borderline on lots of tasks, so it coin-flips
them". That explanation is dead, and e2b is what killed it: e2b fails more tasks than
gpt-oss - 25 against 16–24 - with range 0. Being bad is stable. Being erratic is something
else.

What separates them is how they fail. gpt-oss loses tasks to agent mechanics: a bare
top-level return producing ERR_INVALID_TYPESCRIPT_SYNTAX, SyntaxError: Unexpected end of input, literal \n escape sequences written into a .mjs instead of real newlines. Its worst
run lost five tasks to syntax, not reasoning. Whether a generation comes out as a parseable
file is close to a coin flip. Missing an edge case, by contrast, is a stable property of what the
model knows.

⇒ A model's error bar is predicted by its failure mode. This was a hypothesis after the first
replication night and it then held as a prediction on e2b, at the opposite end of the quality
range.

The practical consequence: you cannot import someone else's error bar, and you cannot import
mine. If you are comparing two local models and the gap is under 3 points, you do not have a
result until you know each model's own spread. That is 2–3 extra runs per arm, and for some models
it is the difference between a finding and a coin flip.

Result 5: "deterministic" turned out to mean "deterministic within one session"

This one is new, it is a benchmark-design finding, and I have not seen any local-model leaderboard
report it.

qwen3.5-4b ran three times at temperature 0: 31, 33, 33. Ordinary-looking noise. Then I
looked at which tasks failed:

r2 and r3 have byte-identical failure lists - 23 tasks, same ones. Perfectly reproducible.
r1 differs from both on ten tasks.
r1 was run on a different day. r2 and r3 were run consecutively in one session.

And the confirmation: a fourth run had been started on r1's day and killed partway through. Over
the 33 tasks it graded, it matches r1 exactly. So day one has two agreeing runs, day two has
two agreeing runs, and the two sessions disagree with each other.

⇒ ★ Every determinism claim in this campaign is scoped to a single session. e2b's "31/56
three times, range 0, identical failure list" and e4b's byte-identical replicates were all
gathered with the replicates consecutive inside one session. That design cannot distinguish
"deterministic" from "deterministic within a session". This is the first cross-day replicate I
ever ran, and it moved.

I do not know the mechanism yet. Candidates: the model server was restarted between sessions and
something non-obvious differs in its state; a driver or engine load-order effect; machine load
(see the next result, which accounts for 3 of the 10 flips but not the other 7). What I can say is
that running your replicates back-to-back will understate your error bar, and that is exactly
how everybody runs replicates, including me.

If you take one methodological thing from this piece, take this: space your replicates across
sessions, or state that you did not.

Result 6: the aggregate is stable-ish; the failure list is not

The champion configuration, six runs at matched constants: 49, 50, 50, 50, 48, 52. Median 50,
range 4.

Now look at which tasks failed:

run	failures
1	Q03 Q05 Q25 Q45 Q50 Q51 Q52
2	Q03 Q05 Q13 Q25 Q51 Q52
3	Q03 Q05 Q13 Q25 Q51 Q52
4	Q03 Q05 Q13 Q26 Q36 Q51
5	Q01 Q03 Q05 Q13 Q25 Q46 Q49 Q51
6	Q03 Q25 Q46 Q52

Thirteen distinct tasks rotate through the failure slots. Exactly one - Q03 - failed all six
times. Frequencies: Q03 6/6, Q51 5/6, Q25 5/6, Q05 5/6, Q52 4/6, Q13 4/6, Q46 2/6, and six
tasks that failed exactly once. This is a model sitting at a stable competence level with a pool
of borderline tasks resolving stochastically. It is not a fixed list of things it cannot do.

So almost every "this local model can't handle X" claim you read is derived from a single run,
and on this evidence that claim is unfounded roughly 12 times out of 13.

I know because I made exactly that mistake. Three runs in, I had confidently written down a
"consistent failure set" of five tasks. The next run passed two of them and produced two
first-time failures I had never seen. My five-task set was a three-sample artifact.

It cost me two wasted evaluations. I built a cheap screening gate on those five tasks; it
green-lit two candidate configurations, both of which then tied the champion exactly on the full
battery. The gate was measuring rotation, not quality.

Worse, when I rebuilt the screen properly - 12 tasks, scored in aggregate, with a rejection
threshold derived from a bound rather than picked - it still failed. The bound assumed a
challenger would pass everything outside the screen. Three different models broke it, one by
eight tasks. A subset screen now gets used only to reject fast, never to promote, bound, or crown.

If you benchmark local models: score in aggregate, over more than one run, and never conclude
anything from the identity of a failing task.

Part 3 - what the box did

Result 7: KV cache quantisation and context size moved wall-clock 2.8×, and the score not at all

A popular claim is that q8 KV cache costs quality against fp16. I tested it twice:

config	ctx	score	wall-clock
KV q8_0	32768	50/56	23.2 min
KV f16	32768	50/56	26.2 min
KV f16	65536	50/56	58.8 min

+0 quality at either context size, and fp16 was slower in both - 1.13× at matched context,
2.79× once context went to 65536 and VRAM hit 96.7%.

Note carefully what that 2.8× actually was. It was not KV precision - it was context size
pushing VRAM to the edge. I only know because the second fp16 run changed one variable; the
first had moved two at once and would have supported the wrong conclusion, confidently.

The embarrassing part: I had been daily-driving that slow corner for two days, on the strength of
a promising subset-screen result that turned out to be the rotation from Result 6.

There is a matching cliff on the other side. Splitting MoE experts between GPU and CPU has a
usable band of about 15.1–15.6 GB on a 16.3 GB card, and both errors are expensive. Too many
experts on CPU strands the card: one model at a bad split ran 113 s/task and at a good one 75
s/task - 35 minutes saved on a single row, with no score change. Too few thrashes into Windows
shared memory: at 97.7% VRAM one task took 306 seconds, and after re-splitting, 37 seconds.
8.3×.

Result 8: generation-speed probes do not predict end-to-end speed

I have a standard probe: three prompts, streaming, temperature 0, median tok/s, against a 15 tok/s
floor.

It was wrong twice, in opposite directions. One configuration probed faster than the champion
(73 vs 70 tok/s) and then ran 2.48× slower on same-verdict tasks. Another probed 1.74× faster
after a tuning change and delivered 2.2× end-to-end.

The reason is structural: an agent battery is dominated by prompt processing over a growing KV
cache, not by token generation on short prompts. The probe measures the thing that is not the
bottleneck.

Use generation-speed probes as a floor for rejecting hopeless candidates. Never cite one as
evidence a configuration is fast.

Result 9: my wall-clock-graded tasks were measuring the box, not the model

Four of the 56 tasks grade performance by wall-clock - naive vs linear-time, sized for a wide
separation. Comparing qwen3.5-4b's cross-day replicates exposed what that really measures:

task	day 1	day 2	verdict
Q12 (perf, rust)	90 s	24 s	FAIL → PASS
Q32 (perf, js)	51 s	29 s	FAIL → PASS
Q41 (perf, ts)	-	-	flipped the other way

Identical weights, identical config, 3.75× on the same task. Day 1's whole pass ran 27% slower
because I was actively using the machine. Those verdicts measured my box.

This is a real defect in my corpus and I would rather report it than quietly widen the thresholds.
Anyone grading agent performance by wall-clock inherits it. The fixes available are: run perf tasks
only on a verified-idle box, record a load probe per run and reject contaminated runs, or grade
complexity structurally instead of by clock. I have not picked one yet.

⚠️ It does not explain the session split in Result 5: 7 of the 10 flipped tasks are not perf
tasks. Machine load is one contaminant, not the mechanism.

Result 10: the failure mode a score table hides

Several models lost tasks not to bad reasoning but to not being able to emit a valid file:

gpt-oss-20b lost 3 of 16 failures this way in its first run, and five in its worst - two TypeScript tasks dying on a bare top-level return, one writing literal \n escape sequences into a .mjs.
north-mini-code-1.0 lost 2 the same way - an unterminated triple-quoted string, and a Rust E0425.
granite-4.1-8b likewise.
devstral reached for a crate that was not vendored, in an offline sandbox.

These are edit-mechanics and environment-awareness failures, not intelligence failures, and they
are invisible in a bare pass rate. They are also the most actionable thing here for anyone building
an agent: the model that reasons well but cannot reliably write a file is worse in practice than
its benchmark score suggests - and, per Result 4, it is also the model whose score you can trust
least.

It is the strongest argument I have that agent benchmarks should grade mechanics separately from
reasoning. Mine currently does not, and that is a gap.

Part 4 - the method lesson that cost the most

The spec for this benchmark had always listed "held constant for every row: ctx 32768, KV q8_0,
parallel 1". Nothing ever recorded or verified it.

The KV cache setting in LM Studio is sticky, global, per-model, survives unloads, is not a load
flag, and is not reported by lms ps --json. It silently flipped to fp16 on my box and stayed
there for two days. Rows from before and after were compared as though they were comparable.

A held-constant nobody measures is not held. Every run now emits its actual ctx, KV type,
parallelism, VRAM, GPU/CPU expert split, model path, agent version and binary mtime into a
metadata file, and historical rows are explicitly marked inferred rather than measured so a
reconstruction can never be cited as evidence.

Enforcement caught more than it had any right to. A pre-load config writer found:

model	field	found	should be
gpt-oss-20b	`numParallelSessions`	4	1
qwen3-coder-30b	ctx	8096	32768
nemotron-3-nano-omni	ctx	24000	32768
qwen3.5-4b	ctx	32800	32768
qwen3.5-4b	KV cache	unset → f16	q8_0

The numParallelSessions: 4 one is my favourite, because it splits the KV allocation across four
slots behind an lms ps that proudly reports CONTEXT 32768. Two older numbers I had been citing
as priors turned out never to have been measured at the benchmark's own constants.

Three ops traps each came within minutes of publishing a fake row:

lms load never creates the per-model config file. A never-configured model does not have one, so the pre-seeder has nothing to correct, and parallelism silently defaults to 4. Only opening the model in the GUI once creates it.
lms load fuzzy-matches when no key matches exactly. The bare id I had written in every note for months - qwen3.6-35b-a3b-mtp - silently loaded a different model, because an earlier fix had given a sibling quantisation its own repo directory and the resulting key extended the champion's prefix. There is no bare key; the real one ends @iq3_s. Caught about 40 seconds into a run that would otherwise have looked completely normal.
The GUI holds a stale copy of the config and can write it back. After correcting a config on disk, the CLI and the JSON both read the new value while the GUI still displayed the old one - and saving from the GUI would have pushed the stale value back over the fix.

The general lesson, and the reason this section exists at all:

A benchmark's integrity failures do not look like errors. They look like successful runs.

The engine control - asked, and answered

Between the early rows and the late ones, LM Studio auto-updated its inference engine from 2.25.2
to 2.27.1 at 01:21, with auto-update on by default. So I owed a control: re-run the champion,
unchanged, on the new engine.

The first control scored 48/56 against a baseline whose runs had never gone below 49, with two
never-before-failed tasks and 1.33× slower wall-clock. Two signals pointing the same way. It would
have been very easy to write "the engine update cost 2 points".

The second control scored 52/56 - the highest the champion has ever scored - in 21m23s, its
normal speed. Neither the deficit nor the slowdown reproduced. Across all six matched runs the
median is still exactly 50. No engine effect is established, and nothing gets re-based.

I am reporting the sequence rather than just the conclusion, because the one-run version of this
was a publishable-looking finding that was simply wrong, and it is the same lesson as Result 6
arriving from a different direction. Your inference engine is still a variable: pin it, record it,
and expect it to update itself while you sleep - mine did, and its auto-delete setting can
garbage-collect the old engine you would need to reproduce anything.

Honest limitations

One box, one card. RTX 5060 Ti 16 GB. VRAM-resident vs spilled is the dominant speed variable here and yours will differ.
56 tasks is small, and seven of the sixteen configurations are single runs. Those rows carry an error bar I can now size but did not measure for them.
Replicates are mostly within-session - see Result 5. The one cross-session replicate I have moved by two points and by ten task-verdicts.
It is a short-horizon instrument. Iteration depth per task: p50 = 4, p90 = 8, max 15; only 5 of 56 tasks reach 9 or more. It grades first-shot code quality. It cannot see context compaction, prefix-cache behaviour, model escalation, or permission handling - the harness runs with auto-approve, so those paths are never exercised. Anyone "proving" a compaction fix on this benchmark is measuring net zero and concluding wrongly.
The top of the range is saturating. The crown sits at 55/56; against the two strongest gemma configs the corpus is nearly out of headroom. Difficulty and discrimination turned out to be nearly orthogonal - the hard tail I added moved the champion by one point, while its single most reliable failure is a three-character guard on an empty range.
But it is not saturated in general. Across all 34 ranking runs there are 454 task-failures, 51 of 56 tasks defeat somebody, and only five are never failed by anyone (Q21, Q31, Q33, Q39, Q55). Adding the two weakest models to the sweep dropped that from seven to five.
Model licences vary and I have not audited every one for score-publication terms.

The corpus is fully open - with a contamination date

The corpus works because the hidden tests and reference solutions are hidden. Publishing them
destroys the instrument for every model trained afterwards. Publishing nothing makes every number
above an unverifiable claim.

I had planned a split - ship 15 tasks complete, seal 41. That plan is moot: the full corpus has
been publicly cloneable since 2026-07-25, on a branch of a public repository, hidden tests and
reference solutions included. I found this while preparing to publish. Rather than force-push over
it and pretend otherwise, the honest move is to state it precisely:

Contamination date: 2026-07-25. Every model in the table above was released before that date
and therefore could not have trained on this corpus. All 36 runs stand. Any model released after
it must be treated as potentially contaminated on these 56 tasks.

So: everything ships - all 56 fixtures, all hidden verifiers, all reference solutions, the
harness, the per-run metadata, and RESULTS-q56.csv with every run's full failure list. Clone it
and check any number in this piece.

That makes Q56 a one-shot instrument against future models, which is a real cost and I am not going
to dress it up. The successor corpus is being built sealed, and the design lesson is one I would
pass on to anyone building a benchmark they intend to keep: decide your publication posture before
the first push, not after 36 runs. A public repo is public in every branch.

What survives publication completely is the part I actually care about: the methodology and the
measurement results. That leaderboards anti-correlate, that error bars are model properties, that
determinism is within-session, that speed probes mispredict, that a held-constant nobody measures is
not held - none of that depends on the tasks staying secret.

What I would want from readers

The harness runs from any clone and takes a model id, so "bench your own model" is a real ask
rather than an aspiration. The three most useful contributions, in order:

A cross-session replicate. Run any model three times in one sitting, then a fourth a day later. Result 5 rests on a single model and would be either confirmed or killed by one more.
A second high-variance model. gpt-oss-20b is the only model I have measured with a large range. If the failure-mode theory in Result 4 is right, any model that loses tasks to unparseable output should swing similarly.
KV q8 vs fp16 on a card where the model is not VRAM-resident - Result 7 may well be a resident-model artifact.

The corpus, the harness, all 36 runs and every run's failure list are at github.com/mrdushidush/claudette - MIT OR Apache-2.0.

How I Run 88% of AI Coding Tasks for Free on a $300 GPU (and Built a C&C Red Alert UI for It)

david — Thu, 12 Feb 2026 20:04:33 +0000

`# I Route 88% of AI Coding Tasks to a Free Local Model — Here's What I Learned

Running AI coding agents through cloud APIs gets expensive fast. Claude Sonnet at ~$0.04/task, Opus at ~$0.075 — it adds up when you're running hundreds of tasks.

So I built a system that routes 88% of tasks to a free local model and only escalates to paid APIs when necessary. Then I wrapped it in a Command & Conquer Red Alert-style interface because… I grew up in the 90s.

The Cost Problem

Most AI coding agent frameworks send everything to the best model available. But "create a function that adds two numbers" doesn't need the same model as "implement an LRU cache with O(1) operations."

I tested this systematically — 40 coding tasks scored on a complexity scale of 1–9, all executed by a local 7B parameter model running on a $300 used GPU:

Complexity	Example Tasks	Success Rate
C1–2	Add function, greet function	100%
C3–4	Parse CSV, validate emails, factorial	80–100%
C5–6	Calculator with history, prime checker	60–100%
C7–8	Merge sorted lists, binary search, word frequency	80–100%
C9	LRU cache, stack class, RPN calculator	80%

Result: $0.002 average cost per task instead of $0.04 — that's 20x cheaper.

The 7B model handled everything from trivial one-liners to LeetCode-medium problems. It even added type hints unprompted on some solutions. Only multi-class architectural tasks needed escalation to cloud APIs.

How the Routing Works

The system scores every task on a 1–10 complexity scale using dual assessment:

Task → Complexity Assessment ├─ Rule-based: keyword matching, structural analysis └─ Haiku AI: semantic understanding (~$0.001/call) ↓ Smart weighting: if Haiku rates 2+ higher, trust Haiku ↓ ├─ C1–6 → Ollama (free, local GPU) ├─ C7–8 → Haiku (~$0.003/task) ├─ C9–10 → Sonnet (~$0.01/task) └─ Decomposition → Opus (review only, never writes code)

The complexity scoring draws from Campbell's Task Complexity Theory in organizational psychology, adapted for code tasks:

Component complexity — How many steps, files, and functions are expected?
Coordinative complexity — How many dependencies exist between parts?
Dynamic complexity — How much ambiguity and decision-making is required?

The smart weighting was a key breakthrough. The rule-based router uses keyword matching ("LRU", "cache", "linked list"), but sometimes misses semantic complexity. A cheap Haiku call provides that semantic understanding. When Haiku rates a task 2+ points higher than the rules, we trust Haiku's score directly instead of averaging — this prevents complex tasks from being under-routed to Ollama.

Hard-Won Lessons Running Ollama in Production

These took weeks of debugging. If you're building anything agentic with local models, this might save you some pain.

1. Temperature = 0 is mandatory for tool calling

This was the single biggest improvement. Small models with temperature > 0 will randomly output raw code instead of calling tools through the proper function-calling interface. The model might generate a perfect Python function… and dump it into stdout instead of calling file_write.

Temperature 0 gives deterministic, reliable tool usage. It took our success rate from ~60% to 90%+ overnight.

2. Context pollution is real (and sneaky)

After ~5 consecutive tasks on the same Ollama instance, the model starts generating syntax errors — missing quotes, unclosed parentheses, garbled output. The accumulated context from previous tasks bleeds into new ones.

The fix is surprisingly simple: a 3-second rest delay between tasks, plus a full memory reset every 3 tasks. This alone took us from 85% to 100% success rate on C1–C8 complexity tasks.

We even built a cooldown system with WebSocket events so the UI shows when an agent is "resting" between tasks.

3. 7B > 14B on 8GB VRAM (counterintuitive)

I tested qwen2.5-coder:14b-instruct-q4_K_M expecting better results from the larger model. Got a 40% pass rate vs 95% for the 7B model.

Why? The 14B model weighs in at ~9GB. On an 8GB VRAM card, it overflows into system RAM. CPU offloading makes inference slow enough that tool calling breaks down — the model times out or generates truncated responses.

The 7B model sits at ~6GB VRAM with room for context and tools. No CPU offload needed. If you have 8GB VRAM, 7B is your ceiling. Don't go bigger.

4. Agent personas work surprisingly well

This one surprised me the most. Giving the model a "CodeX-7" elite military identity with a specific pattern — "one write, one verify, mission complete" — plus three concrete examples of ideal 3-step execution dramatically improved task completion.

Without the persona, the model would often loop: write code, read it back, rewrite it, read it again. With the persona, it follows the trained pattern: write the file, run the test, report results. Done.

The technical explanation is probably that the persona plus examples act as strong few-shot conditioning, biasing the model toward a specific execution trajectory. But honestly, it also just makes the logs more fun to read.

The Fun Part: C&C Red Alert UI

Because staring at terminal logs is boring, I built a Command & Conquer Red Alert-inspired interface:

Bounty board — Task cards with complexity badges and priority colors
Active missions strip — Real-time agent health indicators (green = idle, amber = working, red = stuck)
Tool log — Terminal-style feed of every file_write, shell_run, file_read
Agent minimap — Visual representation of agents with connection lines
Voice feedback — "Conscript reporting!" when an agent picks up a task, "Shake it baby!" on completion
Cost dashboard — Real-time cost tracking with daily budget limits and token burn rate

Every agent action triggers a C&C voice line. When an agent gets stuck in a loop, a warning klaxon plays. It's ridiculous and I love it.

Architecture

The whole system runs in Docker:

UI (React:5173) → API (Express:3001) → Agents (FastAPI:8000) → Ollama/Claude ↓ PostgreSQL:5432

React UI — Real-time WebSocket updates, no polling
Express API — Task routing, cost tracking, budget enforcement, rate limiting
FastAPI + CrewAI — Agent orchestration with tool wrapping and execution logging
Ollama — Local LLM with GPU passthrough
PostgreSQL — Tasks, execution logs, code reviews, training data

Every tool call is captured to the database with timing, token usage, and cost. The system detects stuck tasks (>10 min timeout) and automatically recovers them. Loop detection prevents agents from repeating failed actions.

Tasks can run in parallel when they use different resources — an Ollama task and a Claude task can execute simultaneously, yielding a 40–60% speedup on mixed-complexity batches.

Try It

`bash
git clone https://github.com/mrdushidush/agent-battle-command-center.git
cd agent-battle-command-center
cp .env.example .env

Add your ANTHROPIC_API_KEY to .env

docker compose up --build

Open http://localhost:5173

Requirements:

Docker Desktop
NVIDIA GPU with 8GB+ VRAM (recommended) — or CPU-only mode (slower)
Anthropic API key (only needed for complex tasks — Ollama tasks are free)

The first startup takes ~5 minutes to download the Ollama model.

What's Next

The project is fully open source (MIT). Some things I'm working on:

Multi-language support — Currently Python-only; adding JavaScript/TypeScript
Demo mode — Simulated agents so anyone can try the UI without a GPU
Docker Hub image — One-command deploy without building
More voice packs — Community suggestions include StarCraft and Age of Empires

There are 8 good-first-issues open if you want to contribute. We already merged our first community PR (keyboard shortcuts) on day 3.

GitHub: agent-battle-command-center

Come hang out in Discussions if you want to chat about AI agent orchestration, cost optimization, or which RTS game had the best unit voice lines.

"One write, one verify, mission complete." — CodeX-7`