DEV Community

Ikkun
Ikkun

Posted on Originally published at agentjournal.dev

The 2B model scored 10%. 93% of its answers were right

I expected small local models to be worse than a hosted decision API at Japanese classification. I did not expect most of the gap to be something other than judgement. On the hardest of three tasks one model scored 10.8% overall while getting 27 of the 29 answers it produced right. The other 221 rows it never answered.

What I ran, and what the accuracy number counted

Three Japanese tasks, 250 identical rows each. livedoor is 9-class news-topic classification (mean 1,308 characters), rakuten is 2-class review polarity (165), chabsa is 3-class financial-sentence polarity (100). Chance is 11.1%, 50.0% and 33.3%. Every system got the same prompt asking for one label as JSON, and a row counted correct only if the parsed label matched gold. Unparseable output counted as wrong.

System livedoor rakuten chabsa
Jev (hosted API) 76.8% 94.4% 74.0%
Gemma 4 26B-A4B (local MoE) 65.2% 94.4% 75.2%
SemIf (Qwen3.5-4B) 68.8% 92.0% 64.0%
phi-4-mini 3.8B Q4 38.0% 89.6% 71.6%
MiniCPM5-2B Q4 36.4% 89.2% 61.6%
LFM2.5-2.6B Q4 10.8% 75.6% 26.4%

How many rows actually produced an answer?

This is the column I should have read first:

Model livedoor rakuten chabsa
phi-4-mini 3.8B 243/250 249/250 250/250
MiniCPM5-2B 197/250 250/250 250/250
LFM2.5-2.6B 29/250 199/250 87/250

Score only the rows that answered and the ranking moves. LFM2.5-2.6B's livedoor figure becomes 93.1% (27/29), and its rakuten figure 95.0%, above phi-4-mini's 89.6% and level with the hosted API's 94.4%. These models were not failing to judge. They failed to finish.

Why a small model returns nothing

I reproduced it on the mini-PC on 2026-10-26 (llama.cpp 60081bb, CPU-only) on the same livedoor row. With a 512-token budget the response body came back empty with finish_reason: length and 512 completion tokens spent. The tokens went to a populated reasoning_content field holding the model's step-by-step working: it ran out of budget before the answer.

Raise the budget to 2,048 and that row returns {"label": "MOVIE ENTER"}, correct, finish_reason: stop, 790 tokens. Same weights and prompt, temperature 0.

Raising the budget is not a complete fix. On the first 40 livedoor rows at 2,048: 28 answered (70%, against 11.6% at 512), but 11 still hit the cap and returned nothing, and one answered with an out-of-vocabulary label (IT Lifehack) that a strict parser rejects. Answer rate and answer validity are separate failure modes.

A reasoning-capable model spends tokens thinking, and a budget sized for the answer truncates it to nothing, which the harness scores as wrong. My earlier Gemma 4 E2B run shows the same shape at the format layer: 142 of 250 rows emitted valid JSON (56.8%), and 140 of those 142 were correct (98.6%).

Does a 2B model ever beat the API?

On accuracy, in my data, no. Paired McNemar on identical row ids, two-sided: phi-4-mini vs Jev −38.8pt on livedoor (p=1.2e-21), −4.8pt on rakuten (p=0.029), −2.4pt on chabsa (p=0.59); MiniCPM5-2B −40.4pt / −5.2pt (p=0.0072) / −12.4pt (p=0.0042); SemIf 4B −8.0pt (p=0.0012) / −2.4pt (p=0.070) / −10.0pt (p=0.014). The gap narrows as the task gets shorter and more lexical: 4.8 points on the 165-character polarity task, 39 to 40 on the long 9-class one.

Costs do not favour the small side either. Summed per-row latency over all 750 rows: MiniCPM5-2B 20 minutes, phi-4-mini 39, SemIf 4B 40, LFM2.5-2.6B 191.

Meanwhile 2,000 Jev requests across eight experimental arms produced zero unparseable responses. A closed-choice API cannot format-fail; a chat model can, and that difference dominates the table.

Measure answered rows before accuracy: a number averaging "wrong" and "silent" tells you nothing about which one you are buying. Background is in the zero-shot versus trained comparison and the CPU throughput limits.

Measured 2026-09-19 to 2026-10-26 on a Ryzen 9 7940HS mini-PC, llama.cpp 60081bb, Q4_K_M weights, temperature 0, 250 rows per task.

If you have sized a token budget for a local model's answers rather than its thinking, what budget actually cleared it?


Originally published at agentjournal.dev.

Top comments (2)

Collapse
 
recallrundev profile image
RecallRun •

The answered-rows table should be the first table in more benchmark posts. 27 of 29 vs 10.8% is a good example of an average hiding two different failures.

A question on the empty rows: did they cluster by input length? livedoor's mean is 1,308 characters against 100-165 for the other two, and LFM's answer rate falls in roughly that order (29, 87, 199 of 250 if I read the table right, though chabsa is shorter than rakuten so it isn't only length). If longer inputs mean longer reasoning, a budget that scales with input length might clear more rows than a flat 2,048, and it would be cheaper on the short tasks.

The same "silent counts as wrong" problem shows up in retrieval evals: an "I couldn't find this" gets scored as a miss unless abstentions are counted separately, so a system that guesses looks better than one that abstains.

Collapse
 
officialmailkr profile image
오피셜메일 •

전체 기준 정확도와 응답한 행만의 정확도를 나눠 보여줘서 실패 원인을 이해하기 좋네요. 특히 토큰 한도를 늘려도 미응답과 허용 목록 밖 레이블이 따로 남는다는 점이 중요해 보여요. 다음 비교에서는 올바른 최종 응답 한 건을 얻기까지 든 재시도·시간을 함께 보면, 운영 시 어떤 실패를 먼저 줄일지 판단하기 더 좋을 것 같습니다.