Simon Willison’s local test found 23.57% numeric accuracy without reasoning, versus 167 correct answers in a smaller 169-case reasoning run.
Qwen3.8 27B achieved 23.57% numeric accuracy across 5,070 addition problems when reasoning was disabled, according to a local experiment published by Simon Willison on October 4. The model had to add two positive integers and return the exact result using English words only. Although just 1,195 answers were numerically correct, 96.17% complied with the requested output format.
Willison ran the quantized Qwen3.8-27B-Q4_K_M.gguf model on an NVIDIA DGX Spark. The test covered every ordered pairing of operand lengths from one to 13 digits, producing a 13-by-13 matrix of 169 combinations. With reasoning disabled, each combination received 30 attempts, for 5,070 cases overall.
Performance deteriorated as the operands grew. Willison’s report gives 97.04% accuracy for operands between one and three digits, compared with 6.44% for those between ten and 13 digits. The gap between numeric accuracy and 96.17% format compliance is important: in this experiment, producing a properly worded response was much easier for the model than calculating the correct sum.
A second run with medium reasoning enabled produced 167 correct answers from 169 attempts, or 98.82%. That result came from a much smaller sample because reasoning took substantially longer: Willison used only one fixed pair for each operand-length combination rather than 30. He explicitly cautioned that another one-shot run would likely produce different results.
Published traces from some larger calculations show the model aligning the operands by place value, adding from right to left and carrying digits between columns. Those traces provide a plausible explanation for the improvement: the reasoning mode exposed a conventional long-addition procedure instead of immediately producing an answer. They do not establish that the same accuracy would hold with different numbers or generation settings.
The two headline results therefore do not have equal statistical weight. The reasoning-disabled series used 30 samples in every cell, while the medium-reasoning pilot used a single sample in each. The experiment demonstrates a large behavioral difference for this local quantized model under the stated protocol, but repeated reasoning runs at comparable scale would be needed to assess how stable that difference is.
| Mode | Number of tasks | Result |
|---|---|---|
| Reasoning disabled | 5,070 | 1,195 correct, 23.57% |
| Medium reasoning | 169 | 167 correct, 98.82% |
Sources
Originally published by AI Feed: Qwen3.8 27B Scored 23.57% on Addition Without Reasoning.
Top comments (0)