Ask anyone tuning local LLMs where quality lives and you'll hear about quantization. Q4 versus Q6 versus Q8, perplexity curves, "never go below Q5 for reasoning." It's the knob everyone debates because it's the knob with numbers attached.
Here are my measured results on a 32B model, same benchmark suite, fixed seed:
Quantization comparison (20-question MCQ suite, seed 42):
- Q8_0: 15/20 correct, 19.6 tokens/sec, ~39.5 GB
- Q6_K: 15/20 correct, 20.3 tokens/sec, ~29 GB
A full quantization step moved accuracy not at all. Zero questions. The smaller quant was actually faster (less memory traffic) and left 10 GB more room for KV cache. That one table set my fleet's default: Q6_K as the daily driver, Q8_0 as a stretch when VRAM allows.
Now the same model, same quant, same seed, changing only the chat template:
Template comparison (scored /10):
- Model's own template: 3-4
- ChatML template: 0
- no_think mode + model template: 3
The wrong template zeroed the model. Not degraded it, zeroed it. A model that scores reliably with its own template produced nothing scoreable when wrapped in ChatML, the template half the internet's example configs default to.
Template choice had a larger effect size than any quantization decision I measured. It isn't close.
Why this surprises people
Quantization is a continuous, well-instrumented knob with academic literature behind it. Templates are a formatting detail buried in a GGUF's metadata or a server flag. One looks like engineering, the other looks like plumbing.
But think about what each one actually perturbs. Quantization adds small rounding error to every weight, and modern quant schemes are engineered to keep that error away from what matters. A wrong template perturbs the input distribution itself: the model sees token sequences that never appeared in its training in that arrangement, malformed role markers, missing control tokens. From the model's perspective, quantization is a slight blur; the wrong template is a foreign language.
The failure is also silent. A mis-templated model still generates fluent text. It doesn't crash, it doesn't warn, it just gets steadily and confidently worse at its job. If my benchmark hadn't scored outputs, I could have shipped that config and spent weeks blaming the quant, the sampler, or the model itself.
The discipline that caught it
Nothing clever caught this. A boring benchmark suite did: fixed seed, fixed question set, scored outputs, run per candidate config before promotion. The suite exists because eyeballing model quality is how you fool yourself; scores at a fixed seed are how the difference between "seems fine" and "scores zero" becomes visible.
The order of operations this experience installed:
- Verify the template first. It's the highest-leverage, least-discussed setting in local inference.
- Then choose the smallest quant that holds your benchmark scores. The savings go to context or speed.
- Distrust any quality comparison, including between whole models, that doesn't control for template. Some fraction of "model A beats model B locally" posts are template bugs wearing a costume.
Check the plumbing before you argue about the engineering. The cheap setting nobody benchmarks moved my scores more than the expensive setting everybody does.
Top comments (0)