A quantization "quality ladder" where the full-precision model performs worse than its own 4-bit version is not really measuring quality.
It is measuring noise.
I kept running into this while looking at quantized versions of Unlimited-OCR, Baidu's new 3B OCR model released under the MIT license. Most quantized repos had one of two problems: either they shipped no quality numbers at all, or the numbers were difficult to trust.
Some baselines were losing to their own quants. Some character error rates were above 100%. In many cases, runaway repetition loops were silently inflating the averages.
So I built the ladder I wanted to read.
I tested the two runtimes people are most likely to use locally on a Mac:
- MLX for Apple Silicon
- GGUF through llama.cpp
Every number below comes from a reproducible evaluation run, and the full harness is open source.
The results were useful, but two findings genuinely surprised me.
Why I Did This
Quantization benchmarks are easy to make look better than they are.
A model can pass a quick smoke test on one clean page and still fall apart on dense text, invoices, tables, or small fonts. For OCR, that matters. The difference between "mostly works" and "silently corrupts numbers" is not small.
I wanted to answer a more practical question:
If I run this model locally, how much quality do I actually lose at each quantization level?
Not in theory. Not by vibes. On a controlled OCR task with known ground truth.
The Methodology Most People Skip
The hard part was not only quantizing the model.
The hard part was making the evaluation readable.
Three choices made the results much cleaner.
1. Use Exact Ground Truth
Instead of scoring against a real-world form dataset with sparse annotations, I generated a synthetic OCR corpus.
The corpus contains:
- 24 rendered pages
- Three difficulty tiers:
- clean prose
- dense small-font pages
- digit-heavy invoices
- deterministic seed
- ground truth known character-for-character
This matters because sparse annotations can make OCR models look worse than they are.
For example, if the dataset only labels a few fields but the model transcribes the entire page, the character error rate can be inflated for the wrong reason.
For this test, I wanted exact text-to-text comparison.
2. Compare Each Quant Against Its Own Runtime Baseline
Every quantized model is measured against the full-precision BF16 conversion in the same runtime.
That means:
- GGUF quants are compared against GGUF-BF16 under llama.cpp
- MLX quants are compared against MLX-BF16 under mlx-vlm
No cross-runtime baselines.
This is important because different runtimes can fail differently, even with the same model and prompt.
3. Surface Loops Instead of Hiding Them
All decoding runs use:
- temperature 0
- repetition suppression turned off
That second part is intentional.
If quantization makes the model unstable, I want that instability to show up clearly as a loop page. I do not want it quietly buried inside an average.
Loop pages are flagged using the output/reference length ratio. I also report loop counts separately so it is easier to tell the difference between normal OCR degradation and total decoding collapse.
Results
GGUF Ladder
Measured against GGUF-BF16 using llama.cpp on an M3 Max.
| Variant | Size | Overall CER | Δ vs BF16 | Loops |
|---|---|---|---|---|
| BF16 | 5.47 GiB | 0.78% | — | 0/24 |
| Q8_0 | 2.91 GiB | 0.78% | +0.00 | 0/24 |
| Q6_K | 2.43 GiB | 0.78% | +0.00 | 0/24 |
| Q5_K_M | 2.07 GiB | 0.74% | within noise | 0/24 |
| Q4_K_M | 1.82 GiB | 15.64% | +14.86 pp | 1/24 |
| Q4_0 | 1.59 GiB | 44.02% | +43.24 pp | 2/24 |
MLX Ladder
Measured against MLX-BF16 using mlx-vlm.
| Variant | Size | Overall CER | Δ vs BF16 | Loops |
|---|---|---|---|---|
| BF16 | 6.67 GB | 1.62% | — | 0/24 |
| 8-bit | 3.92 GB | 1.62% | +0.00 | 0/24 |
| 6-bit | 3.19 GB | 1.62% | +0.00 | 0/24 |
| 4-bit uniform | 2.45 GB | 123.61% | +121.99 pp | 7/24 |
| 4-bit mixed | 2.70 GB | 20.86% | +19.24 pp | 1/24 |
Two things stand out.
Finding 1: 4-Bit Was Not the Sweet Spot
A lot of people treat 4-bit quantization as the default sweet spot.
That rule often comes from perplexity studies on large dense chat models.
Unlimited-OCR is not that.
It is a 3B Mixture-of-Experts OCR model with roughly 570M active parameters. It is also doing a precision-heavy task where small errors matter. That is exactly the kind of setup where the usual 4-bit heuristic can break.
In this test, there was no measurable loss down to 6-bit.
Q8_0 and Q6_K reproduced the BF16 recognized text identically on all 24 pages.
Then came the cliff.
The damage was concentrated on dense, small-font pages. Clean pages still stayed under 2% CER even at 4-bit, which explains why a casual one-page test can be misleading.
An easy page will not catch this.
You need a hard tier. You also need a real baseline.
Finding 2: The Kind of 4-Bit Quantization Matters
Uniform 4-bit was bad.
Mixed-precision 4-bit was much better.
In GGUF, mixed precision beat uniform 4-bit by about 2.8×.
In MLX, it beat uniform 4-bit by about 5.9×.
But it did not fully fix the problem. It softened the cliff. It did not remove it.
I also ran an MLX ablation to understand where the recovery was coming from. I kept only the vision-to-language projector in float while leaving everything else identical to uniform 4-bit.
It still looped on 7 out of 24 pages.
That suggests the recovery is happening on the decoder side: attention projections, embeddings, and the LM head.
It is not mainly coming from the visual path.
The Finding I Did Not Expect: Repetition Suppression Is a Tradeoff
The official Unlimited-OCR pipeline uses repetition suppression with:
no_repeat_ngram_size=35
llama.cpp has a related mechanism through the DRY sampler.
Since only the 4-bit quants were looping, I re-ran the affected GGUF model with upstream-style DRY settings.
At first, it looked like a fix.
Q4_K_M overall CER improved:
15.64% → 10.47%
The worst loop pages were rescued.
But the tradeoff was not clean.
Pages that were previously fine got worse:
3.88% → 10.86%
The biggest damage showed up on legitimately repetitive documents, such as template invoices.
Even worse, DRY introduced a brand-new catastrophic loop on a page that was fine without it:
0% → 218% CER
So repetition suppression is not a free fix.
It trades one failure mode for another.
If your documents contain honest repetition — tables, invoices, forms, templates — a default-on DRY sampler may quietly reduce accuracy.
That is why the primary ladder above is scored with repetition suppression off.
Bonus: A First Look at R-SWA
Unlimited-OCR's headline mechanism is R-SWA, or Reference Sliding Window Attention.
The goal is to keep KV cache memory constant during long-horizon parsing.
No MLX port implements R-SWA yet, but llama.cpp has an open PR, #24975, that does. This is not my work. Full credit goes to the PR author.
The same GGUF file loads with:
n_swa = 128
on the PR branch, compared with:
n_swa = 0
on mainline.
I tested:
- same weights
- same 24 pages
- both attention regimes
The recognized text was identical on 21 of 24 pages.
On the three pages that differed, R-SWA produced a small net improvement. It did not introduce any loops.
Important caveat: this only tests single-page fidelity parity.
The real reason R-SWA exists is multi-page, constant-memory parsing. That is not tested here.
Two Runtime Details Worth Knowing
A couple of smaller runtime findings are also worth mentioning.
First, the same prompt can fail differently depending on runtime.
For example:
Free OCR.
This prompt emits immediate EOS in llama.cpp, but runs away into a repetition loop in mlx-vlm 0.6.3.
Second, mlx-vlm 0.6.3 loads this tokenizer through a slow path that skips byte-level BPE decoding.
That means raw output contains markers like:
Ġ
Ċ
You need to decode those yourself.
llama.cpp detokenizes correctly.
Reproduce It
Everything is public:
- Six model repos and a collection on Hugging Face
- Evaluation harness on GitHub — corpus generator, CER/WER scorer, loop diagnostics (the scoring core is backend-agnostic; the runner is MLX-specific)
The exact command behind the GGUF numbers is:
llama-mtmd-cli \
-m unlimited-ocr-Q5_K_M.gguf \
--mmproj mmproj-unlimited-ocr-F16.gguf \
--image page.png \
-p "document parsing." \
--chat-template deepseek-ocr \
--temp 0 --repeat-penalty 1.0 --flash-attn off \
-n 2600 -c 16384
Caveats
These numbers are specific to:
- this model
- this corpus
- synthetic clean renders
- English documents
- single-page evaluation
- these runtime builds
If your documents look different, run the harness on them.
That is what it is for.
For quantization, the useful question is not:
Does it work?
The useful question is:
How much quality do I lose on my data, and can someone else verify it?
That is the difference between a number and evidence.
Top comments (0)