DEV Community: Vimal Nakrani

I Tested Quantized Unlimited-OCR on Mac. 4-Bit Was Not the Sweet Spot.

Vimal Nakrani — Sat, 04 Jul 2026 14:24:14 +0000

A quantization "quality ladder" where the full-precision model performs worse than its own 4-bit version is not really measuring quality.

It is measuring noise.

I kept running into this while looking at quantized versions of Unlimited-OCR, Baidu's new 3B OCR model released under the MIT license. Most quantized repos had one of two problems: either they shipped no quality numbers at all, or the numbers were difficult to trust.

Some baselines were losing to their own quants. Some character error rates were above 100%. In many cases, runaway repetition loops were silently inflating the averages.

So I built the ladder I wanted to read.

I tested the two runtimes people are most likely to use locally on a Mac:

MLX for Apple Silicon
GGUF through llama.cpp

Every number below comes from a reproducible evaluation run, and the full harness is open source.

The results were useful, but two findings genuinely surprised me.

Why I Did This

Quantization benchmarks are easy to make look better than they are.

A model can pass a quick smoke test on one clean page and still fall apart on dense text, invoices, tables, or small fonts. For OCR, that matters. The difference between "mostly works" and "silently corrupts numbers" is not small.

I wanted to answer a more practical question:

If I run this model locally, how much quality do I actually lose at each quantization level?

Not in theory. Not by vibes. On a controlled OCR task with known ground truth.

The Methodology Most People Skip

The hard part was not only quantizing the model.

The hard part was making the evaluation readable.

Three choices made the results much cleaner.

1. Use Exact Ground Truth

Instead of scoring against a real-world form dataset with sparse annotations, I generated a synthetic OCR corpus.

The corpus contains:

24 rendered pages
Three difficulty tiers:
- clean prose
- dense small-font pages
- digit-heavy invoices
deterministic seed
ground truth known character-for-character

This matters because sparse annotations can make OCR models look worse than they are.

For example, if the dataset only labels a few fields but the model transcribes the entire page, the character error rate can be inflated for the wrong reason.

For this test, I wanted exact text-to-text comparison.

2. Compare Each Quant Against Its Own Runtime Baseline

Every quantized model is measured against the full-precision BF16 conversion in the same runtime.

That means:

GGUF quants are compared against GGUF-BF16 under llama.cpp
MLX quants are compared against MLX-BF16 under mlx-vlm

No cross-runtime baselines.

This is important because different runtimes can fail differently, even with the same model and prompt.

3. Surface Loops Instead of Hiding Them

All decoding runs use:

temperature 0
repetition suppression turned off

That second part is intentional.

If quantization makes the model unstable, I want that instability to show up clearly as a loop page. I do not want it quietly buried inside an average.

Loop pages are flagged using the output/reference length ratio. I also report loop counts separately so it is easier to tell the difference between normal OCR degradation and total decoding collapse.

Results

GGUF Ladder

Measured against GGUF-BF16 using llama.cpp on an M3 Max.

Variant	Size	Overall CER	Δ vs BF16	Loops
BF16	5.47 GiB	0.78%	—	0/24
Q8_0	2.91 GiB	0.78%	+0.00	0/24
Q6_K	2.43 GiB	0.78%	+0.00	0/24
Q5_K_M	2.07 GiB	0.74%	within noise	0/24
Q4_K_M	1.82 GiB	15.64%	+14.86 pp	1/24
Q4_0	1.59 GiB	44.02%	+43.24 pp	2/24

MLX Ladder

Measured against MLX-BF16 using mlx-vlm.

Variant	Size	Overall CER	Δ vs BF16	Loops
BF16	6.67 GB	1.62%	—	0/24
8-bit	3.92 GB	1.62%	+0.00	0/24
6-bit	3.19 GB	1.62%	+0.00	0/24
4-bit uniform	2.45 GB	123.61%	+121.99 pp	7/24
4-bit mixed	2.70 GB	20.86%	+19.24 pp	1/24

Two things stand out.

Finding 1: 4-Bit Was Not the Sweet Spot

A lot of people treat 4-bit quantization as the default sweet spot.

That rule often comes from perplexity studies on large dense chat models.

Unlimited-OCR is not that.

It is a 3B Mixture-of-Experts OCR model with roughly 570M active parameters. It is also doing a precision-heavy task where small errors matter. That is exactly the kind of setup where the usual 4-bit heuristic can break.

In this test, there was no measurable loss down to 6-bit.

Q8_0 and Q6_K reproduced the BF16 recognized text identically on all 24 pages.

Then came the cliff.

The damage was concentrated on dense, small-font pages. Clean pages still stayed under 2% CER even at 4-bit, which explains why a casual one-page test can be misleading.

An easy page will not catch this.

You need a hard tier. You also need a real baseline.

Finding 2: The Kind of 4-Bit Quantization Matters

Uniform 4-bit was bad.

Mixed-precision 4-bit was much better.

In GGUF, mixed precision beat uniform 4-bit by about 2.8×.

In MLX, it beat uniform 4-bit by about 5.9×.

But it did not fully fix the problem. It softened the cliff. It did not remove it.

I also ran an MLX ablation to understand where the recovery was coming from. I kept only the vision-to-language projector in float while leaving everything else identical to uniform 4-bit.

It still looped on 7 out of 24 pages.

That suggests the recovery is happening on the decoder side: attention projections, embeddings, and the LM head.

It is not mainly coming from the visual path.

The Finding I Did Not Expect: Repetition Suppression Is a Tradeoff

The official Unlimited-OCR pipeline uses repetition suppression with:

no_repeat_ngram_size=35

llama.cpp has a related mechanism through the DRY sampler.

Since only the 4-bit quants were looping, I re-ran the affected GGUF model with upstream-style DRY settings.

At first, it looked like a fix.

Q4_K_M overall CER improved:

15.64% → 10.47%

The worst loop pages were rescued.

But the tradeoff was not clean.

Pages that were previously fine got worse:

3.88% → 10.86%

The biggest damage showed up on legitimately repetitive documents, such as template invoices.

Even worse, DRY introduced a brand-new catastrophic loop on a page that was fine without it:

0% → 218% CER

So repetition suppression is not a free fix.

It trades one failure mode for another.

If your documents contain honest repetition — tables, invoices, forms, templates — a default-on DRY sampler may quietly reduce accuracy.

That is why the primary ladder above is scored with repetition suppression off.

Bonus: A First Look at R-SWA

Unlimited-OCR's headline mechanism is R-SWA, or Reference Sliding Window Attention.

The goal is to keep KV cache memory constant during long-horizon parsing.

No MLX port implements R-SWA yet, but llama.cpp has an open PR, #24975, that does. This is not my work. Full credit goes to the PR author.

The same GGUF file loads with:

n_swa = 128

on the PR branch, compared with:

n_swa = 0

on mainline.

I tested:

same weights
same 24 pages
both attention regimes

The recognized text was identical on 21 of 24 pages.

On the three pages that differed, R-SWA produced a small net improvement. It did not introduce any loops.

Important caveat: this only tests single-page fidelity parity.

The real reason R-SWA exists is multi-page, constant-memory parsing. That is not tested here.

Two Runtime Details Worth Knowing

A couple of smaller runtime findings are also worth mentioning.

First, the same prompt can fail differently depending on runtime.

For example:

Free OCR.

This prompt emits immediate EOS in llama.cpp, but runs away into a repetition loop in mlx-vlm 0.6.3.

Second, mlx-vlm 0.6.3 loads this tokenizer through a slow path that skips byte-level BPE decoding.

That means raw output contains markers like:

Ġ
Ċ

You need to decode those yourself.

llama.cpp detokenizes correctly.

Reproduce It

Everything is public:

Six model repos and a collection on Hugging Face
Evaluation harness on GitHub — corpus generator, CER/WER scorer, loop diagnostics (the scoring core is backend-agnostic; the runner is MLX-specific)

The exact command behind the GGUF numbers is:

llama-mtmd-cli \
  -m unlimited-ocr-Q5_K_M.gguf \
  --mmproj mmproj-unlimited-ocr-F16.gguf \
  --image page.png \
  -p "document parsing." \
  --chat-template deepseek-ocr \
  --temp 0 --repeat-penalty 1.0 --flash-attn off \
  -n 2600 -c 16384

Caveats

These numbers are specific to:

this model
this corpus
synthetic clean renders
English documents
single-page evaluation
these runtime builds

If your documents look different, run the harness on them.

That is what it is for.

For quantization, the useful question is not:

Does it work?

The useful question is:

How much quality do I lose on my data, and can someone else verify it?

That is the difference between a number and evidence.

Why did my DataFrame lose rows? Debugging silent pandas pipeline failures

Vimal Nakrani — Sat, 13 Jun 2026 17:49:27 +0000

If you've written more than a handful of pandas pipelines, you know this feeling: the row count at the end is wrong, the numbers are slightly off, and somewhere across fifteen transformation steps, something changed your data without telling you. No exception. No warning. Just a quietly wrong answer.

These are the worst bugs in data work, because they don't crash — they ship. A dashboard shows a number that's 3% low. A model trains on rows that shouldn't exist. A report goes to a client missing a region. And by the time anyone notices, the pipeline has run a hundred times.

This post is about why these failures happen, the usual (painful) way people debug them, and a small open-source tool I built called dframe-trace that automates the tedious part.

The three silent killers

Almost every silent pipeline bug falls into one of three buckets.

Rows disappear. A merge with how="inner" quietly drops every row without a match. A filter is slightly too aggressive. A dropna removes more than you intended. The pipeline still runs; it just runs on less data.

Nulls appear. A left join against an incomplete lookup table introduces blank values in the new columns. A reindex or pivot creates gaps. Downstream, those nulls become zeros, or get dropped, or silently skew an average.

Dtypes drift. A column of integers becomes floats after a merge with missing values. A date column comes back as a string. An astype does something subtly different from what you expected. Nothing breaks immediately — but a join key that flipped from int64 to float64 will silently fail to match later.

The common thread: none of these raise an error. Your code is "correct" in the sense that it executes. It's just wrong.

The usual way to debug this

When the final number looks off, most of us reach for the same tool — print statements:

df = load_data()
print(df.shape)                                   # (10000, 8)
df = df.merge(meta, on="id", how="left")
print(df.shape, df["region"].isna().sum())        # (10000, 9), 240 nulls?!
df = df[df.amount > threshold]
print(df.shape)                                   # (8800, 9)
df = df.dropna(subset=["region"])
print(df.shape)                                   # (8560, 9)

This works. It's also miserable. You're editing working code to add instrumentation, re-running the whole pipeline, eyeballing a wall of numbers, then deleting it all once you've found the culprit — until next time, when you add it all back. You're manually reconstructing information the pipeline already had and threw away.

What you actually want is to run your code once, normally, and then ask questions about what happened.

A different approach: trace first, ask later

That's the idea behind dframe-trace. Instead of declaring rules up front or instrumenting by hand, you turn on recording, run your normal code, and interrogate the trace afterward.

pip install dframe-trace

It has no required dependencies — you bring your own pandas and/or polars.

The lowest-friction way to use it patches the DataFrame methods that most often cause silent bugs, so you don't have to touch your functions at all:

import pandas as pd
from dframe_trace import trace, autopatch

autopatch.install()   # one line at the top of your script

with trace() as t:
    df = raw.merge(meta, on="id", how="left")   # recorded automatically
    df = df.astype({"id": "float64"})           # recorded automatically
    df = df.dropna(subset=["region"])           # recorded automatically

print(t.where_null_introduced("region"))   # -> "merge"
print(t.report())

The report() gives you a step-by-step diff of what each operation did:

dframe-trace report
============================================================
[0] load  (0.5 ms)
    start: 4 rows, 2 cols
[1] merge_meta  (1.4 ms)
    +cols: ['region']
    nulls region: 0 -> 1  [WARN]
[2] filter  (0.4 ms)
    rows: -1

Instead of bisecting by hand, you get a direct answer: the merge introduced the nulls in region, and a later step dropped a row. The questions you can ask map onto the three silent killers:

t.where_null_introduced("region")   # which step first added nulls to this column
t.where_rows_lost()                 # [(step_name, negative_delta), ...]

If you'd rather not patch anything globally, there's a decorator form — wrap the functions you care about with @traced("name") and run them inside the trace() block. Same recording, more explicit control.

How this differs from Great Expectations and Pandera

The Python data-validation space is crowded and mature, so it's worth being precise about where this fits.

Tools like Great Expectations, Pandera, and Hamilton check your data against rules you write in advance: "this column must never be null," "row count must stay above 1,000." They're excellent and they're the right choice when you already know what correct looks like and want to enforce it in production.

dframe-trace is the opposite philosophy: zero rules. You declare nothing. It records what every step did and lets you ask, after the fact, where something changed. It's closer to a profiler for data shape than to a schema checker.

So the rule of thumb is:

Use Pandera / Great Expectations when you know your expectations and want to enforce them.
Use dframe-trace when something is already wrong and you need to find which step did it — or when you want a cheap, always-on record of how data flows through a script.

They're complementary; nothing stops you from using both.

Catching regressions in CI

Once you've found a bug, you usually want to make sure it stays fixed. A trace can become a build-failing assertion:

from dframe_trace import trace, guards

with trace() as t:
    run_pipeline()

guards.assert_no_new_nulls(t)
guards.assert_no_row_loss(t, allow={"filter"})    # allow expected drops
guards.assert_no_silent_casts(t, allow={"astype"})

Each guard raises with a structured list of violations — "merge introduced 2 null(s) in 'region'" — so a failing build tells you exactly what regressed and where.

Is it expensive to leave on?

No, and that's deliberate. A snapshot is structural only: row count, column names, dtypes, per-column null counts, and estimated memory. No row values are ever copied or stored. Outside an active trace() block, autopatch adds a single is None check per call. That's cheap enough to leave installed in development without thinking about it.

Honest limitations

A debugging tool you can't trust is worse than none, so here's what it doesn't do yet:

Boolean-mask filtering (df[df.x > 0]) isn't auto-traced — it goes through __getitem__, which is too broad to patch safely. The row loss still shows up in the next recorded step's delta; for precise attribution, wrap that step in @traced.
groupby terminal methods aren't traced yet (it's on the roadmap).
polars support is newer than the pandas path, which is more thoroughly tested.

It's a young project and a debugging aid, not a correctness guarantee.

Try it

If you've ever lost an afternoon to a pipeline that returned the wrong number for no obvious reason, this is built for exactly that afternoon.

pip install dframe-trace

PyPI: https://pypi.org/project/dframe-trace/
Source: https://github.com/vimalnakrani08/dframe-trace

Issues and pull requests are welcome — there are tagged good-first-issues on the roadmap (groupby tracing, Mermaid lineage export, more guards) if you want to contribute. And if you try it on a real pipeline, I'd genuinely like to hear what it caught — or missed.