Ward Ed

Posted on Nov 8

olmOCR-bench on Hugging Face: Three Rows Use a Different Average, and One Moves From 13th to 7th or 19th

#ai #ocr #machinelearning #benchmark

TL;DR

On November 8, 2026 the Hugging Face olmOCR-bench board lists 24 rows from 24 repos, all higher-is-better pass rates, 0 verified. 18 rows sit on a repo's main branch and 6 come from pull requests that are still open, all opened by one account.
The board's overall is the plain average of eight columns (seven test categories plus a baseline column) for 15 of the 24 rows. Three rows, ranks 13, 20 and 23, use a different average that leaves out headers_footers, and their notes say so. Six more rows cannot be rechecked from their files: one is gated, one stores only a mean, four store fewer than eight columns.
Rank 13, LightOnOCR-2-1B, is listed at 83.2. Its own eight columns average 75.24. Drop headers_footers from everyone and it is 7th of 22. Keep it and the row is 19th of 21. The board's 13th is a mix of the two.
Rank 1 survives every recomputation I tried, and rank 2 survives all but a test-count weighting. Rank 1 leads 3 of the 8 columns, is 16th of 17 on baseline, and among the rows I could read, six different repos lead the eight columns between them.
Ranks 1 to 3 are 0.78 points apart. The dataset card's own results table puts a ± of 0.9 to 1.2 on every Overall it lists, so treat the top of this board as a tier.

What does the olmOCR-bench board on Hugging Face rank?

olmOCR-bench is a dataset of 1,403 PDF pages plus 7,010 unit tests. A test checks one property of an OCR system's markdown output: a sentence is present, a header is absent, two spans appear in the right order, a table cell sits next to its neighbour, a formula is rendered. A model's score is the share of tests it passes.

The Hugging Face page for the dataset carries a leaderboard. It is assembled from files that model repos publish themselves: a small YAML file under .eval_results/, read either from the repo's main branch or from an open pull request. The board JSON is public at /api/datasets/allenai/olmOCR-bench/leaderboard, so anyone can pull the same 24 rows I used.

The dataset card splits the 7,010 tests into seven categories: arxiv_math (2,927 tests), table_tests (1,020), multi_column (884), headers_footers (753), old_scans (526), old_scans_math (458) and long_tiny_text (442). The row's headline number is stored under the task id overall.

How is the board's overall computed?

The card's results table answers part of it. It lists a Base column next to the seven categories, and its Overall is the unweighted mean of those eight numbers: for the first row, GOT OCR, (52.7 + 94.0 + 93.6 + 29.9 + 42.0 + 22.1 + 52.0 + 0.2) / 8 is 48.3, the listed Overall. The card's table of 7,010 tests has no Base category, and the repo's bench_data folder has seven test files. One board row's notes describe baseline as an automatic category of 1,394 checks that is "not a transcription-accuracy metric".

So the convention is the card's, and the question is whether each row on the board follows it. Most YAML files carry the seven categories plus an entry called baseline, and averaging all eight entries without weights reproduces overall to within 0.06 points for 15 rows. Here is the whole check:

import json, urllib.request, yaml

def get(url):
    req = urllib.request.Request(url, headers={"User-Agent": "Mozilla/5.0"})
    return urllib.request.urlopen(req, timeout=120).read().decode()

COLS = ["arxiv_math", "headers_footers", "long_tiny_text", "multi_column",
        "old_scans", "old_scans_math", "table_tests", "baseline"]
board = json.loads(get("https://huggingface.co/api/datasets/allenai/olmOCR-bench/leaderboard"))

for r in board:
    pr = r.get("pullRequest")
    ref = f"refs%2Fpr%2F{pr}" if pr else "main"
    try:
        text = get(f"https://huggingface.co/{r['modelId']}/raw/{ref}/{r['filename']}")
    except Exception:
        continue                                  # gated repos answer 401
    v = {e["dataset"]["task_id"]: e["value"] for e in yaml.safe_load(text)}
    if not all(c in v for c in COLS):
        continue                                  # a column is missing: cannot recompute
    mean8 = sum(v[c] for c in COLS) / 8
    if abs(mean8 - r["value"]) > 0.06:
        print(r["rank"], r["modelId"], "board", r["value"], "8-column mean", round(mean8, 2))

On November 8 it prints three lines:

13 lightonai/LightOnOCR-2-1B board 83.2 8-column mean 75.24
20 lightonai/LightOnOCR-1B-1025 board 76.1 8-column mean 71.05
23 zai-org/GLM-OCR board 75.2 8-column mean 77.8

An unweighted mean of categories is a choice. The same 7,010 tests weighted by test count would give arxiv_math 42 percent of the score. Neither is wrong, but a reader comparing a row against its own breakdown needs to know which one is in use, and for these three rows it is not the card's.

Which rows use a different overall?

The three mismatches are not errors. Each file says what it did.

Board rank	Repo	Listed overall	Mean of its 8 columns	Mean without `headers_footers`	Note in the file
13	LightOnOCR-2-1B	83.2	75.24	83.17	H&F rewards omission, so it is excluded
20	LightOnOCR-1B-1025	76.1	71.05	76.13	Excluding Headers & Footers category
23	GLM-OCR	75.2	77.80	75.23	Excluding Headers & Footers category, run through an API

Dropping headers_footers and averaging the other seven columns, baseline included, reproduces all three listed numbers to within 0.04. These rows are correctly labelled in their own files. The board just sorts them next to rows that kept the category.

The effect is large for the LightOn rows. Their headers_footers scores are 19.7 and 35.5, against a median of 92.3 for the 20 rows that report the column. The listed 83.2 is 8.0 points above the same row's eight-column mean. GLM-OCR goes the other way: its headers_footers score is 95.8, so leaving it out costs it 2.6 points.

Why is headers_footers the column that causes the trouble?

Because its 753 tests all ask for text to be absent. A page header, a footer, a page number: the test passes if the model leaves it out. The LightOn authors argue in the file that this "rewards omission, not transcription thus a model that outputs nothing scores perfectly". Their model is trained for full-page transcription and keeps headers on purpose, which lowers the score under the original scoring.

That is a reasonable position, and it is a position. The category is also the one with the widest spread on the board, 19.7 to 96.6, so any choice about it moves ranks. The baseline column has the opposite character: across the 17 rows that store it, Kraken excluded, the median is 99.6 and the range is 93.7 to 99.9. Including it lifts a row's average by 1.4 to 4.1 points, about 2.3 at the median, and it separates rows far less than any test category does.

Neither the exclusion nor the inclusion is hidden. What the board does not do is mark which rows chose which, so the sort key hides the choice.

Does the order change when every row is scored the same way?

I built three common yardsticks from the stored per-category values. They are my constructions, not board metrics. Rows that cannot be recomputed (the gated file at rank 3, the mean-only file at rank 9) drop out, so every rank below is among the rows that remain.

Row	Board rank (of 24)	6 categories, no `headers_footers` (of 22)	7 categories, no `baseline` (of 21)	7 categories weighted by test count (of 21)
OXR-1.0	1	1	1	1
Infinity-Parser2-Pro	2	2	2	3
LightOnOCR-3-4B	4	3	3	2
LightOnOCR-2-1B	13	7	19	17
LightOnOCR-1B-1025	20	18	20	20
GLM-OCR	23	19	16	16

Rank 1 holds under all three yardsticks. Rank 2 holds under two and drops to 3rd when tests are weighted by count, because arxiv_math then carries 42 percent of the score. The distortion that matters is in the middle of the table. LightOnOCR-2-1B is 7th when nobody is scored on headers_footers and 19th when everybody is. The board's 13th matches neither.

Which other rows cannot be checked?

Six rows cannot be recomputed, and their problems differ.

Rank 3, ProKope-4B-Eagle-Eye (87.06): the file sits in a gated repo, so the raw YAML answers 401. The board JSON shows a note that describes the number as an overall "across 7 document OCR splits", which may or may not include baseline. I cannot tell.
Rank 9, Surya OCR 2.5 (83.7): the file, from an open pull request, stores only overall.
Ranks 14, 18 and 21 (chandra, dots.ocr, DeepSeek-OCR): all seven categories are stored and baseline is not. The listed overall minus the seven categories implies a baseline of 99.6, 99.6 and 99.4, which is right where stored baselines sit, so the headline is probably an eight-column mean with one column left out of the file. That is an inference, not a check.
Rank 19, DeepSeek-OCR-2 (76.3): six categories stored, no headers_footers and no baseline, and the source is a tweet.

Two more details. Ranks 11 and 12, Surya OCR 2 and its ONNX conversion, carry identical values in all nine entries. That is one measurement stored twice, so the board has 23 distinct results, not 24. And the Falcon-OCR file marks every entry "English-subset only" while the dataset card lists the language as English, so I cannot tell which pages were left out.

Rank 24 is a different kind of row. It is a reproducible run of a segmentation and OCR pipeline with no table reconstruction, no LaTeX generation and no header suppression, scored by the official scorer. It lists unrounded values, 36.77414 overall, and is the only row that reports an interval, a 95% bootstrap range of 35.9 to 37.7. It works as a floor for the benchmark, and it shows what an interval on this board looks like.

Where does rank 1 actually lead?

SoMarkAI/OXR-1.0 is first on arxiv_math (92.76), old_scans (59.32) and table_tests (93.05), and second on old_scans_math. Among the rows that store the column it is 5th of 20 on headers_footers, 7th of 21 on multi_column, 11th of 21 on long_tiny_text and 16th of 17 on baseline.

Among the rows I could read, the other five column leaders are five different repos: MinerU2.5 on headers_footers (96.6), LightOnOCR-3-0.8B on long_tiny_text (94.1), Falcon-OCR on multi_column (87.1, with the subset note above), Infinity-Parser2-Pro on old_scans_math (91.3) and jina-ocr-v1 on baseline (99.9). A single overall number hides that the best row for tables, old scans and math is not the best row for multi-column layouts or headers.

The rank 1 file is also the only readable one with no source and no notes. Its model card states the result and points to a benchmark document with reproduction instructions. I did not run that reproduction, and a card pointing to one is not the same as a file carrying its source. Every other readable row names one, such as a model card, a technical report or the benchmark's own repository.

How big is a 0.24-point lead on 7,010 tests?

Rank 1 is 87.84 and rank 2 is 87.60. The gap to the gated rank 3 at 87.06 is 0.78, and to rank 4 at 86.30 it is 1.54.

The card's own results table is the best guide, because it was produced on these same pages. Every Overall in it carries a ± between 0.9 and 1.2, and the card does not define the symbol in the text. The only interval on the board itself, for the last row, is a 95% bootstrap range of 35.9 to 37.7 around 36.77, a half-width near 0.9. Those are different scores from different pipelines, so I would not apply them row by row. As an order of magnitude they say the same thing: the first three or four rows of this board are a tier, and the board gives them four distinct ranks.

What can a reader do with this board?

Recompute from the categories. Pick the columns you care about and average those. If you do not care about headers_footers or baseline, leave them out for every row, not only some.
Read the notes field. It is in the board JSON. Three of the 24 rows tell you the overall was computed differently, and one tells you it is a subset.
Count the columns before you compare. A row with seven stored columns, six, or one cannot be compared with an eight-column row without an assumption.
Treat a tie as one measurement. Identical values under two repo names are a copy.
Prefer rows with a reproducible source. 6 of 24 rows come from open pull requests and none is marked verified.

For the board itself, two small changes would remove most of the ambiguity: define overall in eval.yaml, and show which columns each row's overall includes.

FAQ

What is the olmOCR-bench leaderboard on Hugging Face?
It is a board on the dataset page for olmOCR-bench, which has 1,403 PDF pages and 7,010 pass-fail tests. Model repos publish YAML result files and the board lists them, sorted by overall.

How is overall computed on this board?
For 15 of 24 rows it equals the unweighted mean of eight columns: seven test categories and a baseline column. Three rows leave out headers_footers, and six cannot be rechecked from their files.

Does the board's rank 13 deserve to be 13th?
It depends on headers_footers. LightOnOCR-2-1B is listed at 83.2 with that category excluded. With it excluded for every row the row is 7th of 22, and with it included for every row it is 19th of 21.

Are these scores verified?
No. verified is false on all 24 rows. Eighteen files sit on the repo's main branch, six come from open pull requests, and the scores are whatever each file says.

Is the rank 1 row clearly ahead?
On the stored numbers it is first under every recomputation I tried, but the margin over rank 2 is 0.24 points and over rank 3 is 0.78, while the card's own table reports a ± of 0.9 to 1.2 on Overall. I would treat the top three as a tier.

Why is headers_footers controversial?
Its 753 tests check that text is absent, so a model that outputs nothing passes them. The LightOn files argue that this rewards omission and leave it out of their overall, and so does the GLM-OCR file. Most other rows keep it, so the board's rows disagree about whether to count it.

DEV Community