The number is in the paper. It's just one row down.

This is a submission for the Hacktoberfest Open-Source AI Challenge, Week 2. This week's theme is Side Hustle: rowcheck is a pre-submission number check that a freelance manuscript editor can sell, run on one gaming GPU with an open-weight model.

What I Built

Table 4 of the OLMo paper compares instruction-tuned models. Two rows sit one above the other: TÜLU 2, with an AlpacaEval win rate of 73.9, and TÜLU 2+DPO, with 85.1.

Now take a sentence like "TÜLU 2 reaches an 85.1% win rate on AlpacaEval." Every part of it is on that page. Search the PDF for "85.1" and you land in the right table, next to the right column, half an inch from the right model name. The claim is still wrong.

I gave that sentence to a standard retrieval pipeline: split the paper into chunks, fetch the three best, and ask an open model to judge, with room to write its evidence before its verdict. It answered supported, twice out of two, and its own evidence said: "the row for 'TÜLU 2+DPO' shows a value of 85.1". The model saw the right row and still matched the claim to it.

rowcheck checks a numeric claim against a paper's tables and points to the exact cell it rests on:

$ python -m rowcheck olmo.pdf "TÜLU 2 reaches an 85.1% win rate on AlpacaEval."
CONTRADICTED  TÜLU 2 reaches an 85.1% win rate on AlpacaEval.
              Table 4 (p. 8) reports 73.9, not 85.1%, for TÜLU 2 / AlpacaEval.
              85.1 is TÜLU 2+DPO's AlpacaEval (the next row), Table 4 p. 8.

With --out report.html every verdict comes with a crop of the page: the row is boxed, the cell the claim should have used is filled, and the number it actually used is outlined.

It also catches the quieter mistakes:

• The column. "Mistral 7B reaches 52.2% on MATH." Table 2 gives 13.1% for MATH; 52.2% is the GSM8K cell beside it.
• The setting. "Gemma-2 9B scores 68.6 on GSM8K in a 0-shot setting." The number is right, but Table 13 measured it 5-shot. Verdict: contradicted, and it says why.
• The unit. "Stack-Edu has 21.8T Python tokens." The header says billions.
• What the table can't tell you. "Mistral 7B scores 60.1% on MMLU in a 0-shot setting." The number is right, but Table 2 never states the shot count. Verdict: ambiguous, not supported.

Who pays for this

Freelance academic editors already sell language editing per manuscript. What they rarely sell is a number check: does every figure in the abstract and discussion match the tables it came from? Doing that by hand means reading each claim against each table, and it is exactly where row and column slips survive peer review.

rowcheck turns that into an add-on the editor can price per paper. The editor pastes the claims from the text, runs the tool on the PDF, and sends the author an HTML report with every mismatch boxed on the page. Each verdict carries its own evidence, so the editor checks the tool's work in one look before anything goes to the client.

Two things make the business work, and both come from the model being open:

• The manuscript never leaves the editor's machine. A client's unpublished paper is confidential. A cloud API means a third party sees it; a local model doesn't.
• The cost per paper is close to zero. Gemma 4 runs on a 16 GB consumer card at about 6 seconds per claim, so a paper with 40 numeric claims takes about four minutes. There is no per-token bill eating the margin.

I haven't landed a paying user yet. The next step is to offer it free to two editors for their next manuscripts and ask what they would pay.

Demo

The report for the six claims above, with every crop: exochard.github.io/rowcheck. On a phone, each crop opens scrolled to the cell it checked.

Code

github.com/exochard/rowcheck (MIT). ./fetch-papers.sh downloads the ten CC BY papers and checks their hashes; python -m eval.run dev reruns the development table.

How I Built It

Three stages. Each one does the job it's good at.

1. Tables as positioned words, not text. PDF tables in papers are mostly borderless, with stacked headers and two-column pages. Text extraction flattens them into a run of numbers. rowcheck keeps every word with its box on the page. It groups the words into visual lines and finds each table from its caption. Then it works out the columns from where the numbers line up across rows, and hangs the header words over each column. A row comes out as Mistral 7B Pretrained | MMLU=60.1% | HellaSwag=81.3% | …. When it can't label a row one-to-one, it passes the raw line on. A wrong label is worse than none.

2. An open-weight model reads the table. Gemma 4 (26B-A4B, Q4) runs locally through llama.cpp on a 16 GB card. It gets the claim and the three best-matching tables. Under a JSON schema, it names the table, the row, the column and the cell, plus the status of every setting the claim states. I run it with thinking off, about 6 s per claim; with thinking on it spent its whole token budget recounting columns and returned nothing.

3. Code decides the numbers. The model's reading is a proposal:

• the cited cell must be printed on that exact line;
• the claimed value is compared to it with a tolerance;
• the unit is checked against the column header;
• an explicit shot count in the claim is checked against the row, the header and the caption;
• if the model picks "TÜLU 2+DPO" for a claim about "TÜLU 2", the row whose name matches exactly wins.

The model never gets the last word on a number.

Does it work?

Before running anything, I wrote 30 claims from six CC BY papers (Mistral 7B, OLMo, OLMo 2, Phi-3, Gemma 2, SmolLM2) and labelled each one from the page. I also froze a split: four papers for development, two held back. Then I added six trap claims that swap a row, a unit, a denominator or a setting.

	rowcheck	chunk retrieval + same model
Development claims (20)	19	18
Held-out claims (10)	7	6
Trap claims marked "supported" (6)	0	1

The comparison uses the same Gemma model and lets it write its evidence before its verdict. An earlier version of my baseline asked for the verdict first, and that made it look worse than it is. Overall accuracy is close, and I won't pretend otherwise: the plain pipeline is good on most of these claims. The difference is what you can do with the answer: rowcheck gives you the page, the row and the cell, so checking its work takes one look instead of a re-read. And on the held-out papers it made one mistake I care about. It called a 10-shot TruthfulQA number "0-shot" supported. Shot counts are now checked in code. That fix came after the held-out run, so it isn't in these numbers.

Then I tried it on four papers it had never seen: Gemma, Mixtral, Nemotron-4 and Qwen, with 22 new claims labelled before any run, nine of them traps. It broke. It got 9 of 13 claims right where plain retrieval got 12, and it called four of the nine traps "supported". The failures were all about reading the table, not the model's judgement: a header like "Gemma" printed once above "2B 7B", a family name like "LLaMA" printed once beside four size rows, an asterisk that was a different Unicode character. I fixed the reader, not the prompts.

Four unseen papers (13 claims, 9 traps)	rowcheck	chunk retrieval + same model
Claims right, first run	9	12
Traps marked "supported", first run	4	4
Claims right, after fixing the table reader	13	12
Traps marked "supported", after the fixes	0	4

The "after" rows are on the papers I fixed against, so read them as development numbers. On the two original held-out papers the same fixes take rowcheck from 7 to 8 of 10.

Limits

• Tables drawn as images, and rotated tables, are out of reach.
• When two tables carry the same row, it can pick the appendix copy that lacks the unit.
• Ten papers and 58 claims make a small test, and the first contact with new papers went badly.

Why Open Innovation Matters

The papers here are open (CC BY), and the model reading them is an open-weight model running on my own GPU. Checking someone's numbers shouldn't require sending their paper, or your draft, to a service.

For a side business the open model is the whole margin. A closed API would bill every claim and see every client manuscript. Here the weights sit on my disk, the tables never leave the machine, and when a new paper layout broke the reader I fixed the code that reads tables instead of waiting on a provider.

Prize Categories

Best use of Gemma: Gemma 4 26B-A4B does the table reading, served locally by llama.cpp.