DEV Community: Giuseppe Castelluccio

The number is in the paper. It's just one row down.

Giuseppe Castelluccio — Tue, 13 Oct 2026 07:30:36 +0000

This is a submission for the Hacktoberfest Open-Source AI Challenge, Week 2. This week's theme is Side Hustle: rowcheck is a pre-submission number check that a freelance manuscript editor can sell, run on one gaming GPU with an open-weight model.

What I Built

Table 4 of the OLMo paper compares instruction-tuned models. Two rows sit one above the other: TÜLU 2, with an AlpacaEval win rate of 73.9, and TÜLU 2+DPO, with 85.1.

Now take a sentence like "TÜLU 2 reaches an 85.1% win rate on AlpacaEval." Every part of it is on that page. Search the PDF for "85.1" and you land in the right table, next to the right column, half an inch from the right model name. The claim is still wrong.

I gave that sentence to a standard retrieval pipeline: split the paper into chunks, fetch the three best, and ask an open model to judge, with room to write its evidence before its verdict. It answered supported, twice out of two, and its own evidence said: "the row for 'TÜLU 2+DPO' shows a value of 85.1". The model saw the right row and still matched the claim to it.

rowcheck checks a numeric claim against a paper's tables and points to the exact cell it rests on:

$ python -m rowcheck olmo.pdf "TÜLU 2 reaches an 85.1% win rate on AlpacaEval."
CONTRADICTED  TÜLU 2 reaches an 85.1% win rate on AlpacaEval.
              Table 4 (p. 8) reports 73.9, not 85.1%, for TÜLU 2 / AlpacaEval.
              85.1 is TÜLU 2+DPO's AlpacaEval (the next row), Table 4 p. 8.

With --out report.html every verdict comes with a crop of the page: the row is boxed, the cell the claim should have used is filled, and the number it actually used is outlined.

It also catches the quieter mistakes:

The column. "Mistral 7B reaches 52.2% on MATH." Table 2 gives 13.1% for MATH; 52.2% is the GSM8K cell beside it.
The setting. "Gemma-2 9B scores 68.6 on GSM8K in a 0-shot setting." The number is right, but Table 13 measured it 5-shot. Verdict: contradicted, and it says why.
The unit. "Stack-Edu has 21.8T Python tokens." The header says billions.
What the table can't tell you. "Mistral 7B scores 60.1% on MMLU in a 0-shot setting." The number is right, but Table 2 never states the shot count. Verdict: ambiguous, not supported.

Who pays for this

Freelance academic editors already sell language editing per manuscript. What they rarely sell is a number check: does every figure in the abstract and discussion match the tables it came from? Doing that by hand means reading each claim against each table, and it is exactly where row and column slips survive peer review.

rowcheck turns that into an add-on the editor can price per paper. The editor pastes the claims from the text, runs the tool on the PDF, and sends the author an HTML report with every mismatch boxed on the page. Each verdict carries its own evidence, so the editor checks the tool's work in one look before anything goes to the client.

Two things make the business work, and both come from the model being open:

The manuscript never leaves the editor's machine. A client's unpublished paper is confidential. A cloud API means a third party sees it; a local model doesn't.
The cost per paper is close to zero. Gemma 4 runs on a 16 GB consumer card at about 6 seconds per claim, so a paper with 40 numeric claims takes about four minutes. There is no per-token bill eating the margin.

I haven't landed a paying user yet. The next step is to offer it free to two editors for their next manuscripts and ask what they would pay.

Demo

The report for the six claims above, with every crop: exochard.github.io/rowcheck. On a phone, each crop opens scrolled to the cell it checked.

Code

github.com/exochard/rowcheck (MIT). ./fetch-papers.sh downloads the ten CC BY papers and checks their hashes; python -m eval.run dev reruns the development table.

How I Built It

Three stages. Each one does the job it's good at.

1. Tables as positioned words, not text. PDF tables in papers are mostly borderless, with stacked headers and two-column pages. Text extraction flattens them into a run of numbers. rowcheck keeps every word with its box on the page. It groups the words into visual lines and finds each table from its caption. Then it works out the columns from where the numbers line up across rows, and hangs the header words over each column. A row comes out as Mistral 7B Pretrained | MMLU=60.1% | HellaSwag=81.3% | …. When it can't label a row one-to-one, it passes the raw line on. A wrong label is worse than none.

2. An open-weight model reads the table. Gemma 4 (26B-A4B, Q4) runs locally through llama.cpp on a 16 GB card. It gets the claim and the three best-matching tables. Under a JSON schema, it names the table, the row, the column and the cell, plus the status of every setting the claim states. I run it with thinking off, about 6 s per claim; with thinking on it spent its whole token budget recounting columns and returned nothing.

3. Code decides the numbers. The model's reading is a proposal:

the cited cell must be printed on that exact line;
the claimed value is compared to it with a tolerance;
the unit is checked against the column header;
an explicit shot count in the claim is checked against the row, the header and the caption;
if the model picks "TÜLU 2+DPO" for a claim about "TÜLU 2", the row whose name matches exactly wins.

The model never gets the last word on a number.

Does it work?

Before running anything, I wrote 30 claims from six CC BY papers (Mistral 7B, OLMo, OLMo 2, Phi-3, Gemma 2, SmolLM2) and labelled each one from the page. I also froze a split: four papers for development, two held back. Then I added six trap claims that swap a row, a unit, a denominator or a setting.

	rowcheck	chunk retrieval + same model
Development claims (20)	19	18
Held-out claims (10)	7	6
Trap claims marked "supported" (6)	0	1

The comparison uses the same Gemma model and lets it write its evidence before its verdict. An earlier version of my baseline asked for the verdict first, and that made it look worse than it is. Overall accuracy is close, and I won't pretend otherwise: the plain pipeline is good on most of these claims. The difference is what you can do with the answer: rowcheck gives you the page, the row and the cell, so checking its work takes one look instead of a re-read. And on the held-out papers it made one mistake I care about. It called a 10-shot TruthfulQA number "0-shot" supported. Shot counts are now checked in code. That fix came after the held-out run, so it isn't in these numbers.

Then I tried it on four papers it had never seen: Gemma, Mixtral, Nemotron-4 and Qwen, with 22 new claims labelled before any run, nine of them traps. It broke. It got 9 of 13 claims right where plain retrieval got 12, and it called four of the nine traps "supported". The failures were all about reading the table, not the model's judgement: a header like "Gemma" printed once above "2B 7B", a family name like "LLaMA" printed once beside four size rows, an asterisk that was a different Unicode character. I fixed the reader, not the prompts.

Four unseen papers: 22 claims, 9 of them traps	rowcheck	chunk retrieval + same model
Honest claims right (of 13), first run	9	12
Traps marked "supported" (of 9), first run	4	4
Honest claims right (of 13), after fixing the table reader	13	12
Traps marked "supported" (of 9), after the fixes	0	4

The "after" rows are on the papers I fixed against, so read them as development numbers. On the two original held-out papers the same fixes take rowcheck from 7 to 8 of 10.

Limits

Tables drawn as images, and rotated tables, are out of reach.
When two tables carry the same row, it can pick the appendix copy that lacks the unit.
Ten papers and 58 claims make a small test, and the first contact with new papers went badly.

Why Open Innovation Matters

The papers here are open (CC BY), and the model reading them is an open-weight model running on my own GPU. Checking someone's numbers shouldn't require sending their paper, or your draft, to a service.

For a side business the open model is the whole margin. A closed API would bill every claim and see every client manuscript. Here the weights sit on my disk, the tables never leave the machine, and when a new paper layout broke the reader I fixed the code that reads tables instead of waiting on a provider.

Prize Categories

Best Use of Gemma: Gemma 4 26B-A4B does the table reading, served locally by llama.cpp on a 16 GB card. It is a mixture-of-experts model with about 4B parameters active per token, which is what brings a claim down to about six seconds on consumer hardware. It answers under a JSON schema with thinking off, and code checks every number it cites before a verdict goes out.

An Italian copy reviewer that cites its style guide, built on Sanity Context

Giuseppe Castelluccio — Wed, 30 Sep 2026 08:55:41 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content

What I Built

I'm Italian, and part of my work is checking the Italian copy of software websites. The same mistakes keep shipping: "Fidato da" for "Trusted by", "Utensileria" (a hardware store) as the menu label for "Tools", "5 Mld+ Altoparlanti nativi" (native loudspeakers) for "5B+ native speakers". A general model catches some of these and misses others, and when it does catch one it cannot tell you which house rule it applied.

So I built a reviewer agent. You give it a string, where it appears and, if you have it, the English source. It reads the team's Italian style guide from a Sanity Knowledge Base, answers ok or error, proposes the corrected text, and lists the Knowledge Base entries it relied on. The style guide is structured content in Sanity: 21 rules and 24 glossary terms. Fourteen rules carry real strings that shipped on live product pages next to their native correction; six are house decisions the team made.

I also measured it, and I report every run. On the 12 held-out generic pairs the Knowledge Base never saw, the Knowledge Base and the model alone are within one pair of each other: with the same code filter behind both arms, the latest run gives 11 against 10 pairs fully right, and the Sep 30 run rescored the same way gives 9 against 10. A general model already knows most of what makes Italian copy wrong, so a style guide cannot add much there. The Knowledge Base earns its place on rules a model cannot know: house decisions that differ from generic Italian. On a separate slice of 16 pairs built only from such rules, the agent with the Knowledge Base gets 10 of 16 pairs fully right and the same model without it gets none. The details are below.

Demo

Three real reviews in the local web UI, grounded in the Knowledge Base. Each took 12 to 17 seconds on the free Gemini tier; the GIF cuts the wait, and every card shows the real time.

The same three reviews as a video: https://vimeo.com/1231260014

A review looks like this (real output from the Sep 30 test run, gemini-3.1-flash-lite, input Fidato da oltre 100.000 imprese):

{
  "verdict": "error",
  "reason": "The phrase 'Fidato da' is a literal calque of 'Trusted by' and is considered unnatural in Italian marketing copy.",
  "evidence": [
    "Fidato da"
  ],
  "corrected": "Scelto da oltre 100.000 imprese",
  "rules": [
    "Calques, False Friends & Idiom Errors/\"Trusted by\" calque (major)"
  ]
}

Code

https://github.com/exochard/italian-copy-reviewer

TypeScript on Node 22 with the Vercel AI SDK; a Sanity Studio schema; a small Python script that turns the rules into Sanity documents. npm test runs without network.

How I Used Sanity

The content model. Two document types in the Studio schema:

l10nRule: title, slug, category (calque, agreement, number format…), severity, the rule in plain words, preferred forms, forms to avoid, and an array of real shipped examples (product type, where the text appears, English source, shipped Italian, native correction).
glossaryTerm: the English term, the preferred Italian, the forms to avoid, a note.

The examples are the point. "Don't translate literally" is useless to a model; "a software company shipped Fidato da oltre 9,500 marchi e organizzazioni globali and a native reviewer changed it to Scelto da oltre 9.500 marchi e organizzazioni in tutto il mondo" is not.

The Knowledge Base. Two sources: the production dataset above, and Microsoft's public Italian Localization Style Guide (a PDF). They sometimes disagree. When it built the Knowledge Base, Sanity Context flagged three such conflicts as Issues instead of merging them silently; I read each one and resolved it in the dashboard before the final runs.

The agent. The AI SDK connects to the Context MCP endpoint in knowledge-base mode. The system prompt tells the model to call initial_context first and then knowledge_base_read on the entries that could apply. The output is a typed object (verdict, reason, the wrong fragments quoted from the text, corrected text, cited entry paths), so every answer says which rules it used.

The first measured run taught me something. With the Knowledge Base the agent flagged correct text more often than the model alone, because it applied entries whose trigger was not there: it flagged "AI" in a sentence with no "AI" in it, and asked for "oltre" on a number that had no "+". So every flag now has to quote the wrong words, and code drops a flag whose quote is not in the text. The prompt also says an entry applies only when the text contains what the entry is about. I tuned this on the training pairs only.

That was not enough. The second test run showed flags whose quote was the whole string: the agent read the capitalisation entry and called Automazione batch con un clic and RECENSIONI DEGLI UTENTI "Title Case". So a flag that relies only on the Title Case, "+" or AI entry now survives only when the text holds that entry's trigger: a capitalised word in mid-sentence (ALL CAPS never counts), a plus sign, or "AI" and "IA" mixed in one string, which is exactly what the AI entry says. Re-applied offline to the saved training outputs, this took the grounded agent from 12/17 to 16/17 without losing a caught bug. I first applied this filter to the grounded arm only, which is not a fair comparison: re-applied offline to the saved Sep 30 test run, the same filter moves the model-alone arm from 9/12 to 10/12. So the like-for-like numbers below put both arms behind the same filter (--shared-filter in src/eval.ts), and I also show the arms with no filter at all.

One fix came from reading a test failure, and I'd rather say so. On USER REVIEW both arms read the text between the prompt's <<< and >>> markers as an empty placeholder. The prompt now says the text between the markers is the literal string.

In every grounded run on Oct 1, 24 of 24 on the test pairs and 32 of 32 on the house pairs, the agent read at least one entry before answering.

Keeping the test honest. The benchmark has 29 pairs. The train half feeds the Knowledge Base; the test half never enters it, and a unit test asserts the two halves are disjoint. Each pair is scored only when the agent flags the shipped string and leaves the native fix alone, so a reviewer that flags everything scores zero.

Every test run, same 12 pairs, gemini-3.1-flash-lite, pairs fully right. These are the original runs, where only the Knowledge Base arm had the filter; the like-for-like table follows:

Test run	Changed before it (tuned on training pairs)	Model alone	+ Sanity Knowledge Base
Sep 28	first version	7/12	4/12
Sep 29	quote check; "entry applies only when triggered" prompt; two entries reworded	9/12	5/12
Sep 30	per-rule trigger check; literal-text prompt	9/12	9/12

The Sep 30 run in full:

	Bugs caught	Fixes left alone	Pairs fully right
gemini-3.1-flash-lite alone	11/12	10/12	9/12
gemini-3.1-flash-lite + Sanity Knowledge Base	10/12	11/12	9/12

Like for like, the Sep 30 run rescored offline with src/rescore.ts, pairs fully right:

	No filter	Same filter on both arms
Model alone	9/12	10/12
+ Sanity Knowledge Base	9/12	9/12

A fresh run on Oct 1 with the shared filter on, after the Knowledge Base was rebuilt with the house rules, gave 11/12 for the Knowledge Base and 10/12 for the model alone.

The model alone moves between runs too: both arms share the prompt, and the model is not fully repeatable even at temperature 0. Twelve pairs is a small test, so one pair is eight points.

What still fails in the Sep 30 run: both arms passed Transcrizione, which should be Trascrizione, so neither checks spelling letter by letter. With the Knowledge Base the agent called the correct Ottieni informazioni utili e pratiche a calque of "Get", and let an English string, Larger file per task, through as correct. The model alone failed on two correct strings instead: it carried the "+" of the English source into oltre 100 siti and called it redundant, and it objected to the preposition in per attività. What the Knowledge Base does well is the explanation: every flag names the entry it came from, so a reviewer can see why it fired and fix the entry instead of the prompt.

Where the Knowledge Base cannot be replaced. A keyword search over a style guide would not beat a model on generic Italian, and on that test it does not. So I added a held-out slice of 16 pairs (house in content/split.json, pairs in content/house.jsonl) whose verdict depends on six rules and five glossary terms the team chose and that exist only in the Knowledge Base: always "tu" and never "Lei", error messages that open with "Non siamo riusciti a", no exclamation marks in product copy, "persona" instead of "utente", no "per favore", the euro sign after the amount, "spazio di lavoro" for "workspace", "pagamento" for "checkout", plan names kept in English as "Free" and "Pro". Both halves of every pair are correct Italian in general; only the shipped one breaks a house rule. For example: Acceda al suo account per continuare against Accedi al tuo account per continuare, Invita un utente nel progetto against Invita una persona nel progetto, Da €29 al mese against Da 29 € al mese. No prompt, filter or rule was tuned on this slice, and none of its strings is in the seed (a unit test checks both the split and the seed). Pairs fully right, both arms behind the same filter: Knowledge Base 10/16, model alone 0/16. Neither arm flagged a single correct string. The six the agent missed: checkout, e-mail and two uses of utente went through although the glossary covers them, and so did an error opening with Errore: and an exclamation mark. I wrote these pairs myself to test the claim, so they say what the Knowledge Base can do when the rules are in it, not how often real copy breaks house rules.

Sanity Project Details

Project ID: 3ezm3tm1 (dataset production). The schema is in studio/schemaTypes/ and the seed documents in content/seed.ndjson.

The dataset is public, so anyone can query it without a token, for example the six house rules:

curl -G "https://3ezm3tm1.api.sanity.io/v2025-02-19/data/query/production" --data-urlencode 'query=*[_type=="l10nRule" && category=="house_style"]{title, rule, prefer, avoid}'

How this was made

I wrote the benchmark strings and their corrections from my own QA audits; they predate the challenge, and the brand names are masked. The code and the Knowledge Base content were written during the challenge with an AI coding assistant (Claude Code): I set the spec and the acceptance checks, it drafted the code, and I reviewed, ran and tested every part. The numbers above come from npm run eval runs (results/eval-*.json) and are reported as measured, including the ones that went against the Knowledge Base. src/rescore.ts re-applies the flag checks to a saved run without calling any model and prints the arms with and without the filter.

AI reviewers catch Italian UI bugs, then flag the good fixes too

Giuseppe Castelluccio — Mon, 28 Sep 2026 13:29:15 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Companies ship Italian that no Italian would write. In my own QA audits of software websites this year I found a transcription tool whose "Tools" menu said Utensileria (a hardware store), a homepage bragging about 5 Mld+ Altoparlanti nativi (five billion native loudspeakers), and more than one Fidato da, a word-for-word "Trusted by" that reads as machine output.

Models are now the reviewers many teams rely on, so I wanted to know: shown a string on its own, the way a reviewer sees it, can a model tell a shipped defect from its native fix?

The benchmark has 29 real pairs: the string as it shipped, and the version a native speaker would ship. Brand names are masked. The model sees each half separately, with the product type, where the text appears and the English source when there is one, and answers ok or error.

The scoring is the part I care about. A pair counts only when the model flags the shipped string and leaves the fix alone. A model that flags everything scores zero, and so does one that approves everything. Reviewers that cry wolf are useless in practice, and a plain accuracy number would hide them.

The defects cover ten types: 8 untranslated leftovers, 5 calques, 4 agreement errors, 3 number formats, 2 each of wrong word sense, misspelling and broken grammar, and one each of a broken encoding, an unidiomatic phrase and English-style capitalisation.

Models Tested

I ran the task on the 18 models Kaggle Benchmarks offered me, from Google, Anthropic, OpenAI, xAI, Alibaba, DeepSeek, Zhipu and the open Gemma and gpt-oss families, big and small, because a localisation team picks a reviewer by cost as much as by quality.

Not all of them produced answers. In the first runs, with all models in parallel, seven returned API errors on most calls. I re-ran those seven one at a time on Oct 1. Five then answered cleanly, while three open models still hit rate limits and one was not found, so I report those four as not measured rather than as zero. A run counts only when at most 2 of its 58 answers were unreadable. That leaves 14 models and 21 clean runs.

Findings

Pairs fully right, bugs caught and fixes left alone, out of 29. Where a model has two clean runs, both are shown.

Model	Pairs fully right	Bugs caught	Fixes left alone
Gemini 3.8 Flash	26 · 25	27 · 26	28 · 27
Gemini 3.7 Flash	26 · 23	27 · 27	28 · 25
Gemini 3.5 Flash-Lite	26 · 25	26 · 27	29 · 27
GPT-6 Astra	26	27	28
Gemma 4 31B	23 · 24	25 · 27	26 · 26
Gemini 3.1 Pro (preview)	23 · 24	27 · 27	25 · 26
Claude Opus 5	24	26	27
Claude Sonnet 5	23	26	26
Claude Haiku 4.5	22	24	27
GPT-5.5	22	29	22
Gemini 2.5 Flash	21	24	26
GPT-5.4 mini	19 · 19	24 · 24	23 · 23
GLM-5	19	26	22
GPT-5.4 nano	18 · 16	26 · 26	20 · 19

Catching the bug is the easy half. Every clean model caught 24 or more of the 29 shipped defects. What separated them was the other half: leaving a correct native string alone. GPT-5.4 nano catches as many bugs as the leaders and then flags a third of the native fixes too. GPT-5.5 is the extreme case: the only model to catch all 29 bugs, and one that flagged 7 of the 29 native fixes on the way. A reviewer like that sends a translator to "fix" good Italian, which is worse than no reviewer.

Numbers are the blind spot. Across the 21 clean runs, models caught every untranslated leftover and broken encoding, 103 of 105 calques and 83 of 84 agreement errors. They caught the three number-format errors only 25 times in 63, and only GPT-5.5, the model that flags everything, caught all three. All three are the English "+" glued to a number: 6+ piattaforme, 15.000+ computer, and oltre 100+ siti, which says "more than" twice. Italian writes oltre 6. The strings look like numbers, so models wave them through.

The small model tied the big ones. Gemini 3.5 Flash-Lite, the cheapest Gemini on the list, left all 29 native fixes alone in one run and tied the top score. Gemini 3.1 Pro did not beat the Flash models, and Claude Opus 5 scored two pairs above Haiku 4.5, inside the noise below.

One run is not a result. The same model moved by up to three pairs between runs (Gemini 3.7 Flash scored 26 and then 23; my first notebook run gave 25). On a 29-pair benchmark, a one- or two-pair gap between models is noise; I would not rank models on it.

What it changed about how I think

I used to judge an AI reviewer by what it catches. After this I judge it by what it leaves alone. Recall is cheap: the weakest clean model still caught 24 of 29 bugs. The cost sits in the false alarms, because each one sends a translator off to defend a correct string.

I also stopped trusting "give the model the style guide" as the fix. I tested that on a smaller follow-up for another DEV challenge: the same 29 pairs split so the 12 test pairs never reached the guide, a different model (Gemini 3.1 Flash-Lite), and a real Italian style guide served to it as a knowledge base. On its first run the style guide made it worse, 4 of 12 held-out pairs right against 7 without it, because the model applied rules whose trigger was not in the string. It took two rounds of fixes to bring the grounded version level with the plain one, 9 of 12 each. In that test, the rules raised false alarms long before they lowered anything.

What I'd measure next: more pairs per defect type, especially numbers and idiom, so each type gets its own reliable score, and whether a rule written for exactly the "+" pattern closes the number blind spot without new false alarms.

My Benchmark

The benchmark and its leaderboard are public on Kaggle: https://www.kaggle.com/benchmarks/giuseppecastelluccio/italian-ui-strings-shipped-vs-fixed

The task behind it, with its notebook and every model's run: https://www.kaggle.com/benchmarks/tasks/giuseppecastelluccio/italian-ui-strings-shipped-vs-fixed

How this was made

The strings and their corrections come from my QA audits, made before the challenge; I checked every correction again as a native speaker for this benchmark. The task code was written during the challenge with an AI coding assistant (Claude Code): I set the scoring rule and the checks, it drafted the code, and I reviewed and ran it. The two charts were drawn from the same run results with a short matplotlib script. A stub model that always answers right scores 1.0 and one that flags everything scores 0.0.