Every PDF-to-Markdown tool claims it "preserves the structure". I build one of them (CleanMD), and I started it because a DeepSeek paper I wanted in my notes came out of every tool I tried as soup. So I had an obvious bias and an obvious question: how would I know if that claim were false about mine?
I ended up writing a small benchmark. Five public PDFs from five genres, three tools with default settings, and a ground truth that comes from the documents themselves, never from any tool's output. Here is what I found, losses first.
The setup
Documents (all public, none picked after seeing results):
| Document | Why it is hard |
|---|---|
| RFC 9110 HTTP Semantics (194 pp.) | 291 numbered sections, ABNF grammar, ASCII diagrams, running footer |
| BERT (arXiv 1810.04805, 16 pp.) | two-column paper with result tables and captions right next to them |
| Think Python 2e (292 pp.) | technical book: table of contents, 500+ code listings |
| Loper Bright v. Raimondo (SCOTUS, 114 pp.) | legal opinion: parts I/II and sub-parts A/B are single centered letters at body size |
| NIST CSF 2.0 (32 pp.) | standard with appendices and tables with multi-line cells |
Tools: MarkItDown 0.1.7 (Microsoft), pymupdf4llm 1.28.2 (Artifex), and CleanMD 0.93.0 (mine, the same engine that runs in the browser, executed in Node). Pandoc is not in the table because it cannot read PDF at all (PDF is an output format for it): return code 21 on all five files.
Ground truth: RFC sections from the table of contents of the official rfc9110.txt; Think Python's 126 entries from the book's own ToC; BERT's 29 numbered headings from the lines set in the Medium weight; NIST's 8 entries from its ToC. For the court opinion there is no ToC, so the 31 part markers were defined geometrically (a single token, centered, at body size), and that definition is close to the heuristic my own engine uses, which I flag on the page. Weigh that document accordingly.
Metrics: sections recognised as headings (and at a depth consistent with the numbering), fenced code blocks, tables and "prose cells" (a table cell holding a sentence = a fake or contaminated table), plus a few document-specific checks like "is the collected ABNF one block?".
Where CleanMD loses
- RFC 9110 section recall, first run: pymupdf4llm 290/291, CleanMD 280/291. Eleven bold, body-size headings (8.8.2 Last-Modified, 13.1.3 If-Modified-Since, 15.5.10 409 Conflict…) came out of CleanMD as plain paragraphs. The cause turned out to be embarrassing and specific: pdf.js splits a line into two font ids when a glyph (the hyphen, here) comes from a second subset of the same bold font, and my detector only looked at lines with one uniform font. Fixed the same day; the re-run is 291/291. I am keeping the first number in the article and on the page, because a benchmark that only shows the after is marketing.
-
Fence count on Think Python: pymupdf4llm 657, CleanMD 569. With a caveat: 328 of pymupdf4llm's fences are single-line fences wrapping inline code words. The operator listing that opens §5.2 (
x != y … x >= y) is a fence in CleanMD and not there. - Prose cells on NIST: MarkItDown 0, CleanMD 23. Most of CleanMD's are the Tier table in Appendix B, whose cells are paragraphs by design (it now comes out as one table per page instead of one block per Tier, so the metric counts every wrapped line); MarkItDown's tables are cleaner on that document. It also emits zero headings on it, so I wouldn't take that trade, but the row is there.
Where it wins
- Code survives. RFC 9110: 158 fenced blocks vs 0 and 0. The ABNF grammar, the message diagrams and the examples come out of the other two as prose; the collected ABNF appendix is one block only in CleanMD.
- Running footers are gone. "Fielding, et al. Standards Track Page N" left in the text: 0 vs 187 (MarkItDown) vs 194 (pymupdf4llm).
- Hierarchy has depth. Think Python: 122 of 126 sections at the right depth for CleanMD; pymupdf4llm finds 113 but emits all of them as H4, so the outline is flat. On the court opinion, 28 of 31 part markers become headings (I → H2, A → H3, 1 → H4); pymupdf4llm gets 18 and promotes the running page header to an H3 on 80 pages.
- No fake tables. MarkItDown turns BERT into 134 pseudo-tables (the title becomes a 9-column table) and Think Python's code listings into 683 tables.
The numbers
| CleanMD | MarkItDown | pymupdf4llm | |
|---|---|---|---|
| RFC 9110: sections / 291 (right depth) | 291 (291) · first run 280 | 0 | 290 (275) |
| RFC 9110: fences / ABNF one block | 158 / yes | 0 / no | 0 / no |
| RFC 9110: footer lines in text | 0 | 187 | 194 |
| BERT: sections / 29 | 26 | 0 | 26 |
| BERT: tables (prose cells) | 11 (0) | 134 (0) | 9 (0) |
| Think Python: sections / 126 (right depth) | 122 (122) | 0 | 113 (0) |
| Think Python: fences (single-line) | 569 (52) | 0 | 657 (328) |
| SCOTUS: part markers as headings / 31 | 28 | 0 | 18 |
| SCOTUS: running headers promoted | 0 | 0 | 80 |
| NIST: sections / 8 | 8 | 0 | 8 |
| Seconds, RFC 9110 (same laptop) | 1.3 | 5.5 | 10.9 |
What this does not tell you
Five documents are five documents. Nothing here is scanned (all five have a text layer), nothing measures reading order inside dense three-column layouts (where all three tools struggle), nothing measures math or prose quality. It shows failure modes, not a universal ranking.
Reproduce it
The page has the full tables with the best value per row highlighted and a kit you can download: run.sh (downloads the PDFs, installs the two open-source tools in a venv, builds the ground truth, scores everything in about two minutes), the ground truth, the results, and the raw Markdown CleanMD produced, so every sentence above can be checked against the actual output.
→ https://cleanmd.dev/benchmarks/pdf-to-markdown
If you know a public PDF that breaks all three, I want it.
Top comments (0)