I knew something was off when two numbers landed in the same afternoon. Every tool that reads the PDF text layer had scored exactly 0.00 on every chart in my corpus. Fine, I thought, the charts are images, there's nothing in the text layer to read. Then PyMuPDF came in at a perfect 1.00 on clean text. Every document. Every page-count bucket. Not 0.98. 1.00.
The first number was telling me something true about the tools. The second was telling me something true about me. PyMuPDF got 1.00 because the "ground truth" for clean-text documents was PyMuPDF's own output. I'd written the answer key with one of the contestants and hadn't noticed. A reviewer did.
So this is partly a benchmark write-up and partly a list of the ways a benchmark can quietly lie to the person who built it. If you just want to know which extractor to use for a RAG pipeline, there's a decision table near the end. The rest is how I got there, including the parts I'd rather not admit.
TL;DR
- Eight local PDF extractors, 27 documents from 1 to 487 pages, four difficulty tiers. Every tool ran every document under a 12 GB, 30-minute cap on Kaggle (CPU tools on a CPU notebook, the four torch tools on a T4), and nothing failed.
- Docling is the most accurate table extractor: it reconstructed 83% of the real tables' structure (TEDS 0.83) and the 487-page Economic Report's table almost perfectly (0.9993, which rounds to 1.00). MinerU is second (0.80) and alone reproduced the four generated tables exactly.
- Tesseract 5 is the best OCR here, and it runs on a CPU: 17 characters wrong in 100 on a clean scan, 32 on average, against 36 to 42 for the GPU tools.
- Layout models throw chart text away. Only OCR keeps it: EasyOCR recovered 84% of the facts printed on the charts, Tesseract 65%, Docling 3%.
- Through the retrieval pipeline five tools sit within 0.04 of each other: Tesseract, EasyOCR, MinerU, Docling and PyMuPDF4LLM put the answer's chunk in the top five for 79 to 82% of the questions with the production chunker (76 to 80% with the structure-preserving one). What separates them is cost, memory and whether you get table structure back, not retrieval.
- Plain PyMuPDF text beats every table-aware tool on table-question retrieval: 82% against 60 to 66% with the structure-preserving chunker, 82% against 55 to 77% with the production one. Structure helps the generator read the right cell; it does not help the retriever find the chunk.
- Three bugs in my own benchmark nearly became headlines: a pipe that would have faked timeouts, an answer key written by one of the contestants, and a chunk budget counted in the wrong tokenizer.
Why I bothered
Everything downstream of extraction (chunking, embeddings, the reranker, the prompt) can only work with text that made it through. Whatever the extractor drops is gone for good. And in technical documents the facts people actually ask about sit in the places extractors are worst at: tables, charts, scanned pages.
The system in question, DocuMind, is an ordinary RAG stack: a chunker, bge-large-en-v1.5 embeddings, a bge-reranker-large cross-encoder, Qdrant with hybrid search in production. It had been swallowing PDFs through a single text-layer extractor since day one. Before replacing that with something heavier and slower, I wanted numbers on the kind of PDFs that actually show up, which for me means long government reports, RFCs, scans with an old hidden OCR layer, and factsheets made mostly of diagrams.
What I ran
27 documents, 1,567 pages, in four tiers by difficulty and four buckets by length (1–5, 6–30, 31–100, 100+ pages).
| Tier | What it tests | Documents |
|---|---|---|
| T1 | clean born-digital text | four IETF RFCs, 4 to 194 pages |
| T2 | tables and multi-column layout | Economic Report of the President 2024 (487 pages), two of its tables, two Census Bureau reports, four generated table documents |
| T3 | image-only scans | rasterised versions of five of the above, two deliberately degraded |
| T4 | figures and diagrams | five NASA documents (2 to 297 pages), four generated chart documents |
Eight of the documents are generated, so they come with exact truth: the table HTML, the values printed on each chart, and questions with known answers. For the real documents I hand-annotated seven tables (five of those are on the table-tier documents, which is what the table scores use) and wrote 176 questions by hand, 14 of them deliberately unanswerable and not scored. A script checked 155 of the 162 answerable ones against the PDF text. The other 7 ask about figures, so I checked those by eye, and I also hand-checked a random 20 of the whole set against the PDFs. Nothing needed fixing. Add the questions the scans inherit from their originals and the ones the generated documents bring, and each tool ends up facing 233 scored questions.
Eight extractors, each in its own virtualenv because their torch and transformers pins refuse to live together:
| Tool | Version | Kind | Ran on |
|---|---|---|---|
| PyMuPDF | 1.28.2 | text layer | Kaggle CPU notebook |
| pdfplumber | 0.11.10 | text layer + table finder | Kaggle CPU notebook |
| PyMuPDF4LLM | 1.28.2 with pymupdf-layout | layout model, Markdown output | Kaggle CPU notebook |
| Tesseract | 5.5.1 | OCR at 300 dpi | Kaggle CPU notebook |
| EasyOCR | 1.7.2 | neural OCR | Kaggle T4 |
| Docling | 2.130.0 | layout model + TableFormer + OCR | Kaggle T4 |
| Marker | 2.0.0, OCR disabled | layout model, Markdown output | Kaggle T4 |
| MinerU | 3.4.5, pipeline backend | layout model + OCR | Kaggle T4 |
Local tools only; hosted parsing APIs were out of scope for this round. Two of the local ones didn't get a fair shot, and I'd rather say so up front. Marker's OCR mode needs a running llama-server, which I don't have, so Marker ran on the text layer only. PaddleOCR-VL went past 8 GB on a single page in an early trial on a smaller machine and was dropped; I did not retry it under the 12 GB cap.
The rules. Every tool processes every document whole. Each worker gets 12 GB of RAM and 30 minutes per document; go over either and the run is recorded as oom or timeout and scored as zero. I wanted failures to be data, not something I quietly retried. Tesseract renders at 300 dpi, EasyOCR at 200. Both Kaggle machines have four vCPUs and 31 GB of RAM; the T4 notebook adds the GPU. One document needed an exception: Tesseract on the 487-page report hit 30 minutes in the full session, where Kaggle's CPU happened to be running at 4.6 s/page, and finished in 20 minutes when rerun alone with a 60-minute cap. Kaggle's CPU speed moves around between sessions, which matters later for the speed table.
Metrics. Character error rate for scans: the fraction of characters that are wrong, missing or extra against the truth text, so lower is better. TEDS for tables, which is short for Tree Edit Distance Similarity, the standard score from table-recognition research: the extracted table and the hand-annotated truth are both turned into HTML trees, the score counts the edits (cells added, removed, moved or changed) needed to turn one into the other, normalised by table size, and subtracts that from one. So 1.00 is a perfect reconstruction, 0.00 shares nothing with the truth, and a table with the right numbers in the wrong rows lands somewhere in between. For charts, the fraction of the facts printed on the chart that show up in the output. And for everything, hit@5: does a chunk containing the answer's evidence land in the top 5 after chunking, an in-memory dense search over that document's chunks (top 50), and DocuMind's reranker (down to 5). One index per document, so the tools are compared on extraction alone and nothing else. All four scores run from 0 to 1 and I quote them as decimals, so read 0.80 as 80%: 80% of the table structure right, 80% of the chart facts recovered, or the answer's chunk in the top five for 80% of the questions. Character error rate is the one where lower is better.
Three things that broke before I had a single result
The pipe
The runner launched each extractor as a subprocess and read its output through a pipe. Some of these tools are chatty. Progress bars, model-loading logs, a warning per page. A pipe buffer is 64 KiB. Fill it, and the tool blocks on its next write while the runner sits there waiting for an exit that will never come. Thirty minutes later the runner records a perfectly plausible timeout. Marker, whose progress bars go to stderr, would have been the first to go on a long document. A review caught this before any long document ran, so it never actually fired. But it would have, and I would have written "Marker times out on long documents" and believed it. Worker output goes to a file now. If your harness counts failures as data, make very sure the harness can't manufacture them.
Docling and my fake pages
The first version of the generated table documents was sparse: one small table floating on an otherwise empty page. Docling looked at it and returned <!-- image -->. Its layout model had decided the lone table was a picture. My first reaction was to blame Docling. My second, more useful one was that real pages have prose and headings and full-width tables, and that's what the model was trained on. I made the generator produce pages that look like pages, and Docling read them fine. A corpus that doesn't look like production documents ends up benchmarking your generator instead of the tools.
The answer key
There is no independent ground truth for the text of a real born-digital PDF. Nobody has hand-typed RFC 9110. So I'd used PyMuPDF's text layer as the reference for the clean-text tier, which is how PyMuPDF got its 1.00. There isn't a better truth to swap in. What I did instead was relabel that tier as "agreement with PyMuPDF" and make the headline number for clean text the retrieval metric, which only asks whether the evidence for a question can be found and doesn't care whose text it's in. Every benchmark has a tier like this somewhere. Say so in the table header.
Results
Tables
TEDS (tree edit distance similarity, the table-structure score from the metrics section; 1.00 is a perfect reconstruction) on the hand-annotated real tables. Only three documents in the corpus carry those annotations: the 9-page Census ACSBR-023 (one table), the 59-page Census P60-282 (three tables) and the 487-page Economic Report (one table). The last column is the four generated table documents, whose truth is exact because the generator wrote the tables itself: a plain table, one with merged cells, a rotated page, and a table that runs over four pages.
| Tool | Real tables (3 documents) | of which the 487-page report | Generated tables (4 documents) |
|---|---|---|---|
| Docling | 0.83 | 1.00 | 0.56 |
| MinerU | 0.80 | 0.94 | 1.00 |
| PyMuPDF4LLM | 0.76 | 0.91 | 0.57 |
| Marker (no OCR) | 0.74 | 0.58 | 0.54 |
| pdfplumber | 0.11 | 0.20 | 0.57 |
How to read it: 0.83 means that on average 83% of the table structure (which cells exist, where they sit, what they contain) matches the hand-annotated truth. Above 0.9 is close to a copy; below 0.5 is a table you would not trust a model to read a number from; pdfplumber's 0.11 means it found tables but almost none of their cell structure. On the generated tables MinerU rebuilt all four perfectly, merged cells, rotated page and all. Everyone else gets the plain table right (1.00), most of the merged-cell one (0.79 to 0.84), about a third of the four-page one (0.34) and almost nothing of the rotated page (0.00 to 0.10), which is why they all land near 0.55.
Docling is the most accurate table extractor in this corpus and MinerU second. On the real tables every layout tool found every annotated table; MinerU is the only one that also detected the table on the rotated generated page. PyMuPDF4LLM's 0.76, with no GPU and about a sixth of Docling's memory, is the number that kept it in the running as the CPU choice. Marker's real-table average is dragged down by the 487-page report (0.58); on the two Census documents it scored 0.82.
Scans
Character error rate, lower is better. 0.17 means 17 characters in every 100 are wrong, missing or extra compared with the original text. Tesseract and PyMuPDF4LLM ran on the CPU notebook, the other three on the T4.
| Tool | 4 pages, degraded | 9–16 pages | 59 pages | 194 pages | mean of the 5 scans |
|---|---|---|---|---|---|
| Tesseract | 0.29 | 0.36 | 0.44 | 0.17 | 0.32 |
| MinerU | 0.37 | 0.39 | 0.40 | 0.25 | 0.36 |
| PyMuPDF4LLM | 0.31 | 0.41 | 0.57 | 0.17 | 0.37 |
| Docling | 0.42 | 0.43 | 0.48 | 0.21 | 0.39 |
| EasyOCR | 0.34 | 0.48 | 0.62 | 0.18 | 0.42 |
The 0.17 on the clean 194-page scan is as good as it got, and some of that isn't OCR error at all, just reading-order and header differences against a text-layer truth. The short buckets are where the deliberately degraded scans live. The hardest scan, though, is an undegraded one: the 59-page Census report, whose dense statistical tables push every engine above 0.40. Tesseract wins or ties three of the four buckets and wins the mean, at 3.1 s/page and under 540 MB on a CPU. MinerU is the closest of the GPU tools on average and the best of them on the 9- to 59-page scans; on the clean 194-page scan EasyOCR (0.18) and Docling (0.21) beat it. EasyOCR is level with Tesseract only on that clean scan (0.18 vs 0.17) and trails everywhere else (0.34 vs 0.29 on the degraded 4-page scan, 0.62 vs 0.44 on the 59-page report). The three text-layer tools, and Marker without its OCR engine, return nothing on scans, so they are not in this table.
One row here sent me digging. PyMuPDF4LLM is a text-layer tool, so how was it scoring on image-only scans? Its worker logs answered: "Using Tesseract for OCR processing." Version 1.28 with pymupdf-layout runs whatever Tesseract it finds on image-only pages and on pictures, and every one of its 27 logs says so. So PyMuPDF4LLM's scan numbers are really Tesseract's numbers with PyMuPDF's layout analysis on top, and its per-page cost goes from 0.7 s on born-digital pages to 5.1 s on scans for the same reason. I didn't test it with Tesseract uninstalled; presumably it would then look like PyMuPDF on scans. Nothing wrong with the design. The capability table that was wrong was mine, which said "no OCR" for PyMuPDF4LLM until I read its logs.
Charts
Fraction of the facts printed on the four generated charts (titles, axis labels, category names, values) that appear anywhere in the output: 0.84 means 84 of every 100 facts survived extraction, 0.00 means none did.
| Tool | Chart facts recovered |
|---|---|
| EasyOCR | 0.84 |
| Tesseract | 0.65 |
| PyMuPDF4LLM (Tesseract on pictures) | 0.55 |
| MinerU | 0.07 |
| Docling | 0.03 |
| Marker, pdfplumber, PyMuPDF | 0.00 |
I didn't see this one coming. On these four charts the expensive layout tools recognise a chart, classify it as a picture, and emit a placeholder. The values are gone. The crude approach, OCR on the rendered page, keeps them. Four one-page charts and eleven figure questions is not much to stand on, and on the real figure-heavy NASA documents the retrieval stage puts Tesseract and EasyOCR level (0.90 each on the 59 NASA questions, structure-preserving chunker). Still, if your users ask questions whose answers live in charts, go and look at what your extractor does with a picture before you trust it.
Retrieval: hit@5 through the real pipeline
Each tool's output for each document was chunked, embedded with DocuMind's embedder, searched by dense top-50 in memory, and reranked to 5 by DocuMind's reranker. A question counts as a hit if one of those 5 chunks contains its evidence. 233 questions per tool. I ran two chunkers: the production one (with its budget lowered from the app's shipped 600 tokens to 450, under the embedder's 512-token window), and a structure-preserving one I'd been evaluating for the app that never splits a table mid-way. The four tier columns below are for the structure-preserving chunker; the last two columns give the overall score for each chunker.
| Tool | Clean text (48 q) | Tables (65 q) | Scans (57 q) | Figures (63 q) | All 233, structure-preserving | All 233, production chunker |
|---|---|---|---|---|---|---|
| Tesseract | 0.92 | 0.65 | 0.81 | 0.87 | 0.80 | 0.82 |
| MinerU | 0.96 | 0.60 | 0.86 | 0.83 | 0.80 | 0.81 |
| Docling | 0.94 | 0.66 | 0.84 | 0.78 | 0.79 | 0.80 |
| PyMuPDF4LLM | 0.94 | 0.66 | 0.84 | 0.71 | 0.78 | 0.79 |
| EasyOCR | 0.92 | 0.55 | 0.72 | 0.90 | 0.76 | 0.82 |
| PyMuPDF | 0.90 | 0.82 | 0.00 | 0.83 | 0.64 | 0.65 |
| pdfplumber | 0.94 | 0.62 | 0.00 | 0.81 | 0.58 | 0.59 |
| Marker (no OCR) | 0.83 | 0.60 | 0.00 | 0.67 | 0.52 | 0.50 |
How to read it: each cell is the share of that column's questions for which a chunk holding the answer's evidence made the top five, so Tesseract's 0.82 means 192 of the 233 questions. A 0.00 in the scans column means the tool produced no text for the scanned documents at all. One question moves an "All" score by 0.004 and a tier score by 0.015 to 0.021, so a difference of 0.01 is two or three questions overall and less than one question within a tier.
Three things here changed my mind.
The top five are within 0.04 of each other, and one of them is Tesseract. An OCR engine, pointed at born-digital pages, rendering perfectly good text into pixels and reading it back. With the production chunker it is level with EasyOCR at 0.82 for the best single extractor, two questions ahead. My guess is the reranker simply doesn't mind OCR noise. What actually separates the top five isn't retrieval; it's cost, memory, and whether you get table structure back. Tesseract gives you none. Docling and MinerU give you the best.
The production chunker beat the structure-preserving one for every tool except Marker. Keeping tables whole did not help retrieval on these questions, which was not what I expected when I wrote it.
And then the table tier, where plain PyMuPDF text (0.82) beats every table-aware tool (0.60 to 0.66; Docling reaches 0.77 with the production chunker), while TEDS says the exact opposite: the same table-aware tools win on structure. Retrieval finds the chunk; structure is what the generator needs to read the right cell. I very nearly published that from a run that couldn't support it. The chunker had measured its 450-token budget with tiktoken, and tiktoken packs digits more tightly than the embedder's WordPiece tokenizer. When I went back and counted every chunk of this data set with the embedder's own tokenizer, 13 to 56% of the table-aware tools' chunks were over the 512-token window and had been silently truncated (PyMuPDF4LLM 13%, Marker 25%, MinerU 30%, Docling 56%), against 5.7% of PyMuPDF's and 0.4% of Tesseract's. A reviewer caught it. I fixed the chunker and re-scored everything on a GPU. The fix moved PyMuPDF's table-tier score from 0.75 to 0.82 and PyMuPDF4LLM's from 0.72 to 0.66: smaller chunks helped flat text and hurt Markdown tables, which now split into more pieces. The finding survived its own correction, which is not something I'd have bet on that morning. Measure your chunk budget in the tokenizer your embedder actually uses.
Time per page
Least-squares fit of seconds on pages across each tool's 27 runs, so start-up cost and per-page cost come apart. Start-up is what every document pays (model loading, mostly); a 100-page document costs start-up plus 100 times the per-page figure. The CPU tools ran on Kaggle's CPU notebook, the rest on the T4, and both have the same four vCPUs, so comparing Tesseract with Docling here compares a CPU-only tool with a GPU-assisted one.
| Tool | Ran on | Start-up | s/page | 100-page document |
|---|---|---|---|---|
| PyMuPDF | CPU | 0.1 s | 0.008 | 1 s |
| pdfplumber | CPU | ~0 s | 0.16 | 16 s |
| Marker (no OCR) | T4 | 12 s | 0.28 | 40 s |
| MinerU | T4 | 34 s | 1.0 | 2.3 min |
| PyMuPDF4LLM | CPU | 36 s | 1.1 | 2.4 min |
| Docling | T4 | 16 s | 2.0 | 3.6 min |
| Tesseract | CPU | 49 s | 3.1 | 6.0 min |
| EasyOCR | T4 | 43 s | 3.7 | 6.9 min |
Two things about this table surprised me. The GPU tools are not fast, because the GPU only accelerates their model forward passes; page rendering, OCR pre- and post-processing (Docling's OCR engine, RapidOCR, runs on the CPU), and Markdown assembly all run on the four vCPUs. And Kaggle's CPU is not a constant: the same Tesseract on the same 487-page document ran at 4.6 s/page in one session and 2.4 in another, so treat the CPU-side figures as ±50%. Peak memory ran from 205 MB (PyMuPDF) through 535 MB (Tesseract) and 1.2 GB (PyMuPDF4LLM) to 2.0 GB (EasyOCR), 5.3 GB (Marker), 7.9 GB (Docling) and 10.8 GB (MinerU, on the 194-page scan).
The decision table
| If your PDFs are... | Use | Expect | Why |
|---|---|---|---|
| mixed, one tool, CPU only | PyMuPDF4LLM with Tesseract installed | 1.1 s/page, up to 1.2 GB | hit@5 0.78 to 0.79, TEDS 0.76 on real tables, no GPU needed |
| mixed, CPU only, tables do not matter | Tesseract 5 on every page | 3.1 s/page, up to 535 MB | hit@5 0.80 to 0.82, the best retrieval score of any tool, no table structure |
| mixed, a GPU and 8+ GB | Docling | 2.0 s/page on a T4, up to 7.9 GB | TEDS 0.83 on real tables, hit@5 0.79 to 0.80, drops chart text |
| clean text, speed first | PyMuPDF | 0.008 s/page, up to 205 MB | hit@5 0.90 to 0.96 on clean text and 0.82 on tables |
| tables, structure above all | MinerU or Docling, on a GPU | MinerU 1.0 s/page on a T4, up to 10.8 GB | MinerU TEDS 1.00 generated / 0.80 real, table detection 1.00 |
| scans | Tesseract 5 | 3.1 s/page, up to 535 MB | CER 0.17 on clean scans, 0.32 mean, hit@5 0.81 to 0.84 |
| charts whose numbers matter | EasyOCR on the page image | 3.7 s/page on a T4, up to 2 GB | 84% of chart facts and every figure question; layout tools drop them |
A per-tier router, a different extractor per kind of document, comes out at an estimated hit@5 of 0.88 against 0.80 for the best single tool. Treat that gently: it assumes you know the tier of a document before you extract it, and its clean-text pick wins by one question in 48. For my own system the conclusion is: on the CPU deployment I have today, PyMuPDF4LLM with Tesseract installed; the day a GPU joins the ingestion path, Docling; and either way, OCR on figure images when chart values need to be retrievable.
What could be wrong with all this
Two Kaggle machines rather than one: the CPU tools on a CPU notebook and the torch tools on a T4, same four vCPUs and 31 GB, but the T4 tools' per-page costs are GPU-assisted and the CPU tools' are not. Kaggle's CPU speed varies between sessions by up to a factor of two. One document ran with a 60-minute cap instead of 30 and finished inside 30. An earlier retrieval pass that counted the chunk budget in the wrong tokenizer; the numbers here are from the corrected pass. Different render resolutions for the two OCR engines (Tesseract 300 dpi, EasyOCR 200). Marker without its OCR engine. PaddleOCR-VL not run; local tools only. No truth for clean text beyond agreement with PyMuPDF. Synthetic scans, which the repository rebuilds from the source PDFs rather than shipping. Figure truth from four one-page generated documents, real table truth from five tables on three documents, so differences of a few hundredths are noise. Questions written and checked by me, with a script and a 20-question spot check, not by a second person. And the retrieval stage needs a GPU: the cross-encoder takes about 2 seconds per candidate pair on a CPU, which works out to roughly 60 hours for the whole stage.
Reproduce it
The harness, corpus builder, scoring, retrieval stage and report builder are in the repository: https://github.com/hraju115/pdf-extraction-benchmark. Each stage has its own virtualenv; from the repository root:
PYTHONPATH=. .envs/core/bin/python -m benchmarks.pdf_extraction.corpus.build
PYTHONPATH=. .envs/core/bin/python -m benchmarks.pdf_extraction.run
PYTHONPATH=. .envs/core/bin/python -m benchmarks.pdf_extraction.scoring
PYTHONPATH=.:src .envs/downstream/bin/python -m benchmarks.pdf_extraction.downstream.score_retrieval
PYTHONPATH=. .envs/core/bin/python -m benchmarks.pdf_extraction.report.build_report
Tesseract 5 needs to be installed system-wide, and PyMuPDF4LLM's scan and chart numbers depend on it. The shipped config.yaml caps workers at 4 GB; the Kaggle driver raises it to 12 GB (PDFBENCH_MAX_RSS_GB), and you need that, or the equivalent edit to config.yaml, to reproduce "nothing failed". The retrieval stage really wants a GPU; benchmarks/pdf_extraction/kaggle/ has the notebooks that ran all of this on free Kaggle machines. Runs resume by input hash, failures are kept as results, and every number in this piece comes from the CSVs those commands write, under benchmarks/pdf_extraction/merged/. One disclosure: I built the harness with an AI coding agent, working test-first, with independent reviews between steps. Of the problems in this article, the pipe, the answer key and the tokenizer were caught by those reviews, not by me staring at output; the Docling one was caught by the adapter tests. Make of that what you like. I take it as a point for reviews and tests rather than for the agent.
Top comments (0)