Published: Jun 4, 2025 · Updated: Nov 20, 2026
Comparing PDF and Webpage Extractors: PyMuPDF 2.0, Docling, Reducto, extract-pdf, and More
PDF extraction now encompasses more than retrieving text. Depending on the application, useful output may mean readable prose, tables with intact cell structure, layout-aware Markdown or JSON, or clean HTML suitable for publishing and research.
This revised comparison adds PyMuPDF 2.0 as a candidate for layout-aware document extraction and introduces extract-pdf and its companion, extract-webpage, for converting PDFs and webpages into structured HTML.
It also includes a proposed benchmark corpus, scoring definitions, and sample results tables to show how a reproducible comparison could be reported.
Executive Summary
The most useful comparison starts by separating extraction tasks.
- PyMuPDF should be evaluated for structured document extraction, not only plain-text speed. Its documentation describes integrated layout analysis, reading-order reconstruction, structured output, table extraction, and automatic OCR.
- extract-pdf adds an HTML-oriented workflow that uses the PDF text layer first and can route selected pages through Granite Docling for additional processing.
- extract-webpage extends that approach to main-content extraction and citation metadata from URLs. It belongs in a separate webpage track.
- Docling, Reducto, PDFMiner, Camelot, Tabula, and Unstructured remain comparison candidates, but exact versions, configurations, and supported outputs must be verified before testing.
- Initial output and completed output are different milestones. A fast text-layer response should not be compared directly with another tool’s completed OCR-enriched result.
- No overall winner can be established from product descriptions or the illustrative numbers in this article.
The practical question is not simply “Which extractor is best?” It is “Which configured workflow produces the required output accurately enough, at an acceptable latency and cost?”
What Changed from the Original Comparison
The original article characterized PyMuPDF’s layout analysis as basic, marked OCR support as absent, and primarily recommended it for fast text extraction. The capabilities described in its newer documentation call for a broader evaluation.
The revised scope also adds clean HTML as a distinct output requirement and separates webpage extraction from PDF parsing.
| Area | Original treatment | Revised evaluation |
|---|---|---|
| PyMuPDF layout analysis | Basic | Test advertised integrated layout analysis and reading-order reconstruction |
| PyMuPDF OCR | Listed as unsupported | Test advertised automatic OCR behavior and configuration |
| PyMuPDF structured output | Secondary to text extraction | Evaluate Markdown, JSON, text, table structure, and source grounding |
| HTML-oriented PDF extraction | No dedicated entry | Add extract-pdf’s text-layer and selectively enriched HTML workflows |
| Webpage extraction | Outside the comparison | Add a separate track for main content, metadata, and fallback behavior |
| Numerical rankings | Presented as comparative scores | Withhold product rankings until a reproducible benchmark has run |
These changes expand the questions a benchmark must answer. They do not, by themselves, demonstrate that one product outperforms another.
PyMuPDF 2.0: Beyond Plain-Text Extraction
PyMuPDF’s documentation describes a layout-aware document engine, bringing capabilities previously associated with PyMuPDF, PyMuPDF4LLM, and Layout into the main package.
Advertised capabilities include:
- Natural reading order and recognition of document elements.
- Markdown, JSON, and text output.
- Enhanced table extraction, including cell structure and data-type recognition.
- Automatic OCR based on page-content analysis.
- Connections between extracted content and source locations, including table cells.
These capabilities change how PyMuPDF should be tested. Plain-text extraction remains one workload, while layout-aware extraction and OCR-enabled processing are separate workloads with different completion requirements.
A benchmark should therefore identify the exact API, extraction mode, dependencies, and settings used for each run. Reporting a single “PyMuPDF speed” would obscure those differences.
The website also advertises accuracy and processing-speed results. Those remain vendor-reported measurements unless reproduced on the benchmark’s own corpus under disclosed conditions.
extract-pdf: PDFs as Structured HTML
extract-pdf accepts a PDF URL or an ArrayBuffer and targets structured HTML output. Its documentation describes support for headings inferred from font size, lists, footnotes, code blocks, quotations, and table-of-contents links.
Text Layer First, Additional Processing Where Needed
The project describes a workflow that extracts available text first, then identifies pages that may need additional processing.
A regex-based scan flags scanned pages, tables, and figures. Selected pages can then be processed with Granite Docling, either in-process or through a remote processor.
A benchmark should treat this as a staged workflow:
- Initial text-layer HTML becomes available.
- Selected pages receive additional processing.
- The completed, enriched output becomes available.
Each stage needs its own timing and quality measurements. An initial response may be useful before completion, but its latency does not represent the time required to produce the final output.
The project’s claims of millisecond-scale text-layer extraction should remain clearly labeled as project claims until measured on a defined corpus.
Runtime and Parsing Options
The documentation describes three parsing options:
- A pure-TypeScript block algorithm.
- LiteParse running natively in Node.
- LiteParse through WebAssembly in browsers and Workers.
It also describes obtaining PDF.js through pdfjs-serverless at runtime to keep Node-only engines out of the Worker bundle.
These deployment options make extract-pdf relevant when TypeScript integration, browser or Worker execution, and HTML output are explicit requirements. Each backend should be recorded as a separate benchmark configuration rather than combined into one result.
extract-webpage: Main Content and Citation Metadata
extract-webpage complements the PDF workflow by extracting webpage content and citation metadata.
Its documentation describes running Readability and Mercury side by side, selecting the longer extracted article, and simplifying the result into headings, paragraphs, images, and links.
For citation metadata, the project describes extracting author, date, title, and publisher information from metadata and class names, with author-name checking against a names database.
These fields still require manually checked references. Finding a date is not necessarily the same as identifying the correct publication date, and extracting a name does not establish that it is the article’s author.
Other documented capabilities include:
- Routing PDF URLs through extract-pdf.
- Processing DOCX buffers through JSZip.
- Fetch fallbacks involving a Cloudflare Puppeteer scraper and Jina Reader.
- Bot-detection checks and robots.txt rules.
- Sentence splitting, semantic chunks, stemming, and keyphrase extraction.
Because fallback services can change both completion rates and latency, direct fetching, browser-assisted extraction, and external-reader processing should be reported separately.
Comparison Tracks
A fair comparison groups tools by the output they are configured to produce.
| Track | Candidates or scope | Primary measurements |
|---|---|---|
| Plain-text PDFs | PyMuPDF, PDFMiner, and other applicable configurations | Text fidelity, omissions, reading order, latency, memory |
| Structured PDFs | PyMuPDF, extract-pdf, and verified configurations of Docling, Reducto, and Unstructured | Document elements, reading order, tables, source grounding |
| Tables | Applicable configurations, including Camelot and Tabula where supported | Cell text, row/column assignment, merged cells, structural correctness |
| Clean HTML from PDFs | extract-pdf and other configurations producing comparable HTML | Semantic structure, content preservation, boilerplate, validity |
| OCR and mixed-content PDFs | Verified OCR-enabled configurations | Recognition accuracy, page selection, completed-output latency, cost |
| Webpages | extract-webpage and any additional webpage-specific candidates | Main-content precision and recall, metadata, fallback behavior |
Candidate inclusion does not imply that every tool supports every track. Mark unsupported or intentionally disabled outputs as “not applicable,” not as zero-quality results.
Proposed Benchmark Corpus
The following corpus is a proposed design, not an existing published dataset.
PDF Dataset
Use 100 PDFs totaling 1,000 pages:
| Document category | Documents | Pages | Primary challenges |
|---|---|---|---|
| Simple native-text documents | 20 | 100 | Text fidelity, paragraph boundaries |
| Multicolumn research papers | 20 | 200 | Reading order, captions, footnotes |
| Financial reports | 15 | 300 | Dense tables, repeated headers, numeric fidelity |
| Table-focused documents | 15 | 100 | Borderless tables, merged cells, row/column structure |
| Scanned documents | 15 | 150 | OCR, rotation, low-quality images |
| Mixed native-text and scanned documents | 15 | 150 | Selective OCR, content preservation |
| Total | 100 | 1,000 |
The corpus should also contain lists, code blocks, quotations, footnotes, captions, and multipage tables.
Create reference annotations for the relevant tasks. Reserve a development subset for configuration tuning, then freeze the evaluation subset before comparing tools.
Publish file hashes and document identifiers so every configuration can be tested against the same inputs.
Webpage Dataset
Use a separate collection of 60 webpages:
| Page category | Pages | Primary challenges |
|---|---|---|
| News and editorial articles | 15 | Main content, bylines, publication dates |
| Technical documentation | 15 | Headings, code blocks, lists, navigation removal |
| Cluttered content pages | 15 | Ads, related links, repeated boilerplate |
| Rendering-dependent pages | 15 | Browser fallback, delayed content |
| Total | 60 |
Preserve reference snapshots and required assets for extraction-only tests. Run live-fetch tests separately, because network conditions, site changes, and access restrictions introduce additional variability.
Scoring the Outputs
The benchmark should publish its scoring rules before evaluating the held-out documents.
| Metric | Proposed definition | Preferred direction |
|---|---|---|
| Character error rate | Character insertions, deletions, and substitutions divided by reference characters | Lower |
| Reading-order accuracy | Correctly ordered pairs of matched reference blocks divided by evaluated pairs | Higher |
| Table cell-text F1 | F1 of nonempty cells with correct normalized text and row/column coordinates | Higher |
| Table span accuracy | Correct row/column spans among matched reference cells | Higher |
| HTML semantic F1 | F1 of matched semantic elements, using disclosed content and element-type matching rules | Higher |
| Main-content precision | Correctly extracted reference-content tokens divided by all extracted tokens | Higher |
| Main-content recall | Correctly extracted reference-content tokens divided by reference tokens | Higher |
| Citation-field accuracy | Correct metadata fields divided by evaluated fields | Higher |
| Completion rate | Inputs producing valid, nonempty required output divided by attempted inputs | Higher |
| End-to-end latency | Wall-clock time from submission to completed required output | Lower |
| Peak memory | Peak memory across the process and its child processes | Lower |
Token matching must account for repeated text rather than simply checking whether a word appears anywhere in the reference. Likewise, semantic scoring needs explicit rules for matching elements and evaluating heading levels.
For citation metadata, report author, title, publication date, and publisher separately, as well as any aggregate. Define how absent reference fields, multiple authors, and publication-versus-update dates are handled.
For source grounding, score page-number correctness and bounding-box agreement separately where both the configuration and references support them.
Aggregation and Failures
Publish per-document scores and category-level summaries. State whether each aggregate gives equal weight to documents, pages, characters, tables, or cells.
Quality scores calculated only on successful outputs can conceal failures. Always show the completion rate beside them, and publish any failure-adjusted score as a separately defined measure.
Illustrative PDF Benchmark Results
The table demonstrates reporting for the same hypothetical 100-page scanned-document subset. Assume OCR models are already loaded, concurrency is one, and completed output is required.
| Configuration | Character error rate ↓ | Reading-order accuracy ↑ | Median seconds/page ↓ | p95 seconds/page ↓ | Peak memory, GiB ↓ | Completed pages ↑ |
|---|---|---|---|---|---|---|
| A | 1.8% | 96.2% | 0.84 | 1.76 | 1.2 | 100/100 |
| B | 1.3% | 97.5% | 1.32 | 2.48 | 2.1 | 100/100 |
| C | 2.6% | 93.8% | 0.61 | 1.42 | 0.9 | 98/100 |
| D | 1.1% | 97.0% | 1.65 | 3.12 | 2.8 | 100/100 |
- D has the lowest character error rate.
- B has the highest reading-order accuracy.
- C has the lowest median latency and memory use, but fails on two pages.
- A falls between the other configurations on the displayed quality and latency measures.
The quality and latency figures describe completed pages only. The completion column makes that limitation visible.
Even a real table with these results would not establish a universal winner. It would describe performance on a particular workload under particular settings.
Illustrative Staged-Extraction Results
For workflows that return text-layer output before OCR finishes, report both milestones.
| Stage | Median seconds/document ↓ | Text coverage ↑ | HTML semantic F1 ↑ |
|---|---|---|---|
| Initial text-layer output | 0.18 | 73.4% | 0.81 |
| Completed OCR-enriched output | 4.90 | 98.1% | 0.94 |
These synthetic figures illustrate the reporting distinction. Measure both stages against the same reference documents, and measure elapsed time from the same starting point.
The initial output’s availability time is not a substitute for completed-output latency.
Illustrative Webpage Benchmark Results
The following synthetic table compares three processing policies on the same hypothetical 60-page corpus.
Fallback is attempted only when direct extraction fails a predefined acceptance check.
| Processing policy | Main-content precision ↑ | Main-content recall ↑ | Citation-field accuracy ↑ | Median seconds/page ↓ | Completed pages ↑ |
|---|---|---|---|---|---|
| Direct fetch only | 97.1% | 86.4% | 89.6% | 0.42 | 49/60 |
| Direct fetch + browser fallback | 96.0% | 96.8% | 93.8% | 1.15 | 58/60 |
| Direct fetch + browser/reader fallback | 94.7% | 98.2% | 94.2% | 1.38 | 60/60 |
Because quality is calculated on completed pages, the rows do not have identical quality denominators. An executed benchmark should also publish comparisons on the subset completed by every policy.
Report how many pages used each fallback, the acceptance rule, the provider, external charges, and the full end-to-end time including failed attempts before a successful fallback.
Reproducibility Requirements
A useful results table needs a reproducible execution record.
Software and Configuration
Record:
- Exact package versions and source revisions where applicable.
- Model names, versions, and availability.
- Extraction modes and output formats.
- Parsing backends.
- OCR languages and page-selection settings.
- Timeouts, retry policies, and fallback rules.
Execution Environment
Disclose:
- CPU, RAM, GPU, and operating system.
- Thread counts and concurrency.
- Local versus remote processing.
- Network conditions for live-service tests.
- Whether model loading, downloads, and initialization are included.
Timing Procedure
Measure cold starts separately from warm runs. Use repeated timing runs, disclose the repetition count, and state how medians and p95 values are calculated.
For batch workloads, report throughput as well as latency. Do not derive a batch-throughput claim solely from a single-page timing.
Published Artifacts
Release, where permissions allow:
- Corpus files or stable acquisition instructions.
- Reference annotations.
- Raw extraction outputs.
- Per-input timings and resource measurements.
- Failure and retry logs.
- Scoring scripts.
- Configuration files.
Manual cleanup should be recorded separately. An output requiring human repair should not be presented as equivalent to an automatically usable result without disclosing that intervention.
Pricing, Licensing, and Deployment
Licensing and deployment requirements should be evaluated separately from extraction quality.
The extract project identifies its license as rights.institute/PROSPER. List that stated license rather than assuming MIT, Apache, or equivalent terms. Review the actual terms for the intended deployment.
Do not carry forward prices or licensing summaries from the June 2025 comparison as current quotations without checking them.
For each tested configuration, report:
| Cost category | What to disclose |
|---|---|
| Software licensing | Applicable license or commercial agreement |
| Local infrastructure | Compute time, memory requirements, hardware assumptions |
| OCR processing | Local model execution or remote processing charges |
| API services | Billed requests, pages, tokens, or other charging units |
| Fallback services | Browser and external-reader usage |
| Human intervention | Cleanup time and review assumptions |
If calculating cost per 1,000 pages or documents, specify the workload mix and whether retries, failures, and fallback attempts are included.
How to Interpret the Eventual Results
The benchmark should support task-specific decisions rather than force every workload into one ranking.
- A plain-text workflow should be judged primarily on fidelity, reading order, completion, and speed.
- A table workflow needs cell and structural evaluation, not just readable text.
- A publishing workflow needs semantic HTML and content preservation.
- An OCR workflow needs recognition quality and completed-output latency.
- A webpage workflow needs main-content quality, metadata correctness, and transparent fallback behavior.
An optional overall score can be published, but its normalization and weights must be explicit. Readers should still be able to inspect the underlying measurements.
Conclusion
The revised comparison expands beyond plain-text PDF parsing to cover structured documents, tables, clean HTML, OCR, and webpage extraction.
PyMuPDF’s advertised layout-aware capabilities justify broader testing. extract-pdf adds a text-layer-first HTML workflow with selective additional processing. extract-webpage introduces a separate evaluation problem involving main content, citation metadata, and network-dependent fallbacks.
The sample tables show how those workflows could be reported without inventing product rankings. They are not evidence of actual extractor performance.
Top comments (1)
Nice - I wonder if you could benchmark this again with PyMuPDF >= 2.0 as it now has advanced layout analysis built in. Would be interesting to see how things compare!