A web-to-Markdown request can succeed while leaving your retrieval pipeline with incomplete evidence. The output looks readable, the ingestion job finishes, and the missing material becomes somebody else's debugging problem.
We tested Firecrawl, Jina Reader, and a DIY trafilatura adapter against our own documentation pages with known ground truth. In this sample, Firecrawl’s table cells used an equivalent Markdown escape, Jina missed some exact fenced code content, and trafilatura admitted boilerplate. Their headline content scores alone do not explain those differences.
The ScraperWatch RAG-ingestion benchmark has the results and calculator. The table retains the historical first run: October 19, 2026, 09:26:50–09:28:15 UTC, saved in products/ragbench/data/results.jsonl and data/summary.json. Later offline rescoring is identified separately; there has been no new provider run in this review.
We created ten fictional OrbitForge API documentation pages, roughly 900–1,000 words each, paired with reference Markdown. The fixtures include navigation, sidebars, cookie text, code, and tables. They also include a page rendered from inline JSON using JavaScript and SDK tabs whose reference includes the initially hidden example.
For example, the rate-limits reference keeps a curl request, a throttling warning, and a table mapping id, state, and observed_at to types and meanings, while excluding navigation and cookie text.
Content scoring uses lowercased word and punctuation tokens, ignores whitespace, and measures multiset overlap. Recall measures reference content recovered; precision measures how much extracted content matches the reference. F1 combines them. These scores measure coverage, not ordering, semantic correctness, or downstream answer quality.
Repository adapter settings for this comparison: Firecrawl v1 Markdown with onlyMainContent=True, keyless Jina Reader, and static-HTML trafilatura with tables enabled. Each completed ten page attempts with zero reported failures. Values are means of per-page scores, except median adapter latency; code scores are means of per-page exact fenced-body fractions:
| Tool | Recall | Precision | F1 | Exact code preservation | Legacy literal table match | Boilerplate leakage | Median latency |
|---|---|---|---|---|---|---|---|
| Firecrawl | 96.17% | 99.17% | 97.64% | 100% | 0% | 0% | 1.134 s |
| Jina Reader | 95.55% | 100% | 97.72% | 96.67% | 100% | 0% | 2.888 s |
| DIY trafilatura | 86.53% | 91.64% | 87.94% | 90% | 0% | 10% | 0.017 s |
The table column retains the original scores for traceability. That legacy check requires outer-pipe Markdown tables with matching ordered cells after lowercasing and whitespace normalization, but retains Markdown escapes. It does not recognize HTML tables or valid Markdown without outer pipes. Failure alone establishes neither missing facts nor lost structure.
An October 19 offline audit of those same saved outputs found one recognized table on every Firecrawl and Jina page. Ordered cells match after decoding the escaped underscore: Firecrawl writes observed\_at, while the reference and Jina write observed_at. The narrow table_rate_v2_escaped_underscore check changes Firecrawl from 0/10 to 10/10 pages and leaves Jina at 10/10, with no unavailable outputs. products/ragbench/data/table-score-audit.json pins output and fixture hashes. The current runner adds both table metrics to future scores; it does not backfill historical rows. Reproduce offline with python3 products/ragbench/evaluate_saved.py --output-dir products/ragbench/review/post3-recheck using a fresh directory. This recomputation is neither a new provider run nor the separate synthetic sample. It checks ordered cells in recognized tables, not general structural completeness, rendering or downstream RAG quality.
The original 0/10 versus 10/10 table gap therefore does not demonstrate a structural advantage. If your corpus describes response schemas, pricing matrices, or compatibility rules, inspect those outputs and their encodings before relying on aggregate F1.
Both providers retained the recognized pipe-table rows and cells in these ten closely related schemas; Jina achieved the highest legacy token F1 here, narrowly. Its 96.67% code score means some fenced block bodies failed exact preservation, including case and whitespace. That is stricter than testing whether code remains executable. Its measured latency was also higher, but the runner includes Jina's pacing wait and starts requests at least three seconds apart. This is not an isolated measurement of provider processing speed.
Trafilatura was fast in this setup, with lower coverage and 10% mean boilerplate leakage. Leakage measures the fraction of output tokens in recognized boilerplate spans, averaged across pages; it is not the percentage of pages affected. The detector uses known phrases, so rewritten boilerplate can escape it. The DIY adapter fetches static HTML and cannot execute the JavaScript fixture, another limit to consider for your corpus.
Before choosing a tool, build a small reference set from your actual document types. Check tables when relationships between cells matter, compare code blocks, and inspect both missing content and unwanted navigation. Keep representative output checks in your ingestion workflow so a successful request does not become your only quality signal.
Then calculate cost at your volume. The calculator's October 19 snapshot assumes one Firecrawl credit per basic Markdown scrape: the saved pricing file lists 5,000 monthly Hobby credits for $16 per month billed annually, or $19 billed monthly. Jina's paid output-token price is unverified in the file; an unknown price is not zero. The DIY $5 monthly VM budget is an assumption, excluding engineering time and extra bandwidth. Throughput determines its cost per page.
Use the benchmark and cost calculator alongside your own output checks. The Firecrawl link on the site is an affiliate link. This small fixture suite shows specific extraction tradeoffs; it does not establish a universal winner.
Written with AI assistance. Every number comes from code over public data.
Top comments (0)