My Crawler Now Tells You What It Missed

The worst failure mode of a data pipeline is not an error. It's silent junk: a 404 page filed as content, a recipe with the ingredients dropped, a forum thread quietly missing its replies — all returned as "success." If you feed LLMs for a living, you have eaten this. I have shipped this.

PageSift 0.80 is the release where I tried to eliminate that failure mode from my own engine. The theme is not speed this time (0.70 was speed). The theme is: results should say what they are, including what they failed to get.

1. Results that confess

Extraction is heuristic and always will be. The bug is pretending otherwise. In 0.80, when extraction likely kept too little, the result says so — with numbers:

• If article extraction keeps only a small share of the page, you get an explicit incompleteness warning instead of a confident stub.
• If a forum page's replies were left out, the warning quantifies it: how many characters sat in comment sections, how much was extracted, and how to ask for the whole page if you want it.
• Video pages with no readable body don't return empty success anymore; they fall back to the page's embedded video metadata and label that they did.
• Error-status pages carry an explicit http_error_status warning and the real status code in metadata. Server errors (and empty error pages) fail hard; nothing passes as silent success.

These labels exist because my own production audit embarrassed me into them: 95% of my old crawler's rendered output was being thrown away, and nobody had noticed, because every result claimed to be fine.

2. PDF: from text extraction to structure preservation

0.80 replaces the third-party PDF library with a file layer I wrote myself. Headings are the difference between a document and a wall of text, so that's what I measured — on a set of real-world PDFs:

• Heading recall: 0.595 → 0.947
• Heading precision: 0.400 → 0.871
• Hierarchy consistency: 0.819 → 0.957
• Phantom tables: 196 → 9 (all 143 real tables kept)
• Memory: −25% at the same speed

3. Documents, all of them, all hand-written

Word, Excel, PowerPoint, EPUB and CSV are now supported, parsed by my own Rust code — including the ZIP and XML layers; the only borrowed piece is a decompression library. On a held-out set of files collected after the rules were frozen:

• Structure retention: 0.988 (MarkitDown: 0.717, pandoc: 0.705)
• 81 files parsed in 657 ms total (MarkitDown: ~48.8 s)
• Median peak memory per file: 14.2 MB (MarkitDown: 140.1 MB)

4. HTML parsing, closer to the spec

The parser now passes 89.3% of the official html5lib test suite (was 77.7%); parsing differences that actually change the extracted content dropped from 143 to 6. Parsing got faster while getting more correct, which was not the direction I expected.

5. Fuzzing: the inputs that used to kill it

A fuzz pass found inputs that hung the engine or blew up memory. Before → after:

• 720 KB of pathological text: 19.5 s → 24 ms
• 1.6 MB of adversarial HTML: 52 s → 0.19 s
• One PDF: killed at >3 GB memory → 81 ms at 73 MB

Browsers orphaned when the process is killed are now reaped on the next start, too.

The rule I held myself to

This was the correctness release, and the standing rule was: correctness may not be bought with speed. Against 0.73, single-thread extraction is 3.7% faster, 16-thread 4.4% faster, PDF memory down 25%, per-request CPU flat. The rule held.

Still open, honestly

Steam's real age gate still slips past classification — it comes back flagged only as thin content, not recognized as a gate. It's been the standing exception for three releases now, and I'd rather list it here than let you find it. Deletion tracking for sync pipelines (gone pages as a first-class signal) isn't built yet either; a reader asked for exactly that, and it's on the list.

GitHub: https://github.com/YOLOW92/pagesift — the 0.80 release notes have the full tables. Next on the 0.8x line: the hosted API. If your pipeline has a silent-junk horror story, I want to hear it in the comments.