DEV Community: ゴエンチョウ

One Engine, Seven Open-Source Projects: PageSift 1.4 and the yolow Family

ゴエンチョウ — Sat, 14 Nov 2026 01:37:54 +0000

PageSift started as the ingestion layer of an AI research platform I was building: something had to fetch web pages and documents and turn them into clean Markdown that an LLM pipeline could actually use. It grew into a self-hosted web scraper, crawler and document parser — Firecrawl- and Jina-compatible API, MCP server, Python/Node SDKs, one Rust binary.

Over the 1.x series I kept doing one thing: every time a piece of the engine turned out to be generally useful, I cut it out and gave it its own repository, its own versioning, and an MIT license. PageSift 1.4 finishes that split — and adds the subsystem the extraction was making room for.

PageSift 1.4

The split, completed. Wave one (1.2.7) turned the base64, PDF, HTML and content crates into thin shims over standalone libraries. Wave two (1.4.0) does the same for the remaining three: pagesift-office → yolowdoc, pagesift-syntax → yolowsyntax, pagesift-compress → yolowzstd. The engine is now a composition of seven independently versioned open-source projects — and since 1.3, PageSift itself is Apache-2.0 (it was AGPL before).

DataGraph. The new crate, pagesift-graph (~4,000 lines), is what the cleanup was for. A normal scraper returns text and forgets everything else. DataGraph keeps the receipts:

Every fetch becomes a node; links between documents become citations.
Provenance is recorded as edges: redirect chains, canonical alias clusters, attachments and their parent documents.
Refetch the same resource and you get version lineage instead of an overwrite.
Extracted fields carry evidence spans — byte offsets into the source document, so any claim can be pointed back at the exact text it came from.
Claims about the same entity (keyed by subject keys like sku) are clustered into facts across sources, and when sources disagree you get a conflict record: both variants, both excerpts, and a preferred variant with its reason. Two pages pricing the same product at $10 and $12 stop being a silent data-quality bug and become a queryable fact.

It's exposed everywhere PageSift is: CLI (scrape --graph, crawl --graph, graph inspect), the HTTP API (a graph slice on scrape responses, plus /v2/graph/history and /v2/graph/conflicts), the MCP tool pagesift_graph_query, and the Python and Node SDKs.

An LLM pipeline doesn't just need text. It needs to know where a claim came from, what it contradicts, and which version is current. That's the layer DataGraph adds.

The six libraries

yolow64 0.3.0 — base64, ~20 GB/s

A fast Base64 codec with runtime-dispatched SIMD (AVX2/NEON), zero runtime dependencies, streaming and data-URL APIs. The 0.3 release added dedicated short-input kernels and fused hash-table tail processing: around 20.4 GB/s from 8 KiB to 1 MiB, and short payloads (8–64 bytes) got 1.5–2.8× faster than in 0.2. Short inputs are the common case in real pipelines — tokens, hashes, embedded blobs — so that's where the work went.

cargo add yolow64 · GitHub · crates.io

yolowpdf 0.1.2 — PDF to Markdown, no C libraries

A PDF-to-Markdown library and OCR engine that parses the format itself — no Poppler, no MuPDF, nothing to link against. The workspace includes GPU-accelerated OCR (PP-OCRv6). The README benchmark clocks 1,900 pages/s on born-digital documents (i7-10700). This is the component that taught the family its hardest rule: every allocation a hostile file can influence is budgeted before it happens.

cargo add yolowpdf · GitHub · crates.io

yolowhtml 0.1.0 — arena HTML5 parsing and extraction

Two crates: yolowhtml-dom, a zero-copy HTML5 parser over a contiguous arena (Vec<Node>, u32 ids, CSS selectors), and yolowhtml, a single-pass main-content extractor that produces Markdown, metadata and publication dates from the same parse. On the README's test corpus it scores 1.000 structural F1 for Markdown across 15 page categories, and 0.945 F1 for publication-date extraction across 20 languages. Both crates are #![forbid(unsafe_code)].

cargo add yolowhtml · GitHub · crates.io

yolowzstd 0.1.2 — pure-Rust Zstandard

A Zstandard (RFC 8878) encoder/decoder written from the spec: zero dependencies, #![forbid(unsafe_code)], multi-core encode/decode that emits standard concatenated frames — interoperability is tested against the official zstd CLI. It's built for the small-record end of the world: on 64-byte payloads it decodes at 3,257 MB/s (1.6× C libzstd) and encodes at 264 MB/s (4.1×) on an i7-10700, which is exactly the shape of PageSift's cache records. One stated gap: no dictionary support yet.

cargo add yolowzstd · GitHub · crates.io

yolowsyntax 0.1.2 — tree-sitter, without the C

A pure-Rust runtime for tree-sitter grammars: the C runtime reimplemented in Rust, with 41 languages' parse tables converted and embedded — 6.22 MiB total, Zstd-compressed (with yolowzstd, its only runtime dependency) and decompressed lazily per language. It extracts code outlines for 32 symbol-bearing languages. No C FFI, #![forbid(unsafe_code)]. Stated limits: no Query DSL and no incremental re-parsing.

cargo add yolowsyntax · GitHub · crates.io

yolowdoc 0.1.1 — every document format, one call

Documents to Markdown behind a single convert(bytes, format, opts): Office (DOCX/XLSX/PPTX, plus legacy DOC/XLS/PPT and XLSB), OpenDocument, RTF, WordPerfect, HWP/HWPX, e-books (EPUB/MOBI/FB2), email (EML/MBOX/MSG), archives (ZIP/TAR/7Z/GZ/BZ2/XZ — the decompressors are pure Rust too), SQLite, dBASE, CSV and Jupyter notebooks. The per-format parsers go deep — pivot tables, charts, equations — and a Limits budget is checked before any work starts.

cargo add yolowdoc · GitHub · crates.io

How they fit together

The family is a small lattice, not a pile: PageSift depends on all six libraries; yolowsyntax depends on yolowzstd; yolowdoc builds on both; yolowpdf uses yolow64. The rules are shared across all of them — pure Rust, no C dependencies, unsafe code forbidden in the libraries, and resource budgets enforced before allocation, because every one of these parsers eats untrusted input for a living.

PageSift is the engine. These six are its organs, now living their own lives — each usable without the others, each on crates.io under MIT, with PageSift itself under Apache-2.0.

Links: PageSift · yolow64 · yolowpdf · yolowhtml · yolowzstd · yolowsyntax · yolowdoc

Try them, break them, and tell me where they break — issues and benchmarks from other machines are the most useful feedback I can get.

yolow64 0.2: a new decode kernel that beats Turbo-Base64 under full validation

ゴエンチョウ — Wed, 11 Nov 2026 15:32:02 +0000

yolow64 0.2: a new decode kernel that beats Turbo-Base64 under full validation

yolow64 is the Base64 codec extracted from PageSift, published as 0.1 on Nov 9. Version 0.2 (plus a docs-only 0.2.1) is out, and the headline is decode.

Decode: new kernel

The bulk decoder was replaced with a four-block AVX2 hash loop using clean-room translation/validation tables, speculative validation, and per-quad constants held in registers. Full validation is retained for every block — nothing was traded away for speed. BMI2 pext tails handle padded 24- and 172-character inputs, and exact-length no-padding inputs of 16/32/64/128 characters bypass the general dispatch entirely.

The comparison that matters is against Turbo-Base64 built with FULLCHECK=1. Turbo's default build checks only the first of four blocks in its hot loop; comparing against that would be comparing against 25% validation. With validation equalized (GB/s, i7-10700, six-process medians):

Input	yolow64 0.2	Turbo-Base64 FULLCHECK	Delta
16 KiB	19.54	19.68	−0.7%, tie
64 KiB	20.08	19.36	+3.7%
256 KiB	20.16	16.76	+20.3%
1 MiB	20.17	16.68	+20.9%
4 MiB	17.06	13.89	+22.8%

Short inputs (ns/call): first at 12 B (7.00), 16 B (8.53), 24 B (10.14), 48 B (12.20), 96 B (15.68) and 128 B (17.18) in an eight-library run. Not first at 8, 32 and 64 B — those cells are still someone else's, and the raw tables in the repo say so.

Encode: streaming and cache-aware

Adaptive non-temporal stores kick in above 3/7 of the detected LLC.
A streaming encode API and a memory-bounded encode_to_writer join the one-shot calls.
An optional parallel feature splits large inputs across cache-aware workers — and stays out of the way when the single-thread streaming path measures faster.
Short 8/16/32-byte fast paths skip the generic tail.

Strictness went up, not down

Padding-bit checks now reject non-zero trailing bits at the short padded lengths and at the generic tail. AVX2 and BMI2 are detected independently: pext paths run only when both are present, AVX2-only and scalar CPUs fall back cleanly — a test-only switch forces the no-BMI2 path on a BMI2 machine to prove the fallback decodes identically. Exact-output canary tests cover caller-provided buffers from 1 B to 8 KiB, so the fast paths cannot write past your buffer.

Install

[dependencies]
yolow64 = "0.2"

Zero runtime dependencies, MIT, MSRV 1.89.

Caveats, as always: one shared machine (i7-10700, 16 GiB, AVX2+BMI2, no AVX-512), medians of repeated runs, snapshots rather than guarantees. Method, raw CSVs and the losing cells are in the repo: https://github.com/YOLOW92/yolow64 — the release record is research/release-0.2.0.md.

Its sibling extraction, yolowpdf, also landed on crates.io this week. If you push Base64 through a hot path, I'd like to hear where 0.2 lands on your hardware.

yolowpdf: a PDF-to-Markdown parser written from scratch in Rust, plus its own OCR engine

ゴエンチョウ — Mon, 09 Nov 2026 15:55:38 +0000

yolowpdf: a PDF-to-Markdown parser written from scratch in Rust — plus its own OCR engine

PDF parsing in Rust usually means wrapping a C library. yolowpdf reads the file format itself: no Poppler, no MuPDF, no native PDF library at all.

It is the PDF engine extracted from PageSift (from the 1.2.4 tree), now an independent MIT-licensed workspace. It is also the second spin-off in a week, after yolow64.

Why it exists

PageSift needed to turn PDFs found in the wild into clean Markdown without installing a native PDF stack on every machine. Wild PDFs are not clean: broken xref tables, object streams, encrypted streams, CID fonts with no ToUnicode map, multi-column layouts, tables, inline formulas, images, and scanned pages with no text layer at all.

Handling all of that grew into a standalone reader: file structure and damaged-file repair, fonts (Type1, TrueType, CFF, CID, predefined CMaps, CJK collections without ToUnicode), layout reconstruction (paragraphs, headings, reading order, headers/footers, lists), ruled and borderless tables, inline math rebuilt as LaTeX from glyph metrics, image extraction (JPEG, JPEG 2000, JBIG2, CCITT, PNG, TIFF, WebP) — and a PageOcr trait for scanned pages.

The OCR side became a second crate: yolowpdf-ocr implements PP-OCRv6 detection and recognition, with optional document-layout and formula models, on CPU or an NVIDIA GPU, with automatic fallback to CPU and a reported reason when the GPU cannot start. The core crate never depends on a model runtime; OCR is opt-in.

The API is deliberately small:

let bytes = std::fs::read("document.pdf")?;
let doc = yolowpdf::extract(&bytes)?;
println!("{}", doc.markdown);
println!("pages: {}", doc.pages.len());
println!("scanned: {}", doc.scanned);

extract_with adds limits, image options and the OCR engine. Errors are Malformed, Encrypted, TooLarge or Timeout — and a parser panic is caught and returned as an error instead of taking the caller down.

Benchmarks (exploratory, with the caveats kept)

Runs on public olmOCR-bench data, one machine (i7-10700, 15 GB RAM, GTX 1070). These are exploratory runs, not certified official scores; tools differ in features and conditions, and the full methodology — actual worker counts, caps, raw results — is in the repo's benchmarks/ report.

Born-digital pages (1,269 pages):

Tool	Completed	Pages/s	p50	Peak MB/worker	Score
yolowpdf	1269/1269	1,900	3.2 ms	24	0.592
pymupdf4llm	1213/1269	6.56	845.6 ms	600	0.489
pypdf	1213/1269	79.4	73.9 ms	186	0.385
MarkItDown	1204/1269	13.6	172.8 ms	1,171	0.373

On the pdf.js corpus of hard files (1,089 files), yolowpdf finished 830 files at 1,663 files/s; pymupdf4llm finished slightly more (841) at 1.31 files/s under different I/O conditions. That cell is in the README too — a fast parse is not the same as understanding a difficult page, and the README says so itself.

For scanned pages, yolowpdf-ocr (GPU, text only) completed all 134 olmOCR-bench scan pages at 2.245 pages/s; adding layout and formula models raised the score to 0.5428.

Install

On crates.io:

[dependencies]
yolowpdf = "0.1"

# Optional OCR:
yolowpdf-ocr = "0.1"

Rust 1.89+, edition 2024, MIT licensed. Models for OCR are downloaded by the caller (pinned revisions and SHA-256 values are listed in the source); nothing is bundled.

Repo: https://github.com/YOLOW92/yolowpdf

If you feed PDFs into LLM pipelines, I'd like to hear what breaks. Hard files especially welcome.

yolow64: a Base64 codec for Rust that outruns the incumbents — extracted from PageSift

ゴエンチョウ — Mon, 09 Nov 2026 04:48:37 +0000

While building PageSift (a Rust web-content engine for LLM pipelines), I kept hitting Base64 in places where throughput and allocation control actually matter: browser screenshots, MCP resources, proxy authentication. The codec that grew inside PageSift is now its own MIT-licensed library: yolow64.

Runtime-dispatched SIMD: AVX2 (with an assembly bulk decode loop on Linux x86-64), NEON on AArch64, scalar fallback; runtime detection is cached.
Zero runtime dependencies, std only, Rust 1.89+.
Slice / append / owned-output APIs, validation without decoding, streaming decode, and a one-shot data-URL builder.
Whitespace-tolerant mode ignores exactly space/tab/CR/LF in a single pass — built for MIME-style input.

Measured on an i7-10700 (same process, three full runs, medians of seven-round medians; full method, charts and raw CSVs in the repo):

Strict mode at 4 KiB–4 MiB: ~6.5–9× faster than the base64 crate, ~1.7–2.5× faster than base64-simd, encode and decode.
CRLF-tolerant decode: ~4× faster than base64-simd's forgiving_decode at the same sizes.
On the same machine it runs level with moego, the fastest Go implementation I measured.
Honest footnote: at ≤128 B inputs the three libraries trade wins — those differences are nanoseconds and noise-sensitive, and the repo's tables print them as-is. The whitespace modes also don't accept exactly the same inputs across libraries; the comparison covers valid padded CRLF inputs.

[dependencies]
yolow64 = "0.1"

Repo: https://github.com/YOLOW92/yolow64 — benchmarks are reproducible from the included harness; corrections welcome.

PageSift 1.2: ITNA, an access-barrier recovery engine — plus the 1.1 roundup

ゴエンチョウ — Sat, 07 Nov 2026 02:20:10 +0000

PageSift started as a Rust crawler chasing extreme performance. It has since grown into a web content engine — web scraping, document parsing, structured extraction, and an AI-agent surface — and 1.2 adds one more engine to it.

GitHub: https://github.com/YOLOW92/pagesift

1.2: ITNA, an access-barrier recovery engine

When an HTTP fetch gets blocked, it decides intelligently whether to escalate to a browser retry
TLS/HTTP2 browser fingerprinting over HTTP, HTTPS, and SOCKS5 proxies
A session identity pool: cookies, egress, and fingerprint managed together, isolated per user
Safe HTTP → browser handoff of session state
Tuned browser automation markers, with waiting for non-interactive JS challenges
A shared request budget that caps retries, timeouts, and resource use

ITNA is off by default and enabled on demand. It does not solve interactive CAPTCHAs, and it does not promise to bypass every anti-bot system.

Catching up on 1.1: documents and languages, massively expanded

I never got around to posting about the 1.1 series, so here it is in one go:

A long list of new file formats: legacy Office, ODF, WordPerfect, e-books, email, archives, databases, and more
373 source-code languages recognized, with function/class-level structure extracted for mainstream languages
New bindings for Go, Java, C#, Ruby, PHP, Lua, Swift, Zig and more, plus LangChain and LlamaIndex integrations

Measured on 1.1:

PDF → Markdown: 2.4 ms/page
HTML → Markdown: 0.24 ms/page
Main-content extraction: 1.0 ms/page
Office parsing: 91.7 files/sec, about 3.40× xberg

1.1.1–1.1.3: still digging into the Rust layer

I implemented Zstandard compression/decompression in pure Rust — FSE, Huffman, LZ matching, cache-aware hash chains, all written from scratch.

Where it stands

PageSift has passed 200 stars on GitHub, and it is still developed and maintained by one person.

The goal was never just a fast crawler. I want it to be genuinely high-performance, low-overhead, self-hostable AI data infrastructure.

Fully open source — search YOLOW92 / PageSift on GitHub. Tests, issues, and technical discussion are all welcome.

PageSift 1.0: the Rust web content engine is feature-complete

ゴエンチョウ — Wed, 04 Nov 2026 03:44:32 +0000

Fifteen days, twenty-five releases. PageSift went public as 0.20 on October 20; today it ships 1.0.

PageSift is a web content engine written in Rust. It turns URLs, PDFs and Office files into clean Markdown and structured data for AI agents and LLM pipelines — one self-hosted binary with a Firecrawl-compatible HTTP API, a Jina Reader-compatible endpoint (including /s/), an MCP server, and Python / Node.js / Rust SDKs.

1.0 is not a speed release — performance landed with the 0.9 architecture rebuild. 1.0 is feature completion.

What 1.0 adds

OCR, built from scratch. Scanned PDFs were PageSift's documented weak spot. 1.0 ships its own OCR pipeline — text detection, recognition, formula recognition (LaTeX out), layout analysis — running on a self-written inference runtime (safetensors executor; no ONNX Runtime) and a self-written CUDA GPU layer. Models are open weights, downloaded on first use, off by default. On the olmOCR-bench scanned set it scores 0.543 vs Docling (GPU) at 0.410.

Selector extraction with memory. Give it a JSON schema; when a site redesign breaks your selectors, it relocates elements using element fingerprints and a statistical posterior — and reports relocated, ambiguous, or missing instead of silently returning wrong data.

Change monitoring. A /v2/monitor endpoint family with block-level diffing that separates noise changes (prices, dates, counters) from substantive ones.

Browser actions, sessions, and an agent endpoint across the HTTP, MCP and Jina surfaces — plus WARC files as input, a hooks + [[rules]] system, and per-host secret injection.

The architecture in three ideas

Measured instead of fixed. The hundreds of hand-tuned timeouts, thresholds and concurrency limits are gone. A statistical kernel derives them from runtime observations — survival analysis (Kaplan–Meier) for timeouts, a Universal Scalability Law fit for global concurrency, and "posterior expected value − shadow price × cost" for render decisions. One global meta-parameter remains: δ = 0.05. With no observations, behaviour matches the old fixed values; an estimate takes over only when the difference is statistically significant.
Scheduling that follows the hardware. One resident CPU worker per core (no blocking thread pool), pinned to NUMA node sets rather than individual cores — because measurements said so: hard per-core pinning cost −33.9% throughput in the 0.70 experiments, node-set pinning gained +8.63%. The code comments cite the measurement behind each such decision.
Failures are reported, never dressed up as success. 0.91 was a release about exactly this: jobs whose results couldn't be saved, replicated, or delivered by webhook previously ended as completed. They now fail loudly, with an errors endpoint. 1.0 extends the rule to the API surface: options the engine cannot honor are rejected (unknown MCP arguments error; unimplementable Jina headers return UNSUPPORTED_OPTION), and the release notes carry a Known limits section that lists what isn't tested and what doesn't work — including security caveats.

Performance (re-measured 2026-11-04)

Workload	PageSift 1.0	Fastest rival	Gap
PDF → Markdown, olmOCR-bench 1,269 pages, single thread	2.4 ms/page	pypdf 50.0 ms/page	×20.8
HTML → Markdown, 600 documents	0.24 ms/page	Turndown 1.9 ms/page	×8.1
Main content, Firecrawl 597 pages, single thread p50	1.0 ms/page	crw 7.4 ms/page	×7.3
Office / e-books, 81 files	93.6 files/s	Kreuzberg 19.5	×4.8
Wide crawl, 200 sites × 100 pages, unthrottled	3,500–4,400 pages/s, 280–316 MB peak	Spider 2,000–2,700 pages/s, 5.5 GB	—

Honest footnotes, because they matter: competitor numbers are the recorded late-October runs, not re-run for this release; and the workloads where PageSift loses are listed in the same document ("Largest gaps") — e.g. a single-host replay looks slow mostly because of the default 1-second per-host politeness interval, which is policy, not engine speed. Full data and reproduction scripts live in the repo (docs/benchmarks.md, benchmarks/).

Battle-tested once, properly

In late October an external team integrated PageSift into their production system and immediately hit a real bug: the official Node binding required a newer glibc than their server had (2.32). 0.81.5 fixed it — build baseline lowered to glibc 2.28, plus a release gate that fails the release if any artifact exceeds it. A user found it; the project fixed it and made it impossible to reintroduce silently. That's the workflow 1.0 is built around.

Who it's for / not for

For: teams feeding RAG or agent pipelines who want a self-hosted Firecrawl/Jina alternative without per-page pricing, and anyone who needs web pages, PDFs and Office files in one pipeline.

Not for: people who need a hosted cloud API (PageSift is self-hosted; a hosted version is not part of 1.0), or bypassing hard anti-bot targets (robots.txt and politeness are respected; captchas are reported, not solved).

PageSift is a solo project. It exists because the crawler feeding my other project was too slow and not clean enough, so I wrote my own. 1.0 closes the feature list. What I want next is more production workloads breaking it in interesting ways.

Repo: https://github.com/YOLOW92/pagesift — AGPL-3.0, commercial licensing available. Issues and head-to-head benchmarks welcome, good or bad.

My Crawler Now Tells You What It Missed

ゴエンチョウ — Tue, 27 Oct 2026 14:25:18 +0000

My Crawler Now Tells You What It Missed

The worst failure mode of a data pipeline is not an error. It's silent junk: a 404 page filed as content, a recipe with the ingredients dropped, a forum thread quietly missing its replies — all returned as "success." If you feed LLMs for a living, you have eaten this. I have shipped this.

PageSift 0.80 is the release where I tried to eliminate that failure mode from my own engine. The theme is not speed this time (0.70 was speed). The theme is: results should say what they are, including what they failed to get.

1. Results that confess

Extraction is heuristic and always will be. The bug is pretending otherwise. In 0.80, when extraction likely kept too little, the result says so — with numbers:

If article extraction keeps only a small share of the page, you get an explicit incompleteness warning instead of a confident stub.
If a forum page's replies were left out, the warning quantifies it: how many characters sat in comment sections, how much was extracted, and how to ask for the whole page if you want it.
Video pages with no readable body don't return empty success anymore; they fall back to the page's embedded video metadata and label that they did.
Error-status pages carry an explicit http_error_status warning and the real status code in metadata. Server errors (and empty error pages) fail hard; nothing passes as silent success.

These labels exist because my own production audit embarrassed me into them: 95% of my old crawler's rendered output was being thrown away, and nobody had noticed, because every result claimed to be fine.

2. PDF: from text extraction to structure preservation

0.80 replaces the third-party PDF library with a file layer I wrote myself. Headings are the difference between a document and a wall of text, so that's what I measured — on a set of real-world PDFs:

Heading recall: 0.595 → 0.947
Heading precision: 0.400 → 0.871
Hierarchy consistency: 0.819 → 0.957
Phantom tables: 196 → 9 (all 143 real tables kept)
Memory: −25% at the same speed

3. Documents, all of them, all hand-written

Word, Excel, PowerPoint, EPUB and CSV are now supported, parsed by my own Rust code — including the ZIP and XML layers; the only borrowed piece is a decompression library. On a held-out set of files collected after the rules were frozen:

Structure retention: 0.988 (MarkitDown: 0.717, pandoc: 0.705)
81 files parsed in 657 ms total (MarkitDown: ~48.8 s)
Median peak memory per file: 14.2 MB (MarkitDown: 140.1 MB)

4. HTML parsing, closer to the spec

The parser now passes 89.3% of the official html5lib test suite (was 77.7%); parsing differences that actually change the extracted content dropped from 143 to 6. Parsing got faster while getting more correct, which was not the direction I expected.

5. Fuzzing: the inputs that used to kill it

A fuzz pass found inputs that hung the engine or blew up memory. Before → after:

720 KB of pathological text: 19.5 s → 24 ms
1.6 MB of adversarial HTML: 52 s → 0.19 s
One PDF: killed at >3 GB memory → 81 ms at 73 MB

Browsers orphaned when the process is killed are now reaped on the next start, too.

The rule I held myself to

This was the correctness release, and the standing rule was: correctness may not be bought with speed. Against 0.73, single-thread extraction is 3.7% faster, 16-thread 4.4% faster, PDF memory down 25%, per-request CPU flat. The rule held.

Still open, honestly

Steam's real age gate still slips past classification — it comes back flagged only as thin content, not recognized as a gate. It's been the standing exception for three releases now, and I'd rather list it here than let you find it. Deletion tracking for sync pipelines (gone pages as a first-class signal) isn't built yet either; a reader asked for exactly that, and it's on the list.

GitHub: https://github.com/YOLOW92/pagesift — the 0.80 release notes have the full tables. Next on the 0.8x line: the hosted API. If your pipeline has a silent-junk horror story, I want to hear it in the comments.

I Profiled My Production Crawler. 95% of Its Output Was Being Thrown Away.

ゴエンチョウ — Sun, 25 Oct 2026 18:46:57 +0000

I run AiSeek, an information platform where every published claim has to trace back to fetched evidence — a source page, a filing, a dataset. That makes the fetch layer the supply chain of the whole product: if it is slow, wasteful, or silently wrong, everything downstream inherits it.

Until October 21, that layer was Crawl4AI — a capable, popular Python framework — driving Chromium for the pages that needed rendering. I had a feeling it was heavy. Instead of trusting the feeling, I audited it: three days of production jobs (8,563 of them, October 18–20), the production database, and local reproductions run under the same limits as the production box (6 cores, 10 GB). Every number below is production data unless I say otherwise.

This is what the audit found, what I built because of it, and the part people usually leave out of launch posts.

What the crawler actually did all day

On a normal day, AiSeek fetched roughly 2,000 articles over plain HTTP and sent 600–900 to the browser — about 28% of article fetches ended up rendered. The two paths lived in different latency worlds:

Path	Jobs	p50	p95	p99
HTTP only	2,879	1.6 s	5.8 s	10.9 s
Rendered (Crawl4AI)	1,249	7.0 s	24.4 s	40.2 s

Rendered pages were mostly the ones that needed it least, as it turned out. But first, the bigger surprise.

Finding 1: I was using a framework as a browser driver

Of 174 successful renders in my local reproduction, 166 used my own trafilatura extraction for the final text. Only 8 used the Markdown Crawl4AI generated. Its Markdown was computed on every single page — and thrown away 95% of the time.

What I actually consumed from this large framework was a minimal surface:

the rendered HTML,
the final URL after redirects,
a status code,
the link list,
a title fallback.

And the irony compounds: the heaviest single parsing step in the pipeline wasn't Crawl4AI's either. My own trafilatura call cost a median of 82 ms per page; Crawl4AI's scraping plus Markdown generation together cost about 40 ms. The library I suspected of being slow was, by itself, mostly innocent.

I wasn't using Crawl4AI. I was renting a browser with a framework attached — and paying the framework's tax on every page.

Finding 2: 55% of render time was pure waste

The decision rule for rendering was one line of logic: if the HTTP fetch produced fewer than 1,200 characters of text, render it in a browser. That rule cannot tell a JavaScript wall from a short article — and a lot of AiSeek's sources are short news briefs, many of them in Chinese and Japanese, where 800 characters is a complete story.

Replaying the production decision logic locally: in 59 cases where HTTP had already retrieved a usable page but the rule sent it to the browser anyway, 37 renders (63%) came back with the same amount of text as HTTP — NHK at 849 characters before rendering, 849 after. Those wasted renders consumed 85 seconds of 154.6 total: 55% of all rendering time, spent re-fetching pages we already had.

This was the most valuable finding of the audit, and it had nothing to do with languages or engines: the biggest speedup available was a better decision about when to render. Rewriting that rule was worth more than any engine swap.

Finding 3: the real costs were Chromium, and a shared event loop

What actually consumed the machine:

Chromium itself. Each concurrent rendered page costs roughly 0.7 CPU cores and a couple of hundred MB, plus a ~1.7 s cold start. No driver language changes that arithmetic.
A single event loop doing synchronous work. Crawl4AI runs its HTML cleaning and Markdown conversion synchronously on the Python event loop. With 4 concurrent pages, my probe measured loop stalls at p99 of 154–172 ms, worst case 2.2 seconds — during which everything else in that process waited, including plain HTTP fetches and the GLiNER and MiniCheck models that shared the process and its 10 GB container. An old code comment of mine, from October 9, reads: "plain fetches took 8 s at the median while pages rendered." Now I knew why.
A low concurrency ceiling. Sweeping the page limit locally on 6 cores: 4 pages gave 1.87 req/s, 8 pages topped out at 2.19 req/s, and 16 pages went backwards to 1.72 req/s with p50 latency ballooning to 18 s. Past a point, more pages just means a longer queue.
Weight. The service image was 6.64 GB. The fetch path alone pulled 115 packages, about 1 GB — including several libraries we never called.

Finding 4: some failures were never the crawler's fault

About 25% of fetches failed in production. I took 70 URLs from the failed group and fetched them simultaneously from my home connection and from the production server. From home, 39 of 70 returned full text. From production: zero. The rest were refused (403/451) or rate-limited (429) — entire publisher groups block our datacenter exit, and some of it is geography, not IP reputation (451 is "unavailable for legal reasons"; the production box sits in Türkiye).

No engine fixes that, and per AiSeek's rules I don't route around refusals: honest user agent, robots respected, a refusal is final. I include this because crawler benchmarks never mention it — in production, a quarter of your failures may have nothing to do with your stack.

Bugs I promised not to copy

The audit also catalogued defects I did not want the replacement to inherit:

Status codes from the first redirect hop. 136 of 174 successful renders reported the 301/302/308 they started with — so a "301 → 404" could be stored as a success.
An SSRF route guard that stacked per page. One more handler on the shared browser context for every page opened — up to 200 by recycle time.
DNS failure reported as an SSRF refusal. Same 400 for "this host is forbidden" and "this host temporarily doesn't resolve" — two situations a caller must handle oppositely.
Network errors collapsing into a bare 500, indistinguishable from a bug in our own code.

The replacement: build the minimal surface, in Rust

So I built PageSift to the audit, not to a feature checklist:

Only the minimal surface. Rendered HTML, final URL, final status code plus the redirect chain, links, and Rust-side extraction — no Markdown generated just to be discarded, no 115-package tax.
A separate process. Its own CPU and memory limits, restartable without touching the models. Nothing synchronous sits on anyone else's event loop.
A smarter render decision first, engine second. HTTP first; rendering is spent only where it buys something.
A browser chain instead of a browser. Lightpanda renders first; Chrome's headless shell takes the pages Lightpanda can't. Both run sandboxed — never --no-sandbox — which on our host meant writing AppArmor and seccomp profiles.
Honest failures. A block is reported as a block, never replaced with invented content. For an evidence platform, a declared gap beats a fabricated page.
The same API contract. The worker kept calling POST /v1/fetch and validating the same response schema. The swap touched one sidecar, nothing else.

On October 21 the sidecar went in, and Crawl4AI left the code and the image the same day. My deploy note from that day records the decision in parentheses: (owner: no week of observation). The audit was the observation.

The part launch posts leave out: production still runs v0.01

Here is the confession. AiSeek's production builds PageSift from a pinned commit — the first working build, call it 0.01 — and it still runs that build today.

Since then, the open-source project has sprinted: 0.50 crawled the Tranco Top 100k, 0.61 followed, and this week 0.70 landed with a rewritten NUMA-aware data plane (about 950k pages/second against a local test origin on a 192-vCPU box), a three-engine adaptive chain, and a memory envelope that squeezes the full pipeline under 888 MiB. Production has none of it. Not the data plane, not the third engine, not six versions of fixes.

Upgrading is deliberately boring: change one pinned commit, update two engine checksums, validate on the server, then push. It hasn't happened, for the same reason the audit happened. At ~2,800 fetches a day, throughput was never AiSeek's binding constraint — the audit showed the pain lived in a judgement rule and a shared process, and 0.01 already fixed both. What production buys with its stillness is that every claim on the site traces to evidence fetched by a binary whose behavior I know exactly.

When the upgrade does happen, it will be audited the same way this replacement was: production numbers first, announcement second. That will be Part 3.

Takeaway, if you run anything like this: profile your crawler before you replace it. You may find you are paying for a framework and using a browser — and that your fastest available optimization is a one-line decision rule, not a new engine.

PageSift is open source (AGPL-3.0, commercial licenses available): github.com/YOLOW92/pagesift. The product that keeps it honest is aiseek.dev.

I crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAM

ゴエンチョウ — Fri, 23 Oct 2026 13:34:32 +0000

TL;DR: I built PageSift, a single-binary web scraping engine in Rust. It's Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0. On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM-killed. Benchmarks, raw data, and reproduction scripts are all in the repo.

Why I built it

I run a small AI product, and web scraping was the part that kept hurting:

Self-hosted Firecrawl spins up 7 services by default and is AGPL.
Crawl4AI is Python + Playwright + Node + Chromium — an 847 MB venv before you even start.
Worst of all, captcha and interstitial pages kept landing in my database as if they were content.

So PageSift is one binary. No Redis, no database, no message queue. And when a site blocks it, you get an honest ACCESS_BARRIER error instead of a fake page.

The 100k live test

Setup: AMD EPYC server (32 cores, 128 GB RAM), Tranco Top 100k live domains, identical conditions for both engines — 768 async concurrency, 3.5 s connect / 10 s total timeout, Blake3 dedup, streaming Markdown extraction.

Metric	PageSift 0.50	Spider 2.53.9
Peak RSS	188.4 MB (flat)	8,978.9 MB
Valid Markdown docs	60,659	59,803
Extraction conversion	96.38%	82.82% (17.2% empty shells)
Memory behavior	zero leaks	leaked to 14 GB, OOM-killed at the 12 GB cap

The full dataset and scripts: bench/results/final/BENCHMARK_100K_PAGESIFT_VS_SPIDER.md

Extraction quality and speed

Main-content extraction: 0.85 ms/page (p50), F1 0.716 on Firecrawl's labelled dataset — crw 0.711, Crawl4AI 0.670. On a 545-page calibrated WebMainBench subset it beats Essence on average, and still loses on 117 pages. Per-page scores are published; I'd rather show the losses than cherry-pick.
PDF → Markdown: 71.0 on olmOCR-bench, formulas recovered as LaTeX at 44.3% (pymupdf4llm: 0%), at 3.4 ms/page vs 320 ms.
RAG: the BM25 query filter answers in 1.35 ms/query, keeping 93% of the answer text in ~1,150 words where Crawl4AI's filter keeps 67% in 13,500 words.

Quick start

docker run -d -p 3002:3002 \
  -e PAGESIFT_API_KEYS=change-me ghcr.io/yolow92/pagesift:0.50.0

# or
cargo install pagesift
pagesift serve   # Firecrawl-compatible API on :3002

import pagesift

doc = pagesift.scrape("https://example.com")
print(doc.markdown)

There's also a native Node package (@pagesift/node), an MCP server for agents, and JS rendering that only fires when a page actually needs it (Lightpanda → moli → Chrome, chosen per host).

How it's fast

A zero-copy HTML parser: text and attribute values borrow from the source string; only entity-containing fragments get decoded.
A Readability reimplementation that scores paragraphs in a single pass over precomputed text statistics — no DOM mutation, no repeated length calculations.
Hand-written scanners instead of regexes in the hot paths (class/id word classification is a compile-time match with a stack buffer).
The workspace is forbid(unsafe_code) — with exactly one exception: a single mallopt call that pins glibc's mmap threshold so page-sized buffers return to the OS instead of bloating RSS. That one call cut peak RSS in local crawl tests from 188 MB to ~75 MB.

It plays by the rules

PageSift respects robots.txt and Crawl-delay, rate-limits per host, uses an honest User-Agent, and does no anti-bot bypassing: no TLS fingerprinting, no captcha solving, no proxy rotation. PRs adding stealth features are not accepted.

Honest limitations

17 capability categories are still marked partial in my own inventory — persistent browser interaction and autonomous agent workflows are explicitly out of scope.
The license is AGPL-3.0 + a commercial option. Fully open for self-hosting; closed-source products need the commercial license.

Repo: https://github.com/YOLOW92/pagesift — docs in English, 中文，日本語 and Español.

If you run Firecrawl or Crawl4AI in production, I'd genuinely like to know: what breaks first for you?