I crawled the Tranco Top 100k with a Rust engine that never passed 188 MB of RAM

ゴエンチョウ — Fri, 23 Oct 2026 13:34:32 +0000

TL;DR: I built PageSift, a single-binary web scraping engine in Rust. It's Firecrawl v1/v2 API-compatible, turns pages into clean Markdown for LLM pipelines, and just hit v0.50.0. On a live crawl of the Tranco Top 100,000 domains, its peak memory stayed flat at 188.4 MB while Spider (also Rust) ballooned to ~9.0 GB and got OOM-killed. Benchmarks, raw data, and reproduction scripts are all in the repo.

Why I built it

I run a small AI product, and web scraping was the part that kept hurting:

Self-hosted Firecrawl spins up 7 services by default and is AGPL.
Crawl4AI is Python + Playwright + Node + Chromium — an 847 MB venv before you even start.
Worst of all, captcha and interstitial pages kept landing in my database as if they were content.

So PageSift is one binary. No Redis, no database, no message queue. And when a site blocks it, you get an honest ACCESS_BARRIER error instead of a fake page.

The 100k live test

Setup: AMD EPYC server (32 cores, 128 GB RAM), Tranco Top 100k live domains, identical conditions for both engines — 768 async concurrency, 3.5 s connect / 10 s total timeout, Blake3 dedup, streaming Markdown extraction.

Metric	PageSift 0.50	Spider 2.53.9
Peak RSS	188.4 MB (flat)	8,978.9 MB
Valid Markdown docs	60,659	59,803
Extraction conversion	96.38%	82.82% (17.2% empty shells)
Memory behavior	zero leaks	leaked to 14 GB, OOM-killed at the 12 GB cap

The full dataset and scripts: bench/results/final/BENCHMARK_100K_PAGESIFT_VS_SPIDER.md

Extraction quality and speed

Main-content extraction: 0.85 ms/page (p50), F1 0.716 on Firecrawl's labelled dataset — crw 0.711, Crawl4AI 0.670. On a 545-page calibrated WebMainBench subset it beats Essence on average, and still loses on 117 pages. Per-page scores are published; I'd rather show the losses than cherry-pick.
PDF → Markdown: 71.0 on olmOCR-bench, formulas recovered as LaTeX at 44.3% (pymupdf4llm: 0%), at 3.4 ms/page vs 320 ms.
RAG: the BM25 query filter answers in 1.35 ms/query, keeping 93% of the answer text in ~1,150 words where Crawl4AI's filter keeps 67% in 13,500 words.

Quick start

docker run -d -p 3002:3002 \
  -e PAGESIFT_API_KEYS=change-me ghcr.io/yolow92/pagesift:0.50.0

# or
cargo install pagesift
pagesift serve   # Firecrawl-compatible API on :3002

import pagesift

doc = pagesift.scrape("https://example.com")
print(doc.markdown)

There's also a native Node package (@pagesift/node), an MCP server for agents, and JS rendering that only fires when a page actually needs it (Lightpanda → moli → Chrome, chosen per host).

How it's fast

A zero-copy HTML parser: text and attribute values borrow from the source string; only entity-containing fragments get decoded.
A Readability reimplementation that scores paragraphs in a single pass over precomputed text statistics — no DOM mutation, no repeated length calculations.
Hand-written scanners instead of regexes in the hot paths (class/id word classification is a compile-time match with a stack buffer).
The workspace is forbid(unsafe_code) — with exactly one exception: a single mallopt call that pins glibc's mmap threshold so page-sized buffers return to the OS instead of bloating RSS. That one call cut peak RSS in local crawl tests from 188 MB to ~75 MB.

It plays by the rules

PageSift respects robots.txt and Crawl-delay, rate-limits per host, uses an honest User-Agent, and does no anti-bot bypassing: no TLS fingerprinting, no captcha solving, no proxy rotation. PRs adding stealth features are not accepted.

Honest limitations

17 capability categories are still marked partial in my own inventory — persistent browser interaction and autonomous agent workflows are explicitly out of scope.
The license is AGPL-3.0 + a commercial option. Fully open for self-hosting; closed-source products need the commercial license.

Repo: https://github.com/YOLOW92/pagesift — docs in English, 中文，日本語 and Español.

If you run Firecrawl or Crawl4AI in production, I'd genuinely like to know: what breaks first for you?

DEV Community: ゴエンチョウ