I run AiSeek, an information platform where every published claim has to trace back to fetched evidence — a source page, a filing, a dataset. That makes the fetch layer the supply chain of the whole product: if it is slow, wasteful, or silently wrong, everything downstream inherits it.
Until October 21, that layer was Crawl4AI — a capable, popular Python framework — driving Chromium for the pages that needed rendering. I had a feeling it was heavy. Instead of trusting the feeling, I audited it: three days of production jobs (8,563 of them, October 18–20), the production database, and local reproductions run under the same limits as the production box (6 cores, 10 GB). Every number below is production data unless I say otherwise.
This is what the audit found, what I built because of it, and the part people usually leave out of launch posts.
What the crawler actually did all day
On a normal day, AiSeek fetched roughly 2,000 articles over plain HTTP and sent 600–900 to the browser — about 28% of article fetches ended up rendered. The two paths lived in different latency worlds:
| Path | Jobs | p50 | p95 | p99 |
|---|---|---|---|---|
| HTTP only | 2,879 | 1.6 s | 5.8 s | 10.9 s |
| Rendered (Crawl4AI) | 1,249 | 7.0 s | 24.4 s | 40.2 s |
Rendered pages were mostly the ones that needed it least, as it turned out. But first, the bigger surprise.
Finding 1: I was using a framework as a browser driver
Of 174 successful renders in my local reproduction, 166 used my own trafilatura extraction for the final text. Only 8 used the Markdown Crawl4AI generated. Its Markdown was computed on every single page — and thrown away 95% of the time.
What I actually consumed from this large framework was a minimal surface:
- the rendered HTML,
- the final URL after redirects,
- a status code,
- the link list,
- a title fallback.
And the irony compounds: the heaviest single parsing step in the pipeline wasn't Crawl4AI's either. My own trafilatura call cost a median of 82 ms per page; Crawl4AI's scraping plus Markdown generation together cost about 40 ms. The library I suspected of being slow was, by itself, mostly innocent.
I wasn't using Crawl4AI. I was renting a browser with a framework attached — and paying the framework's tax on every page.
Finding 2: 55% of render time was pure waste
The decision rule for rendering was one line of logic: if the HTTP fetch produced fewer than 1,200 characters of text, render it in a browser. That rule cannot tell a JavaScript wall from a short article — and a lot of AiSeek's sources are short news briefs, many of them in Chinese and Japanese, where 800 characters is a complete story.
Replaying the production decision logic locally: in 59 cases where HTTP had already retrieved a usable page but the rule sent it to the browser anyway, 37 renders (63%) came back with the same amount of text as HTTP — NHK at 849 characters before rendering, 849 after. Those wasted renders consumed 85 seconds of 154.6 total: 55% of all rendering time, spent re-fetching pages we already had.
This was the most valuable finding of the audit, and it had nothing to do with languages or engines: the biggest speedup available was a better decision about when to render. Rewriting that rule was worth more than any engine swap.
Finding 3: the real costs were Chromium, and a shared event loop
What actually consumed the machine:
- Chromium itself. Each concurrent rendered page costs roughly 0.7 CPU cores and a couple of hundred MB, plus a ~1.7 s cold start. No driver language changes that arithmetic.
- A single event loop doing synchronous work. Crawl4AI runs its HTML cleaning and Markdown conversion synchronously on the Python event loop. With 4 concurrent pages, my probe measured loop stalls at p99 of 154–172 ms, worst case 2.2 seconds — during which everything else in that process waited, including plain HTTP fetches and the GLiNER and MiniCheck models that shared the process and its 10 GB container. An old code comment of mine, from October 9, reads: "plain fetches took 8 s at the median while pages rendered." Now I knew why.
- A low concurrency ceiling. Sweeping the page limit locally on 6 cores: 4 pages gave 1.87 req/s, 8 pages topped out at 2.19 req/s, and 16 pages went backwards to 1.72 req/s with p50 latency ballooning to 18 s. Past a point, more pages just means a longer queue.
- Weight. The service image was 6.64 GB. The fetch path alone pulled 115 packages, about 1 GB — including several libraries we never called.
Finding 4: some failures were never the crawler's fault
About 25% of fetches failed in production. I took 70 URLs from the failed group and fetched them simultaneously from my home connection and from the production server. From home, 39 of 70 returned full text. From production: zero. The rest were refused (403/451) or rate-limited (429) — entire publisher groups block our datacenter exit, and some of it is geography, not IP reputation (451 is "unavailable for legal reasons"; the production box sits in Türkiye).
No engine fixes that, and per AiSeek's rules I don't route around refusals: honest user agent, robots respected, a refusal is final. I include this because crawler benchmarks never mention it — in production, a quarter of your failures may have nothing to do with your stack.
Bugs I promised not to copy
The audit also catalogued defects I did not want the replacement to inherit:
- Status codes from the first redirect hop. 136 of 174 successful renders reported the 301/302/308 they started with — so a "301 → 404" could be stored as a success.
- An SSRF route guard that stacked per page. One more handler on the shared browser context for every page opened — up to 200 by recycle time.
- DNS failure reported as an SSRF refusal. Same 400 for "this host is forbidden" and "this host temporarily doesn't resolve" — two situations a caller must handle oppositely.
- Network errors collapsing into a bare 500, indistinguishable from a bug in our own code.
The replacement: build the minimal surface, in Rust
So I built PageSift to the audit, not to a feature checklist:
- Only the minimal surface. Rendered HTML, final URL, final status code plus the redirect chain, links, and Rust-side extraction — no Markdown generated just to be discarded, no 115-package tax.
- A separate process. Its own CPU and memory limits, restartable without touching the models. Nothing synchronous sits on anyone else's event loop.
- A smarter render decision first, engine second. HTTP first; rendering is spent only where it buys something.
-
A browser chain instead of a browser. Lightpanda renders first; Chrome's headless shell takes the pages Lightpanda can't. Both run sandboxed — never
--no-sandbox— which on our host meant writing AppArmor and seccomp profiles. - Honest failures. A block is reported as a block, never replaced with invented content. For an evidence platform, a declared gap beats a fabricated page.
-
The same API contract. The worker kept calling
POST /v1/fetchand validating the same response schema. The swap touched one sidecar, nothing else.
On October 21 the sidecar went in, and Crawl4AI left the code and the image the same day. My deploy note from that day records the decision in parentheses: (owner: no week of observation). The audit was the observation.
The part launch posts leave out: production still runs v0.01
Here is the confession. AiSeek's production builds PageSift from a pinned commit — the first working build, call it 0.01 — and it still runs that build today.
Since then, the open-source project has sprinted: 0.50 crawled the Tranco Top 100k, 0.61 followed, and this week 0.70 landed with a rewritten NUMA-aware data plane (about 950k pages/second against a local test origin on a 192-vCPU box), a three-engine adaptive chain, and a memory envelope that squeezes the full pipeline under 888 MiB. Production has none of it. Not the data plane, not the third engine, not six versions of fixes.
Upgrading is deliberately boring: change one pinned commit, update two engine checksums, validate on the server, then push. It hasn't happened, for the same reason the audit happened. At ~2,800 fetches a day, throughput was never AiSeek's binding constraint — the audit showed the pain lived in a judgement rule and a shared process, and 0.01 already fixed both. What production buys with its stillness is that every claim on the site traces to evidence fetched by a binary whose behavior I know exactly.
When the upgrade does happen, it will be audited the same way this replacement was: production numbers first, announcement second. That will be Part 3.
Takeaway, if you run anything like this: profile your crawler before you replace it. You may find you are paying for a framework and using a browser — and that your fastest available optimization is a one-line decision rule, not a new engine.
PageSift is open source (AGPL-3.0, commercial licenses available): github.com/YOLOW92/pagesift. The product that keeps it honest is aiseek.dev.
Top comments (4)
The finding that the render decision mattered more than the engine swap is the most useful part here. After moving to the smarter HTTP first rule, what signal do you use now to decide that a page is worth rendering at all?
Good question — the signal comes from the HTTP response itself, not from a per-site list. The rule in the current release (0.72.8):
So rendering became the exception that has to justify itself per page. That inversion mattered more than which engine runs on the pages that do get rendered.
Using embedded data before paying for a render is the most practical step here, since JSON-LD and video metadata often carry the needed facts without a browser. I would also keep the explicit failure labels, because a clear thin content result is far more useful than silent junk.
You're putting your finger on the exact gap. Honest status today: JSON-LD is passed through, and video pages fall back to the VideoObject when the body comes back essentially empty — but embedded data doesn't yet act as a first-class source that can pre-empt a render. Treating it as one is a direction we're actively scoping, for exactly your reason: if the facts are already in the page, paying for a browser is waste.
On labels: agreed, and we're doubling down — the next release extends the same idea from "too thin" to "probably incomplete" (e.g. article extraction that kept only a small share of the page now says so). Silent junk is the one failure mode we refuse to ship.