There's a failure mode in agent tooling that never throws anything and I lost an afternoon to it before I understood what I was looking at. The fetch returns 200 and the body isn't empty and the extractor finds text in that body and turns the text into clean Markdown. Then the model reads my clean Markdown and writes me a fluent summary that happens to be entirely wrong.
The page was never the article. The page was the wall that says Checking your browser before you continue.
Every layer did its job, and no layer had the job of asking whether the page that arrived was the page I had asked for.
That bug is why I spent the last stretch of my free time writing Svipall, a local-first web layer for AI agents, in Rust. Below are the four things it does about it, with what I measured for each, including the measurements that didn't come out in my favour.
1. Nothing in the stack separates "the call worked" from "the page arrived"
A retry won't help here because a retry is the same request, and escalating to a headless browser won't help either if nothing tells you that escalating is the thing this particular page needs. The only fix I found is that whatever fetched the page has to say what it thinks it got back, and that verdict has to travel with the page itself.
So every fetch returns one, and a blocked page carries the evidence that produced it:
{
"status": 200,
"blocked_reason": "vendor challenge",
"wall_kind": "vendor",
"wall_vendor": "kpsdk.io",
"wall_evidence": "header x-kpsdk-ct",
"note": "Interstitial detected. A browser tier can run its script; a bare HTTP request cannot."
}
Twelve wall kinds, each one naming the move it implies. The field I care most about is wall_evidence because it says why the verdict came out this way and gives you something to disagree with. If you can't audit the verdict you're back to trusting the verdict blindly and trusting it blindly is where my whole problem started.
The honest caveat, which lives in the README too: the classification is heuristic, and a clean verdict doesn't prove that the records you wanted arrived intact. What it does is move the failure from silent to visible, and that's the whole claim I'm making for it.
When a page doesn't arrive, escalating is the obvious move, but escalating on every fetch is how you turn a 200 ms fetch into a six-second fetch, forever:
http emulated Chrome TLS + HTTP/2, no browser ~200 ms
│ most pages stop here
▼
browser headless Chromium, runs the page's JS
│
▼
stealth automation surfaces patched to match the identity
│
▼
real visible-but-offscreen browser, persistent profile,
│ pointer along curves, wheel scrolling
▼
warm bounded wait, captcha strategy loop runs inside it
I measured it from a single residential address with no proxy and took the median of three runs with the range: 26/31 on a public target list I didn't write, 7/12 on my own list of hard targets, 3/8 against four named commercial vendors.
Three out of eight. That row stays in my README because a comparison table where every cell is green tells you more about the author than it does about the software. My raw logs are committed alongside it and they include the rounds of work that improved nothing and the rounds where a number went down.
On doing this responsibly
This part isn't boilerplate, so I haven't put it at the bottom. Svipall evades anti-bot fingerprinting and that is all it does. It does not defeat authentication, it does not open paywalled content you haven't paid for and there is no cloud solver farm behind it. I wrote it for research and the job I wrote it for is reading pages a browser would happily show you, from a tool that isn't a browser, without the fetch failing silently.
Being able to reach a system doesn't mean you're allowed to. You pick the addresses and you pick how hard to push and that puts the terms of service and the copyright and the data-protection law on you. The repo ships a DISCLAIMER that says so at length. robots.txt handling is warn by default and obey on request. The crawler stops at whatever budget you set it: pages, tokens, traffic. Point it at a hundred thousand pages because you can and you're the problem, not the wall.
2. The agent shouldn't have to rediscover the route every time
I underestimated this one badly while I was building it. An agent that finds the right way into a site and then forgets that way has learned nothing at all, so tomorrow the agent climbs the same ladder again: same seconds burned, same reputation spent with the same site, same answer at the end of it.
So the route is remembered per domain, and the route survives the session. Two supporting observations can promote a working tier, and the next visit then starts there instead of at the bottom. Cooldowns are stored the same way, and so is per-exit health, and so are the walls a site is known to serve. svipall status prints the lot of it.
The same memory runs deeper than the tier. Captcha strategies are ordered by what actually worked on that route, so the loop tries the winner first and a strategy that declines costs no attempt. A schema you name is fingerprinted per domain, and when the site gets redesigned the selector is relocated by similarity and comes back marked healed. Cookies from a web_login live in a profile you can export. web_notes is key-value memory that outlives the session, for whatever the agent worked out that the tool doesn't model.
The effect on an agent loop is fewer steps, and fewer steps is the same thing as fewer tokens. In a paired run, on the 109 pairs where both arms returned useful content, the automatic ladder spent 368 s against native browsing's 735 s. That difference isn't only fast refusals.
3. The tokens
I fetched one page three ways, just now, on this machine. English Wikipedia, Rust (programming language):
| What I asked for | Characters | Share of the raw HTML |
|---|---|---|
extraction: "html", the raw markup |
1,048,788 | 100% |
| default, extracted Markdown | 106,266 | 10.1% |
query: "borrow checker ownership" |
48,486 | 4.6% |
Roughly 262k → 27k → 12k tokens at four characters per token. The character counts are measured; the token figures are the usual estimate and not a tokenizer run.
Ninety per cent of that page was navigation, reference markup and the apparatus Wikipedia wraps around its prose. Every character of that ninety per cent was billable, and every character of it sat between the model and the part that mattered.
Boilerplate removal has a number attached to it: median ROUGE-LSum F1 of 0.920 over the 3,975 gradable pages of the SIGIR-23 gold standard, against 0.732 with the boilerplate removal switched off. And then the row I'd rather not print, which is that readability scores 0.963 on that same corpus and trafilatura 0.958. They beat me. Both of them are excellent and both of them are in my comparison table for exactly that reason.
Where I spend the difference is query=. It keeps only the blocks that lexically match your words using BM25. No embedding model and no API call, so nothing here bills you twice for reading one page. How much it saves depends on the query and not on the page. On two long articles the query "robots.txt" left 6% and 13% of the text. The query "history of scraping" left 88% of it. Name the fact you want, not the topic.
Then there are the duplicates. I find near-duplicates with SimHash and a banded index, and web_fetch_many reports corroboration, which is how many distinct documents your set actually is, marking each copy with same_text_as. Eight sources that are one press release reprinted eight times aren't eight sources, and an agent has no way to know that unless something says so.
Then there's the plumbing. out_file writes to disk and hands back a path, measured at a 418-character response in place of the 34,746-character page it wrote, and schema: "auto" reads a listing's own repeated structure and returns named columns from one parse, with no model involved.
One rule costs me recall and stays in anyway: hidden text never reaches the model, so anything behind display:none, left:-9999px or aria-hidden gets dropped before extraction. A paragraph parked off-screen reads to an agent exactly like the article does, and that paragraph is the entire prompt-injection surface. There are 123 phrases a human annotator marked as real content sitting behind that rule.
4. Nobody should have to count how many times they've hit a site
A hosted scraper rotates through a pool of addresses, so the question of how much one address has asked a host never comes up. Run the thing locally and your home connection is the entire pool. An agent in a loop has no idea how many calls it has already made against a domain this afternoon, and neither do you, and the site absolutely does.
I learned this from my own benchmark. I ran the target lists against one residential address several times in a day, and a cell that had been passing stopped passing. Nothing in the tool had changed. I had simply spent the address.
So there's a ledger now. Every visit costs what its tier costs, doubled when the page comes back walled:
http |
browser |
stealth |
real |
warm |
|---|---|---|---|---|
| 1 | 3 | 4 | 8 | 12 |
A count of visits would be the wrong unit. An HTTP GET and a headful browser sitting through a twenty-second challenge are not the same event to a host that scores addresses. The budget is 250 by default and it decays with a six-hour half-life instead of resetting at midnight, which makes it a rate and not a daily allowance: a steady spend settles at about 29 points an hour.
Under 70% of the budget nothing happens. Between 70% and 100% the pacer stretches the gap between requests, up to four times. Over it, the fetch is declined before a packet leaves the machine, with blocked_reason: "address_budget", the number of seconds until it isn't, and the way out. It's a labelled answer, never a silence.
The part I'd defend hardest: your own address is a key in that ledger like any other, which is the whole point, because the case that broke my benchmark had no proxy involved at all. Pacing is keyed by (domain, exit) too, so the gap comes from the host's own latency and refusals instead of a global constant: 100 ms to 2 s on http, 400 ms to 5 s on the browser tiers.
None of this makes a site trust you. It stops your agent from spending goodwill it doesn't know it has.
What it is, and what else already exists
It's a single Rust binary that speaks MCP (29 tools) for Claude Code, Claude Desktop and Cursor. It works as a CLI and it turns into a local REST API under svipall serve so any language can drive it.
svipall fetch https://docs.example/api --query "rate limits"
svipall fetch https://shop.example/listing --schema auto # rows from a page you've never seen
svipall crawl https://docs.example/ --pages 50 --out pages.csv
svipall search "rust async runtime" --engine all # no API key
There's nothing to install underneath it and nothing to sign up for: no Node, no Python, no API key, no telemetry. Captchas get attempted locally across 15 widget families and 11 answer modalities, and when none of that works the page parks in a dashboard at localhost:8787/human so a person can answer it once. There's no paid solver in the loop and no path where your pages reach my server, since I don't run one.
I'm not going to pretend the field is empty. Firecrawl is the hosted API you can call from a lambda. If you live in Python, Crawl4AI is mature and ships a Docker server. Scrapling does adaptive parsing with session and proxy control. For plain browser automation over MCP the reference answer is still Playwright MCP. All four are in the repo's comparison table, and if you want a URL you can hit from a serverless function, use a hosted service.
What Svipall adds is a combination I couldn't find anywhere in one place. The fetch tells you whether the page arrived, the route to that page is something the agent learns once, and the tokens get cut during extraction instead of after it. All of that runs on hardware you own. That comparison in the repo describes documented scope on a stated date. It is not a head-to-head benchmark, so validate your own target pages before you switch anything.
My receipts live in the proof page, each one with the command that reproduces it: 1,216 tests, 160/160 automation-tell probes clean over five browser passes, 8/8 identity coherence checks, and a delivery snapshot of 348/459 calls (75.82%) over 48 URLs, unattended.
curl -fsSL https://raw.githubusercontent.com/ilien-dev/svipall/main/install.sh | sh
AGPL-3.0, extraction engine MIT/Apache-2.0 so anything can depend on it.
svipall.ilien.dev · source on GitHub · what it can't do
I built this, and I'd like one answer in the comments: how does your agent currently find out that a fetch returned a wall instead of a page? Every answer I've gotten so far is some version of "it doesn't, I noticed later", and if that's the state of the art, the silent 200 is a bigger hole in agent tooling than the scraping part anyone argues about.
Top comments (0)