A web crawler for AI agents with no LLM in the crawl loop

Rayen Mlayeh — Wed, 14 Oct 2026 13:25:25 +0000

When an AI agent needs an answer that lives a few clicks deep on a website, there are two common ways to get it.

Dump everything. A crawler fetches many pages and the agent's LLM reads all of them. It is slow, it costs a lot of tokens, and the answer drowns in navigation menus.
Let the LLM choose every link. This works, but each step is an LLM call. On my benchmark it cost about $0.19 per question and took a minute.

I wanted the accuracy of the second option at a price closer to the first. So I built jevcrawl, an open-source (Apache-2.0) MCP server and CLI where a small scoring model drives the crawl and no LLM sits inside the crawl loop.

What it does

You give it a question and a start URL. It fetches as few pages as it can and returns only the passages that answer the question, copied verbatim with their source URLs.

uvx jevcrawl crawl https://docs.python.org/3/ \
  --goal "What does the walrus operator := do and which Python version added it?"

On that example it fetched 6 pages and cut about 80,000 tokens of page text down to about 1,900 tokens of answers. That took 11 scoring calls, cost about $0.007, and finished in 8 seconds.

How it works

The decisions come from Jev, a small model from TypeSafe that only answers yes/no questions with a probability. For each page, jevcrawl sends one request:

"Is this passage relevant to the goal?" for every text block
"Does this link lead to the answer?" for every link

Plain code does the rest. It keeps the high-scoring passages, orders the links by priority, and asks once more whether the kept passages already answer the goal, so it can stop early.

Because the model only returns numbers, it cannot invent text, and a page cannot give it instructions. At worst, a prompt-injection attempt biases a score.

A few details turned out to matter:

Sitemap seeding. On big sites with a small page budget, jevcrawl reads the sitemap (including gzip files and sitemap indexes) from the robots.txt Sitemap lines and ranks the entries by how much they overlap with the goal words.
A reserve for weak links. Links under the priority floor go to a reserve queue instead of being dropped, so the crawl can come back to them if the good links run out.
Boilerplate detection across pages. Menus and footers are kept as link candidates but not returned as text.

Benchmark

I used WebWalkerQA (Wu et al., Alibaba Tongyi Lab, ACL 2025), a benchmark of questions buried several clicks deep in real websites. I re-verified 50 tasks against the live sites and did not look at them during development. Every system used the same answering LLM and the same judge.

System	Correct	Found answer page	Cost per task	Time per task
jevcrawl	40%	54%	$0.025	27.5 s
Claude picks each link	30%	33%	$0.194	61.7 s
Breadth-first crawl	8%	11%	$0.228	22.4 s

I also compared against Crawl4AI's best-first deep crawl, a keyword-scored crawler with no LLM. On 20 other tasks, measuring only whether the crawler fetched the answer page, jevcrawl reached it on 10 of 20 and Crawl4AI on 5 of 20. The median time was 29 s against 43 s.

These are small samples (50 and 20 tasks). I trust the cost gap more than the accuracy gap, and the Crawl4AI test measures reaching the page, not final answers. The benchmark code is in the repo's bench/ folder, so you can run it or try your own tasks.

Limits

No JavaScript rendering. Pages that only render with JS are reported in stats.needs_js_urls and skipped. Crawl4AI drives a real browser and handles those. Optional rendering is an open issue.
No robots.txt enforcement yet. It only reads robots.txt for Sitemap lines. Open issue.
It depends on a hosted model. Scoring is done by TypeSafe's Jev API, so you need an API key, and page text is sent to TypeSafe for scoring. A crawl costs roughly a cent in Jev calls. The crawler itself is free and open source. I am not affiliated with TypeSafe.

Try it

For Claude Code:

claude mcp add jevcrawl -e TYPESAFE_API_KEY=<your key> -- uvx jevcrawl

It also works in Claude Desktop and Cursor through a standard mcpServers entry; the README has the JSON.

The repo is at github.com/RayenMlayeh/jevcrawl. There are several good-first-issues open: robots.txt support, a Dockerfile, Windows and macOS CI, and more.