Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

Josh White — Mon, 12 Oct 2026 00:26:09 +0000

This is a submission for the Kaggle Benchmarking Challenge.

How this was made: I built this with an AI coding agent (Claude). It wrote most of the code and the first draft of this post. I reviewed the code, reran the checks and edited the post, and I stand behind every number. All numbers come from the Kaggle runs linked below.

What I Benchmarked

Every web crawler, from Googlebot to the AI agents now browsing on our behalf, is supposed to check a site's robots.txt before fetching a page. Site owners are writing more of these files than ever, often with a chatbot's help, to keep AI crawlers out. So I asked two questions that have exact answers:

Can a model read a robots.txt? Given the file, a crawler name and a URL, is the fetch allowed?
Can a model write one? Given a plain-English policy ("keep GPTBot out of /blog/, keep everyone out of /private/ except /private/press/"), does the file it writes actually do that?

Robots.txt has a real standard, RFC 9309 (2022), so neither question needs a judge model. Every answer is checked by a reference parser.

The three tasks

Task	What the model gets	Cases
Read: rules	a small file testing one rule of the standard (10 categories: longest match, ties, `*`, `$`, the crawler's own group, merged groups, path case, …)	240
Read: real sites	the real robots.txt of 41 public sites (Wikipedia, GitHub, Amazon, NYT, Etsy, …), with URLs placed on the edges of their rules	240
Write: policy	a 3–6 line policy. The file it writes is then probed with every crawler × URL the policy covers	40 policies, 1,836 probes

How the answers are checked. My RFC 9309 matcher agrees with Protego (Scrapy's parser) on all 480 reading cases. For each policy I also generated a reference file that passes every probe, so each policy can be done. A written file passes only if every probe is right, because one wrong rule in a real robots.txt lets a crawler in.

A baseline that surprised me: Python's own urllib.robotparser gets only 70.4% of the rules cases and 73.3% of the real-site cases right. It lets the first matching rule win (the standard says the longest), and it doesn't understand * or $. If your scraper uses the standard library, it is misreading about one robots.txt question in four.

Models Tested

Nine models from five families, small and large, all run through Kaggle's Model Proxy (Kaggle pays for the calls):

Family	Models
Google	Gemini 3.7 Flash, Gemini 3.5 Flash-Lite, Gemini 3.1 Pro (preview), Gemma 4 31B (open weights)
Anthropic	Claude Sonnet 5.5, Claude Haiku 4.5
OpenAI	GPT-6.1 Sol, GPT-5.4 nano
Alibaba	Qwen3-Next 80B-A3B (open weights)

Models I tried and left out, and why. I would rather say it than hide it:

Claude Opus 5.5: too expensive for Kaggle's $10-a-day model quota alongside everything else.
DeepSeek-R1: the daily quota ran out mid-run (184 of 240 rules cases and 159 of 240 real-site cases done). An unfinished run is not a score, so it is excluded, not shown as a low one.
Qwen3-235B and gpt-oss-120b: through the proxy they would not answer even a one-word test prompt within 100 seconds, so their runs timed out with almost nothing done. That looks like a proxy-side problem, not a model one.
Grok 4.6: no quota or time left before the deadline.

Each model ran each task once at the proxy's default settings. Writing scores move between runs: the same model (Gemini 3.7 Flash) wrote 24, then 27, then 22 of 40 policies fully right on three runs of the benchmark, so treat gaps of a few policies as noise. The gaps in the table below between the top and bottom are far larger than that.

Findings

Model	Read: rules	Read: real sites	Write: policies fully right	Write: probes right	Probes that let a blocked crawler in
Claude Sonnet 5.5	0.988	1.000	40 / 40	100.0%	0
Gemini 3.1 Pro	0.996	1.000	28 / 40	92.3%	110
Gemini 3.7 Flash	1.000	1.000	22 / 40	97.5%	0
GPT-6.1 Sol	1.000	0.971	17 / 40	69.9%	552
Gemma 4 31B	0.967	0.992	14 / 40	92.3%	78
Gemini 3.5 Flash-Lite	0.929	0.921	15 / 40	95.3%	74
Claude Haiku 4.5	0.871	0.925	10 / 40	90.1%	125
Qwen3-Next 80B	0.854	0.896	5 / 40	76.4%	372
GPT-5.4 nano	0.792	0.787	5 / 40	77.6%	371

(Gemini 3.1 Pro: one case in each of two tasks failed on Kaggle's side; both runs are above the 95%-completed bar I set before scoring, and the missing case counts as wrong.)

1. Reading is nearly solved, and the hard cases are the ones the standard library gets wrong. Six of nine models read the 240 single-rule files at 92% or better, and four score 99% or better on the real sites. The errors that remain cluster in wildcard rules (17% of answers wrong across models), prefix rules (12%) and $ end anchors (11%). On the real sites, the "near miss" questions, where a URL sits right on the edge of a rule, are wrong 8.8% of the time, against 3.4% for plain ones. The longest-match rule, the one Python's urllib.robotparser gets wrong, was answered correctly in all 216 attempts.

2. Writing is much harder, and "almost right" is the failure to watch. Across 360 attempts (9 models × 40 policies), only 156 files did everything the policy asked. Most models still get 90%+ of individual probes right, which is exactly why this is risky: a file that is 95% correct reads as correct. The worst leak: GPT-6.1 Sol read the rules perfectly (1.000), yet 552 of its probes let a crawler in that the policy blocks, with zero over-blocking. It writes files that are too permissive. The best writer, Claude Sonnet 5.5, passed all 40 policies with no leaks and no over-blocks.

3. Parameters and file extensions are where files break. Policies with a query-parameter rule (such as "block any URL with ?token=") were fully right 25% of the time, against 66% for policies without one. Policies with a file-extension rule ("never serve .pdf") were fully right 23% of the time, against 57% without. The most common slip: Disallow: /*.pdf$ blocks /report.pdf but not /report.pdf?dl=1, because $ anchors the end of the whole URL including the query string (88 leaked probes). Others: writing Disallow: /drafts/$ when the policy says "anything under /drafts/", which blocks only the folder's index page; and a parameter rule written without the =, so ?tokenx=1 gets blocked too (231 over-blocked probes). A crawler that has its own User-agent group stops obeying the * group, so a "no crawler may fetch /drafts/" rule has to be repeated in every named group. That second-largest leak cause (588 probes) is the trap I expected, and the models do fall into it, but it is not the biggest.

What I'd measure next: agents that fetch robots.txt themselves instead of being handed it, Crawl-delay and Sitemap lines, non-ASCII paths, and the same writing task with the RFC pasted in the prompt, to see whether the standard is the gap or the reasoning is.

My Benchmark

Benchmark: https://www.kaggle.com/benchmarks/oldjard/robots-txt-benchmark-can-llms-read-and-write-robo
Dataset (cases, the 41 robots.txt files as fetched on 2026-10-05, the checker code): https://www.kaggle.com/datasets/oldjard/robots-txt-benchmark
Everything is seeded and rebuildable. The reference parser is about 100 lines and the probe checker about 150.

Written with help from an AI agent (see the note at the top). Comments and replies are mine.

DEV Community: Josh White

Can LLMs read robots.txt? Easily. Can they write one? Only one of nine got every policy right.

What I Benchmarked

Models Tested

Findings

My Benchmark