The received wisdom about crawl-delay is that it is dead. It was never in the original robots.txt specification, Google has said publicly that it ignores the directive, and most crawler parsers are modelled on Google's. So the standard advice is: do not bother, rate-limit at the CDN instead.
I maintain a directory of AI crawler user agents, and I was refreshing it this week when I realised I had repeated that advice on 28 pages without checking it against a single operator's documentation. So I read all of them.
Four of the 28 state a position. Two of those support it.
The two that honour crawl-delay
YouBot (You.com). Their documentation says it honours crawl-delay directives, alongside user-agent-specific rules. Worth noting separately: You.com caches your robots.txt for 30 minutes, so a change takes up to half an hour to take effect. That alone explains a lot of "the bot is ignoring my robots.txt" reports.
ImagesiftBot (The Hive). This one is the most precisely documented crawl-delay implementation I found anywhere, from any crawler:
ImagesiftBot also supports the crawl-delay directive in robots.txt files. It interprets the value as the minimum duration, in seconds, between the start of consecutive requests.
And then it spells out the arithmetic. With this in your robots.txt:
User-agent: ImagesiftBot
Crawl-delay: 5
…it splits each day into 5-second intervals and issues at most one request to your domain inside each interval. That is a real, specified contract. Most crawlers do not tell you what a directive means to them, only whether they read it.
The two that explicitly refuse
Amazonbot. Amazon's docs: "They do not support the crawl-delay directive." Plain and unambiguous.
Applebot. Apple's docs: "Applebot does not follow crawl-delay. Applebot is engineered for efficiency and will adjust to minimize the impact on site owners."
Both are useful precisely because they are explicit. You know where you stand.
The other 24 say nothing at all
GPTBot, ClaudeBot, PerplexityBot, CCBot, Bytespider, Google-Extended and the rest publish no position on crawl-delay either way.
Here is the part I had been getting wrong: "undocumented" and "declines to" are different facts, and collapsing them into one loses information.
In practice you should treat undocumented as unsupported — that is the safe assumption and it is almost certainly correct. But it is an assumption, not a finding, and a directory that prints it as a finding is lying slightly. So I keep them as separate values now, and the page tells you which one you are looking at.
What to actually do
Nothing here changes the practical advice much, but it sharpens it:
- Set crawl-delay for the two that honour it if those crawlers are hitting you. It costs one line and it is enforced.
-
For everything else, rate-limit at the server or CDN. A Cloudflare rate-limiting rule or an nginx
limit_reqzone matched on the user agent is enforced; a crawl-delay line is a request that most agents never read. - Remember robots.txt is cached. You.com caches for 30 minutes and it is not the only one. If a bot keeps hitting you right after a robots.txt change, wait before concluding it is misbehaving.
- Verify the agent before you act on it. A user-agent string is self-declared. Only 6 of the 28 publish IP ranges as JSON (OpenAI ×3, Apple, Perplexity ×2) — for everything else you are trusting a header anything can send.
The data
All of this is in the directory I maintain — one page per crawler with the robots.txt rule, the server and CDN equivalents, the published IP range where one exists, and now the crawl-delay position: geoprompttracker.com/bots.
The underlying registry is CC BY 4.0 and generated from a single source file, so the pages and the dataset cannot drift apart. JSON and CSV: /data/ai-crawlers.json and /data/ai-crawlers.csv. The crawl_delay column is yes, no, or empty for undocumented.
Full disclosure: the site is mine and it is free, no accounts.
If you have server logs showing a crawler behaving differently from what its docs claim — on crawl-delay or on robots.txt generally — I would genuinely like to hear it. The compliance column should reflect observed behaviour, and logs beat documentation every time.
Top comments (1)
We need to produce a short YouTube comment, casual, one or two sentences, start with lowercase. Must refer to the video content: crawling delay, AI crawlers. Maybe ask a question: "did any of the crawlers actually respect the delay?" Use slang. No quotes, no hashtags, no markdown. No URLs. No double hyphen. Ensure no em dash. Let's craft: "i was surprised only 4 actually mention crawl‑delay, do you think that's gonna change soon?" Wait can't use hyphen?