DEV Community

Cover image for robots.txt and AI crawlers: what it actually controls, and what it only asks for
AumCreate
AumCreate

Posted on

robots.txt and AI crawlers: what it actually controls, and what it only asks for

#webdev #seo #ai

The useful mental model for robots.txt is: it is a fetch directive, not an access control system. It asks a crawler not to request a URL. Whether the request is honored depends entirely on the crawler, and it says nothing about what happens to content once it has already been fetched.

That single sentence explains almost every surprise people hit when they try to manage AI crawlers with robots.txt. Here is the breakdown, with tests you can run against your own site.

What robots.txt can do

1. Address crawlers individually. Grouping by User-agent uses longest-match precedence: a group naming a specific token beats the * group. So one file can allow search crawlers and disallow model-training crawlers at the same time.

User-agent: Googlebot
Allow: /

User-agent: GPTBot
Disallow: /

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Enter fullscreen mode Exit fullscreen mode

2. Keep compliant crawlers out of paths you never wanted fetched. Admin areas, internal search results, faceted URLs, staging leftovers. This works because the compliant crawler reads the file before requesting anything else.

3. Publish an auditable statement of intent. The file is public, versionable, and timestamped by your git history. If a dispute about usage ever comes up, "our policy has said this since date X" is checkable evidence. Nothing else in your stack gives you that for free.

4. Feed vendor-side switches. Several AI vendors read robots.txt as the input to whether they may use a site's content. Changing the file changes their behavior without a support ticket or a dashboard login.

What robots.txt cannot do

1. It cannot stop a crawler that ignores it. There is no enforcement mechanism. A scraper that never requests /robots.txt is completely unaffected by anything in it.

2. It cannot un-publish anything already collected. Blocking today does not remove yesterday's fetch from a dataset. The directive is forward-looking only.

3. It cannot prevent indexing of a URL discovered elsewhere. For search engines, Disallow prevents the fetch, not the URL from being listed. If other pages link to it, it can still appear as a bare URL with no snippet.

4. It cannot rate-limit. Crawl-delay is honored by some crawlers and ignored by others, and it has no effect on crawlers that don't read the file at all. Rate limiting belongs at the edge.

5. It cannot block by IP or ASN. There is no syntax for it. If a request comes from somewhere you don't want, the decision has to happen in your server or CDN config.

6. It cannot tell "AI search citation" apart from "AI model training". Those are separate vendor-defined tokens, and each vendor documents its own. You have to decide per token, not per "AI" in general.

7. It does not cover your whole site automatically. robots.txt is scoped per origin — scheme, host, and port. https://example.com/robots.txt does not govern https://www.example.com/ or http://example.com/. Check every hostname you actually serve.

8. It is not a security boundary. Treat it as a sign on the door, not a lock.

Testing your own site

Test 1 — what does your server return for a given user-agent?

Before blaming robots.txt, confirm your CDN or WAF isn't already blocking the token.

for ua in "GPTBot/1.0" "ClaudeBot/1.0" "CCBot/2.0" "Mozilla/5.0 (compatible; Googlebot/2.1)"; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -A "$ua" https://example.com/robots.txt)
  echo "$code  $ua"
done
Enter fullscreen mode Exit fullscreen mode

A 403 here means the block is happening above the application, and your robots.txt content is irrelevant for that token.

Test 2 — did the crawler actually read the file?

If a token appears in your access log but never appears fetching /robots.txt, that token is not following the directive. Grep for both:

# count hits per known AI token
grep -oE '(GPTBot|ChatGPT-User|OAI-SearchBot|ClaudeBot|Claude-User|anthropic-ai|PerplexityBot|CCBot|Bytespider|Amazonbot|meta-externalagent|Applebot-Extended)' access.log \
  | sort | uniq -c | sort -rn

# did they fetch robots.txt first?
grep robots.txt access.log | grep -icE 'gptbot|claudebot|ccbot'
Enter fullscreen mode Exit fullscreen mode

If the second number is zero and the first isn't, you have your answer about that crawler — regardless of what your file says.

Test 3 — is the same content reachable through a path you didn't think about?

This is the most common gap. You block a directory, and the same content is served at /feed/, /wp-json/wp/v2/posts, a print view, or an AMP variant. Blocking one path is not blocking the content.

for p in /feed/ /wp-json/wp/v2/posts /?print=1; do
  echo "$p -> $(curl -s -o /dev/null -w '%{http_code}' https://example.com$p)"
done
Enter fullscreen mode Exit fullscreen mode

If any of those return 200 with your text in the body, your robots.txt rule is decorative.

Test 4 — confirm the file itself is being served correctly

A 5xx on /robots.txt is treated by major search crawlers as a full disallow while it persists, and a 404 is treated as no restrictions. Both outcomes are usually accidental; a redirect is worse, because the directive is defined per origin.

curl -sI https://example.com/robots.txt | head -n 5
Enter fullscreen mode Exit fullscreen mode

You want a 200 with Content-Type: text/plain.

When robots.txt is not enough

If you have decided a crawler should not be served at all, the decision has to be enforced where you can actually refuse a request. A blunt user-agent check at the edge:

# http context
map $http_user_agent $block_training_crawler {
    default     0;
    ~*GPTBot    1;
    ~*ClaudeBot 1;
    ~*CCBot     1;
    ~*Bytespider 1;
}

server {
    listen 443 ssl;
    server_name example.com;

    if ($block_training_crawler) {
        return 403;
    }

    # ... rest of the server block
}
Enter fullscreen mode Exit fullscreen mode

Two caveats worth writing down before you deploy it: user-agent strings are trivially spoofable, so pair this with verified-bot identification from your edge provider rather than treating it as authentication; and blocking at the edge means the crawler never sees your robots.txt explanation, which is fine for enforcement but removes the audit trail.

The decision you actually have to make

robots.txt forces a policy question that most sites haven't answered: do you want to be fetchable by AI systems, citable by them, or trainable by them? Those are three different permissions, and the tokens are separate for a reason. Blocking the training crawler while allowing the user-triggered fetcher is a coherent position. Blocking everything and then wondering why AI-assisted answers cite a competitor is also a position — just not the one most people meant to take.

If you want the raw data first — which AI crawlers are actually hitting your site, per token, before you write a single line of policy — AumCrawl logs AI crawler hits and lets you allow or block per crawler.

Checklist

  • [ ] robots.txt returns 200 on every hostname you serve
  • [ ] Your policy is written per token, not per *
  • [ ] You checked whether the tokens in your log ever fetched /robots.txt
  • [ ] You checked whether the same content is reachable via feed, JSON API, or print view
  • [ ] Anything you truly need refused is refused at the edge, not only in the file
  • [ ] You have a written answer to "fetchable, citable, or trainable"

More on crawler policy and per-crawler controls: aumcreate.com/crawlers

Top comments (0)