Here is a robots.txt. Is /private/report.pdf crawlable?
User-agent: *
Disallow: /private/
Allow: /private/report.pdf
If you answered "no, it's under /private/", you have company — including, in my experience, a few online testers. The correct answer is yes, and the reason is a rule that is easy to state and easy to implement backwards.
I ran into this while writing a robots.txt tester, which is a good way to discover that you did not understand a spec as well as you thought.
The three rules that decide everything
RFC 9309 standardised what Google's parser had been doing for years. Three rules, applied in order:
1. One group wins, and only one.
A crawler scans the file for the group whose User-agent names it most specifically. User-agent: * is a fallback, not a base class. If a group names the bot directly, the wildcard group is ignored entirely — its rules are not merged in, not inherited, not consulted.
User-agent: *
Disallow: /
User-agent: GPTBot
Allow: /
GPTBot may crawl everything. It does not "inherit" the Disallow: /. This surprises people who read the file top-down like a firewall.
2. The longest matching rule wins — not the first.
This is the one in the example above. /private/ is 9 characters, /private/report.pdf is 19. The longer pattern matches, so the Allow decides, and that single file stays crawlable while the rest of the folder does not.
Order in the file is irrelevant. You can put the Allow first, last, or between two unrelated lines; the outcome is the same. An implementation that returns the first match will disagree with Google on every file that uses this pattern — and it is a common pattern, because it is the only way to expose one file from a blocked directory.
3. On a tie, Allow wins.
User-agent: *
Disallow: /a
Allow: /a
Same length, both match. /a is crawlable. This is the case I would bet most hand-rolled parsers get wrong, because it only shows up if you deliberately test for it.
A test file for your tester
Paste this into whatever tool or library you rely on, then check the verdicts against the right-hand column. It takes a minute and it is worth doing before you trust a tool with a migration.
User-agent: *
Disallow: /private/
Allow: /private/report.pdf
Disallow: /a
Allow: /a
Disallow: /*.pdf$
User-agent: GPTBot
Allow: /
| URL | Correct verdict | Why |
|---|---|---|
/private/notes.txt |
Blocked | Only Disallow: /private/ matches |
/private/report.pdf |
Allowed | Longer Allow beats shorter Disallow
|
/a |
Allowed | Equal length, Allow wins the tie |
/docs/manual.pdf |
Blocked |
/*.pdf$ matches, $ anchors the end |
/docs/manual.pdf.html |
Allowed |
$ anchors — the path does not end in .pdf
|
/private/notes.txt as GPTBot |
Allowed | Specific group replaces the wildcard group entirely |
If your tool disagrees on rows 2, 3 or 6, it is not implementing RFC 9309, and its verdicts will drift from reality exactly where it matters.
Two more things that bite
An empty Disallow: restricts nothing. It is the standard way of saying "everything is open". More than once I have seen it read as "block everything" — the opposite of the truth. Disallow: / with the slash is what blocks.
* and $ are the only wildcards. No regex, no character classes. * matches any run of characters, $ anchors the end of the path. Everything else is a literal, including ? and ..
Where this matters more than it used to
The rules above have been stable for years. What changed is who is reading the file.
A robots.txt now governs at least three different populations, and they are separate user-agents doing separate jobs:
-
Search crawlers —
Googlebot,Bingbot. They decide whether you appear in search results. -
AI training crawlers —
GPTBot,ClaudeBot,CCBot,Google-Extended,Bytespider. They collect pages to train models. -
Live-fetch bots —
ChatGPT-User,Claude-Web,PerplexityBot. They fetch a page because a user asked something right now.
The distinction that trips people up most: Google-Extended controls Gemini training and has no effect whatsoever on Google Search ranking. Blocking it does not deindex you. Googlebot is what crawls for Search. They are different bots with different rules, and a group naming one says nothing about the other.
Which means a file like this is coherent, and increasingly common:
User-agent: *
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
Search engines crawl everything. Training crawlers get nothing. Nothing about your rankings changes.
What robots.txt still cannot do
It is a crawling instruction. It is not an access control and not an indexing control.
A disallowed URL can still appear in search results if other sites link to it — the crawler never fetches the page, so it never sees your noindex, and it can list the URL from the link alone. If you want a page out of the index, allow the crawl and serve noindex. If you want it private, require authentication. And robots.txt is a public file: writing Disallow: /secret-admin/ publishes the existence of /secret-admin/ to anyone who looks.
I built a robots.txt tester that implements the precedence above and shows you the exact line that decided each verdict, including the AI crawlers. It runs entirely in the browser, so you can paste a file that is not published yet. The six rows in the table are its test suite.
Top comments (0)