DEV Community

Aulvem
Aulvem

Posted on Originally published at aulvem.com

User-agent: * doesn't apply to crawlers you named in robots.txt

#seo #webdev #python #cloudflare

A Disallow under User-agent: * does nothing for a crawler that has its own group in robots.txt. Each crawler reads exactly one group, the one matching its name, and * is only the fallback for crawlers you didn't name.

I learned this from my own logs. I had blocked my /go/ redirect links under *, and Cloudflare-verified Googlebot still fetched them 133 times in three days, ClaudeBot 213 times in five. Here's why, how to fix it, and a small script that catches the mistake before it ships.

The version that doesn't work 

User-agent: Googlebot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: *
Allow: /
Disallow: /go/
Enter fullscreen mode Exit fullscreen mode

It's easy to end up here: you allow AI search crawlers by name, then later add a Disallow under *. In this file, /go/ is allowed for both Googlebot and ClaudeBot.

Both specs say so:

  • RFC 9309, section 2.2.1: if no matching group exists, crawlers MUST obey the * group.
  • Google's robots.txt spec: only one group is valid for a particular crawler, and the others are ignored.

Groups don't inherit from each other, and * is not a shared base.

The fix

Write each blocked path into every named group.

User-agent: Googlebot
Allow: /
Disallow: /go/

User-agent: ClaudeBot
Allow: /
Disallow: /go/

User-agent: *
Allow: /
Disallow: /go/
Enter fullscreen mode Exit fullscreen mode

Keeping Allow: / next to Disallow: /go/ is fine. RFC 9309, section 2.2.2, picks the longest matching rule, so for /go/amazon/... the four-character Disallow: /go/ beats the one-character Allow: /.

Once Googlebot re-read the fixed file, it stopped:

Day (UTC) Googlebot → /go/
Oct 26 49
Oct 27 (fixed at 13:31) 52
Oct 28 – Nov 2 0

It took about four hours to stop, which fits Google caching robots.txt for up to 24 hours.

A script that finds the group you forgot

With a dozen named groups, you will eventually miss one. This script collects every User-agent in the file and checks the paths you want blocked against each of them:

import re
import sys
from protego import Protego  # pip install protego

def named_agents(text):
    """Every User-agent name in robots.txt except *."""
    names = re.findall(r"(?im)^\s*user-agent\s*:\s*([^\s#]+)", text)
    return sorted({n for n in names if n != "*"}, key=str.lower)

def check(text, base, paths):
    rp = Protego.parse(text)
    agents = named_agents(text) + ["SomeUnnamedBot"]  # last one exercises the * group
    leaks = []
    for ua in agents:
        for p in paths:
            ok = rp.can_fetch(base + p, ua)
            print(f"{ua:20s} {p:28s} {'allow' if ok else 'DISALLOW'}")
            if ok:
                leaks.append((ua, p))
    return leaks

if __name__ == "__main__":
    robots_file, base, *paths = sys.argv[1:]
    leaks = check(open(robots_file, encoding="utf-8").read(), base, paths)
    if leaks:
        print(f"\n{len(leaks)} blocked path(s) are still allowed")
        sys.exit(1)
Enter fullscreen mode Exit fullscreen mode 
python check_groups.py public/robots.txt https://example.com /go/amazon/B0CL6H35DY
Enter fullscreen mode Exit fullscreen mode

On my old file, all 13 named groups came back allow and only SomeUnnamedBot was blocked, which is exactly what the logs showed. On the fixed file every row says DISALLOW and it exits 0. Put the paths you care about in CI and a new group without the Disallow will fail the build.

Don't use urllib.robotparser for this

Python's standard urllib.robotparser says everything is allowed, even with the fixed file:

from urllib.robotparser import RobotFileParser

rp = RobotFileParser()
rp.parse(open("robots.txt", encoding="utf-8").read().splitlines())
rp.can_fetch("Googlebot", "/go/amazon/B0CL6H35DY")  # True, even after the fix
Enter fullscreen mode Exit fullscreen mode

It reads a group top to bottom and stops at the first matching line. With Allow: / first, it never reaches Disallow: /go/. Use Protego, the longest-match parser Scrapy uses. I tested with 0.7.0.

nofollow and X-Robots-Tag don't stop the fetch

My redirect links had rel="sponsored nofollow" and returned X-Robots-Tag: noindex, nofollow. Googlebot still fetched 67 different redirect URLs.

If you need the request itself to stop, use robots.txt.

The hour-by-hour data after the fix, why I can't credit the fix for ClaudeBot (it stopped two days earlier), and the unnamed GoogleOther that showed up anyway are on Aulvem → Aulvem | User-agent: * doesn't apply to crawlers you named — Googlebot fetched my disallowed links 133 times

Top comments (1)

Collapse
 
hanin_mp profile image
Hanin Mp •

Using the logs to prove it is what makes this convincing. One related trap: as I read Google's docs, crawlers like Googlebot-Image and Googlebot-News fall back to the Googlebot group when they have no group of their own, not to *. So a Disallow added only under * misses them too. It could be worth adding those product tokens to the agent list in your CI check. Did you see any of them hitting /go/ before the fix?