DEV Community

Hanin Mp
Hanin Mp

Posted on AI-assisted

Which AI crawlers does your robots.txt actually block? Check it in Python

#ai #webdev #python #seo

Most robots.txt files grow by accretion. Someone adds a GPTBot block in 2023, someone else adds a wildcard rule, and nobody can say for sure which AI crawlers can fetch which section of the site today.

If you work on GEO or AEO, this matters. A blocked search crawler means your pages cannot be retrieved for AI answers. A blocked training crawler is often a deliberate choice. You want to know which is which.

Here is a small Python script that answers the question for any URL path, using only the standard library.

The script 

import sys
from urllib.robotparser import RobotFileParser
from urllib.request import Request, urlopen

AI_BOTS = [
    "GPTBot", "OAI-SearchBot", "ChatGPT-User",
    "ClaudeBot", "Claude-User", "PerplexityBot",
    "Google-Extended", "CCBot", "Bingbot", "Googlebot",
]


def load_robots(site):
    req = Request(site.rstrip("/") + "/robots.txt",
                  headers={"User-Agent": "robots-check/1.0"})
    with urlopen(req, timeout=15) as resp:
        return resp.read().decode("utf-8", errors="replace")


def check(robots_txt, site, path="/"):
    parser = RobotFileParser()
    parser.parse(robots_txt.splitlines())
    url = site.rstrip("/") + path
    return {bot: parser.can_fetch(bot, url) for bot in AI_BOTS}


if __name__ == "__main__":
    site = sys.argv[1]
    path = sys.argv[2] if len(sys.argv) > 2 else "/"
    for bot, ok in check(load_robots(site), site, path).items():
        print(f"{bot:16} {'ALLOW' if ok else 'BLOCK'}")
Enter fullscreen mode Exit fullscreen mode

Run it with python ai_bot_check.py https://yoursite.com /blog/some-post.

Testing it on a sample file

I ran check() against this robots.txt:

User-agent: *
Disallow: /admin/

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /blog/
Enter fullscreen mode Exit fullscreen mode

For the path /, only GPTBot comes back as BLOCK. For /blog/post, GPTBot and CCBot are both BLOCK, and every other bot is ALLOW. That second result is the useful one: a bot can be allowed on the homepage and blocked on the section that matters.

Why check per path

Checking only / hides section-level rules. Run the script for your money pages, your blog, and any folder with a Disallow rule. For a B2B or marketplace site, that usually means product pages, category pages and the resources section.

Caveats worth knowing

  • Search and training are different bots. Blocking GPTBot does not block OAI-SearchBot, and the two do different jobs. Read each vendor's crawler documentation before deciding.
  • Google-Extended is a control token, not a crawler. It never appears in your server logs. It only tells Google whether content can be used for certain AI uses.
  • Python's parser is not identical to Google's. The standard library applies rules in file order, while Google uses the most specific matching rule. If a file mixes Allow and Disallow on overlapping paths, verify the result with Search Console's robots.txt report.
  • robots.txt is only one layer. A CDN or WAF can still block these user agents with a 403, and the file will look fine. Compare the results with your server logs.

Next step

Once the script confirms robots.txt, test the real response. Request a page with each bot's user agent string and compare status codes. A mismatch between "ALLOW" in robots.txt and a 403 at the edge is one of the most common reasons a site is invisible to AI search.

How do you audit AI crawler access on your sites? I would like to hear what you check besides robots.txt.

Top comments (0)