A practical breakdown of what robots.txt actually controls for AI crawlers in 2026: which bots like GPTBot, ClaudeBot, and PerplexityBot respect it, which user-triggered fetchers ignore it by design, and how to check your live file for wildcard rules blocking AI traffic you didn't mean to block.
This article was originally published on byfaro.ai/blog/robots-txt-for-ai-crawlers-what-actually-blocks-them. Read the full guide there — it includes step-by-step examples and Faro audit tool links.
What you'll learn
This guide covers the full picture on robots.txt for ai crawlers: what actually blocks them, including:
- Why this matters for AI agent discoverability
- What to check on your own site
- The specific fixes that move your score
Faro audits whether AI agents can find, understand, and act on your business — like GTMetrix but for AI discoverability. Free scan at byfaro.ai.
Top comments (0)