Your firewall is your AI policy — I probed 18 major sites to read it

Everyone's arguing about AI search visibility. Almost nobody is measuring the boring part: whether AI crawlers can fetch your pages at all.

So I built a small open-source tool — geo-crawl-audit — that probes any site with the user-agents of every AI crawler that matters (GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, ChatGPT-User, and friends), measures how each is treated versus a normal browser, and checks the thing most people never think about: how many words exist in the raw HTML before any JavaScript runs. Because here's the detail the industry keeps missing — GPTBot, ClaudeBot, and PerplexityBot don't execute JavaScript. For most sites, Googlebot (feeding Gemini) and Applebot are the only AI-adjacent crawlers that render it. A site can rank #1 in Google and be a blank page to nearly everything else.

I pointed it at 18 major sites on August 7. Five findings worth your time.

1. Access patterns line up with the business relationships

The Guardian — which has a content deal with OpenAI — serves my simulated GPTBot, OAI-SearchBot, and ChatGPT-User a clean 200. The same request wearing ClaudeBot, PerplexityBot, or CCBot: 403, and those names are in its robots.txt disallow list too. Policy and enforcement agree.

The New York Times — in litigation with OpenAI — 403s nearly everyone: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Common Crawl, meta-externalagent. Two user-agents got through: bingbot and Amazonbot.

I want to be careful about the claim here: a status code tells you who is blocked, not why. Any individual 403 has mundane explanations. But when the allow/deny matrix maps this cleanly onto public deals and public lawsuits, the firewall config has become a business document — and it's readable by anyone with a terminal.

2. robots.txt is a polite sign. Some doors are unlocked anyway.

Reddit's robots.txt blocks every AI bot in my list — fourteen tokens, no exceptions. Enforcement tells a different story: my GPTBot UA got a 403 and ClaudeBot and CCBot got rate-limited (429), while OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, and meta-externalagent all received a 200 from the same IP, seconds apart. And there's no clean training-vs-retrieval logic to it — meta-externalagent is a training crawler and walked through. robots.txt is honor-system; the WAF is the actual lock, and it wasn't turned for those names. (Caveat: my probe isn't coming from those vendors' published IP ranges — real-bot treatment can differ. That caveat is a feature of the tool, not a footnote; more below.)

Figma runs the inverse configuration: serves a 200 to every bot UA in the list, but robots-disallows GPTBot, OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot, CCBot, and Google-Extended — the stated policy runs on the honor system, with no user-agent-level enforcement my probe could detect.

3. Some famous sites are invisible to AI — deliberately or not

Airbnb returns 403 to nine of the twelve AI user-agents I probe — every OpenAI, Perplexity, Microsoft, Amazon, Meta, and Common Crawl token — while, curiously, all three Anthropic user-agents get a 200. And its homepage carries ~94 visible words of raw HTML anyway, so even the allowed crawlers read a nearly blank page. LinkedIn served my probe a 23-word bot-check interstitial — whatever a verified crawler negotiates, the raw HTML a stranger gets is effectively empty. Reddit's homepage contains exactly one visible word before JavaScript runs.

Meanwhile the sites doing it right are boringly consistent: Stripe (1,957 raw-HTML words, a 200 to every UA in my list), Anthropic (fastest warm response in the test at 0.102s), Vercel, MDN, Shopify — full server-rendered HTML, fast, no bot differentials against my probe. Score: 97–100.

4. Five sites filtered my probe entirely — and that's a finding about tooling honesty

Quora, OpenAI, Perplexity, Bloomberg, and GitHub challenged even my baseline browser request from my network. My first draft contained a detail I loved: that openai.com 403s a request claiming to be GPTBot because they verify their own crawler by IP range and treat name-only claims as impostors. The data doesn't support it. openai.com also 403'd my plain-browser baseline — so I can't distinguish "they detect GPTBot impostors" from "they challenge everything from my network." The simpler reading wins, and the good story got deleted.

That's why the tool flags these sites BASELINE_ANOMALY instead of scoring them confidently: when the baseline itself was filtered, you measured the site's filter, not the site. It points you at its second mode instead — server-log analysis, which reports what real crawlers actually received, verifies their IPs against vendor-published ranges, and surfaces the silent killer: 499s, the status your logs record when a bot got tired of waiting for your slow origin and hung up. No error page, no alert, no citation.

Most GEO tools would have happily reported "GitHub is invisible to AI." It isn't — my probe was. A diagnostic that overclaims is worse than no diagnostic.

5. What this means if you run websites

The order of operations for AI visibility is unglamorous:

1. Reachability — is your WAF/bot-management (often on defaults) silently challenging AI crawlers?
2. Speed — crawlers hit your uncached long-tail pages, so your cold-start TTFB is the number that counts, not the warm one your monitoring shows.
3. Readability — words in raw HTML, no JS. If you're client-rendered, you're invisible to nearly every AI system that isn't backed by Google's or Apple's rendering crawlers.
4. Permission — robots.txt, including the token-only entries (Google-Extended, Applebot-Extended) that never appear in your logs because they never fetch — Googlebot and Applebot do the fetching.

Everything else the GEO industry sells you sits downstream of these four gates. (And no, llms.txt isn't a gate: no major AI vendor documents fetching it, and analyses of real server logs keep failing to find bot requests for it. I check for it. I weight it at zero.)

If you want the 60-second version of this audit on your own domain, I put a free scanner up at readablebyai.com — same engine, plus it shows you the exact text an AI crawler can extract from your page. And yes: the scanner site itself scores 100/100 on its own audit, because on the first run it scored 70 and I had to eat my own cooking.

How this was built (transparently)

I built this in one working session with Claude as the pair — spec, research, code, and validation loop. The part worth sharing: the first live run produced phantom "WAF block" findings that turned out to be transient connection failures. We caught it because the validation run was against 14 production sites I operate, where I could sanity-check every claim — and the fix (retry logic + a separate probe-error class that never masquerades as a finding) became the tool's most important feature. Before publishing this post, an adversarial fact-check against the dataset killed four more of my favorite sentences — including "Airbnb blocks AI crawlers," which turned out to be two-thirds true: it 403s the OpenAI, Perplexity, Microsoft, Amazon, Meta, and Common Crawl UAs while serving all three Anthropic ones a 200. The meta-skill isn't writing Python. It's knowing what the tool — and the write-up — must refuse to claim.

---

Tool: github.com/abouchard11/geo-crawl-audit — MIT, zero dependencies (Python + curl). Full 18-site dataset in examples/. Longer analysis with the per-crawler status matrix: Your firewall is your AI policy. Corrections welcome — especially from anyone at the companies named, with logs.