DEV Community

Cover image for See Your Site the Way an AI Crawler Does: robots.txt, Status Codes and No JavaScript
Furqan Khalid
Furqan Khalid

Posted on

See Your Site the Way an AI Crawler Does: robots.txt, Status Codes and No JavaScript

#ai #seo #webdev #webscraping

When a page ranks on Google but never shows up in ChatGPT or Perplexity answers, the cause is often technical rather than editorial. Somewhere between the AI crawler and your content, something says no, or returns an empty shell.

Google has spent two decades learning to cope with messy sites, and it renders JavaScript. Many AI crawlers are less forgiving. This post builds a small audit script for the three most common technical blockers:

  1. robots.txt disallows AI user agents, sometimes without anyone deciding to.
  2. The CDN or WAF blocks or challenges them before robots.txt even matters.
  3. The content doesn't exist without JavaScript, so a non-rendering fetcher gets an empty page.

Check 1: robots.txt per AI user agent

Python's standard library can evaluate robots.txt rules per user agent:

# ai_audit.py
from urllib.robotparser import RobotFileParser
from urllib.parse import urljoin, urlparse

AI_AGENTS = [
    # training
    "GPTBot", "ClaudeBot", "CCBot", "Google-Extended", "Applebot-Extended",
    # search indexing
    "OAI-SearchBot", "Claude-SearchBot", "PerplexityBot",
    # user-triggered fetches
    "ChatGPT-User", "Claude-User", "Perplexity-User",
]

def robots_report(page_url: str) -> dict[str, bool]:
    root = f"{urlparse(page_url).scheme}://{urlparse(page_url).netloc}"
    rp = RobotFileParser(urljoin(root, "/robots.txt"))
    rp.read()
    return {ua: rp.can_fetch(ua, page_url) for ua in AI_AGENTS}
Enter fullscreen mode Exit fullscreen mode

Two caveats:

  • RobotFileParser follows the original robots.txt convention and doesn't implement every detail of RFC 9309 (such as longest-match precedence between Allow and Disallow). For edge cases, confirm with a spec-compliant parser like Google's open-source robotstxt library.
  • Some vendors document that user-triggered agents may not follow robots.txt in the same way as crawlers, because a human requested the page. Read each vendor's bot docs for the current policy rather than assuming.

Decide on purpose. Training, search and user-triggered agents are separate so you can, for example, block training while allowing the agents that fetch pages to answer live questions:

# robots.txt
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
User-agent: Claude-SearchBot
User-agent: PerplexityBot
Allow: /
Enter fullscreen mode Exit fullscreen mode

Google-Extended controls whether Google may use your content for Gemini model training and grounding. It does not affect Google Search or AI Overviews, which use regular Googlebot.

Check 2: What does the edge actually return?

robots.txt is advisory. Your CDN or WAF decides what really gets served, and several bot-protection products now block or challenge AI crawlers by default.

Request the page with each bot's user agent and look at the status code and body size:

import requests

UA_STRINGS = {
    "browser": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126 Safari/537.36",
    "GPTBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; GPTBot/1.1; +https://openai.com/gptbot)",
    "ClaudeBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)",
    "PerplexityBot": "Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; PerplexityBot/1.0; +https://perplexity.ai/perplexitybot)",
}

def edge_report(url: str) -> dict:
    out = {}
    for name, ua in UA_STRINGS.items():
        try:
            r = requests.get(url, headers={"User-Agent": ua}, timeout=20, allow_redirects=True)
            challenged = any(s in r.text[:5000].lower() for s in
                             ("just a moment", "cf-challenge", "captcha", "access denied"))
            out[name] = {"status": r.status_code, "bytes": len(r.content), "challenge": challenged}
        except requests.RequestException as e:
            out[name] = {"error": str(e)}
    return out
Enter fullscreen mode Exit fullscreen mode

Read the result relative to the browser row. If browser gets 200 with 80 KB and GPTBot gets 403, or a 6 KB challenge page, the edge is blocking it.

Important limitation: this test sends a spoofed user agent from your IP. Some CDNs verify bots by IP, so they may block your fake GPTBot while letting the real one through. Equally, an allowlist could pass your test and still block the real crawler. Treat a block here as a strong hint, then confirm with:

  • your CDN's bot or firewall event logs filtered by those user agents, and
  • your origin access logs (the previous post in this series covers that), looking for real, IP-verified hits and their status codes.

Check 3: Does the content exist without JavaScript?

This is the most common silent failure on modern frontends. If the main content is fetched and rendered client-side, a crawler that doesn't execute JS sees a <div id="root"></div> and a bundle.

Compare what's in the raw HTML with what a browser renders:

from bs4 import BeautifulSoup

def visible_text(html: str) -> str:
    soup = BeautifulSoup(html, "html.parser")
    for t in soup(["script", "style", "noscript", "template", "svg"]):
        t.decompose()
    return " ".join(soup.get_text(" ").split())

def raw_html_report(url: str, must_contain: list[str]) -> dict:
    html = requests.get(url, headers={"User-Agent": UA_STRINGS["GPTBot"]}, timeout=20).text
    text = visible_text(html)
    return {
        "raw_words": len(text.split()),
        "missing_phrases": [p for p in must_contain if p.lower() not in text.lower()],
    }
Enter fullscreen mode Exit fullscreen mode

must_contain should be a few sentences you know are in the page's main content: the price, the product description, the answer to the page's main question. If they're missing from raw HTML, AI fetchers that don't render JS can't see them.

For a rendered comparison, add Playwright:

from playwright.sync_api import sync_playwright

def rendered_words(url: str) -> int:
    with sync_playwright() as p:
        b = p.chromium.launch()
        page = b.new_page()
        page.goto(url, wait_until="networkidle")
        n = len(visible_text(page.content()).split())
        b.close()
        return n
Enter fullscreen mode Exit fullscreen mode

A raw/rendered word ratio well below ~0.8 on a content page means a meaningful share of it depends on JavaScript.

Bars comparing raw HTML words vs rendered words: blog and docs 98%, pricing 13%, comparison 17%, integrations 13%

Fixing it

  • Next.js / Nuxt / SvelteKit / Astro: make sure content pages use SSR or static generation, not a client-only fetch in useEffect / onMounted. In the Next.js App Router, Server Components render on the server by default. The usual mistake is a "use client" page that fetches its content after mount.
  • SPA with no SSR option: prerender content routes at build time.
  • Tabs, accordions, "read more": keep the text in the DOM and hide it with CSS rather than loading it on click.

Putting it together 

import json, sys

if __name__ == "__main__":
    url, *phrases = sys.argv[1:]
    print(json.dumps({
        "robots": robots_report(url),
        "edge": edge_report(url),
        "raw_html": raw_html_report(url, phrases),
    }, indent=2))
Enter fullscreen mode Exit fullscreen mode 
python ai_audit.py https://example.com/pricing "per seat per month" "Starter plan"
Enter fullscreen mode Exit fullscreen mode

Run it across your top 20 pages by revenue, not your homepage. Pricing, docs, comparison and integration pages are where AI answers pull from most, and they're the pages most likely to be built as client-rendered app routes.

Audit matrix of seven user agents against robots.txt, edge status, challenge page and price-in-raw-HTML checks

Beyond the three blockers

Passing these checks means AI crawlers can read your page. It says nothing about whether they'll use it, which depends on structure, schema, trust signals and whether the page says anything worth quoting. The next post covers that half.

For both halves at once, Vista AI's Page Audit checks a page across seven areas: traditional SEO signals, AI crawlability and rendering, schema, AI content extraction (RAG), E-E-A-T signals, link quality and citation potential. It returns 0–100 scores for AI Visibility, AI Understanding and AI Citation, a prioritised fix list, and side-by-side scoring of competitor pages.

If you just want the quick version this week: check robots.txt, check your CDN's bot settings, and view-source on your five most important pages.

Top comments (1)

Collapse
 
citedy profile image
Dmitry Sergeev •

We need to produce a YouTube comment per guidelines. Should be short, one or two sentences, maybe a fragment. Should start with lower case, a specific reaction or question about the video. Should not be generic praise. Should avoid marketing phrases. Should be casual. No URLs. No markdown. No double hyphen. Should be plain text. The comment: maybe ask about handling robots.txt with no JavaScript, or ask if status code 404 vs 410 matters for AI crawlers. Something like: "any tips on