Can AI crawlers read your site? robots.txt, llms.txt and JavaScript rendering, checked

Originally published on howseen.ai: howseen.ai/llms-txt

By Raphaël Aubry, founder of Howseen

An AI assistant can only cite a page it can read. Before anyone talks about "GEO" or content strategy, three boring things decide whether your site is even in the running:

1. Does your robots.txt let the AI crawlers in?
2. Is your content in the HTML the server sends, or only after JavaScript runs?
3. Do you have an llms.txt, and does it matter?

Here is how to check all three from a terminal, what the official documentation says, and a free tool that runs the same checks on up to 100 URLs at once.

Check 1: robots.txt, bot by bot

curl -s https://yoursite.com/robots.txt

The thing most people miss: AI companies run several crawlers with different jobs, and each one reads its own group in robots.txt. From each company's documentation:

Company	Token	Job
OpenAI	GPTBot	Collects content for model training
OpenAI	OAI-SearchBot	Surfaces sites in ChatGPT search
OpenAI	ChatGPT-User	Fetches pages for user actions in ChatGPT
Anthropic	ClaudeBot	Collects content for model training
Anthropic	Claude-SearchBot	Indexes content for Claude search results
Anthropic	Claude-User	Fetches pages when a Claude user asks
Perplexity	PerplexityBot	Surfaces and links sites in Perplexity search
Perplexity	Perplexity-User	Fetches pages when a user asks a question
Google	Google-Extended	Token for Gemini training and grounding, not a separate crawler

Three consequences:

• Search bots and training bots are independent. You can block GPTBot (training) and keep OAI-SearchBot (ChatGPT search). OpenAI says sites that block OAI-SearchBot do not appear in ChatGPT search answers.
• Google-Extended does not control AI Overviews. Those come from Google's search index, which Googlebot builds. Blocking Google-Extended keeps your content out of Gemini training, not out of AI Overviews.
• The dangerous line is the generic one. User-agent: * followed by Disallow: / blocks every bot that has no group of its own, AI crawlers included. In my experience it is usually a leftover from an old scraper defense that nobody reopened.

Reading robots.txt by eye works until rules overlap. The standard (RFC 9309) says the most specific user-agent group wins, then the longest matching rule, and Allow wins a tie. Here is that logic in the checker I describe at the end of this post:

const specific = parsed.groups.filter((g) => g.agents.includes(token));
if (specific.length) {
    matchedGroup = token;
    rules = specific.flatMap((g) => g.rules);
} else {
    const star = parsed.groups.filter((g) => g.agents.includes('*'));
    if (star.length) {
        matchedGroup = '*';
        rules = star.flatMap((g) => g.rules);
    }
}
let best = null;
for (const r of rules) {
    if (!ruleMatches(r.path, path)) continue;
    const len = r.path.length;
    if (!best || len > best.path.length || (len === best.path.length && r.type === 'allow')) best = r;
}
const allowed = !best || best.type === 'allow';

Note the fallback: a bot with its own group ignores the * group entirely. Adding User-agent: GPTBot with a single Disallow: /private/ means GPTBot no longer sees anything you wrote under *.

One more layer sits before robots.txt: your firewall or CDN. A WAF rule that blocks unknown bots can reject AI crawlers before they read a single line. If your logs show no hits from these tokens at all, check there too.

Check 2: what a crawler sees without JavaScript

Most AI crawlers fetch the raw HTML and parse it. They do not run your JavaScript. If your page is a client-rendered shell, they get an empty <div id="root">.

Fetch the page with a crawler user agent and count the words that are actually in the HTML:

UA="Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko); compatible; GPTBot/1.4; +https://openai.com/gptbot"

curl -s -A "$UA" https://yoursite.com/pricing \
  | perl -0pe 's/<script.*?<\/script>//gs; s/<style.*?<\/style>//gs; s/<[^>]+>/ /g' \
  | wc -w

It is crude (navigation and footer count too), but the order of magnitude tells the story. A few dozen words on a page that looks full in the browser means the content arrives through JavaScript. The same command also shows whether your server or CDN answers this user agent with a 200 at all: add -o /dev/null -w "%{http_code}\n".

The checker uses the same idea with explicit thresholds: under 100 words, the page scores 0 on this check, and if the HTML also contains a framework root, it is flagged as rendered in the browser.

const SPA_MARKERS = [
    ['id="root"', 'React'],
    ['id="__next"', 'Next.js'],
    ['id="app"', 'Vue or similar'],
    ['id="__nuxt"', 'Nuxt'],
    ['ng-version', 'Angular'],
    ['<app-root', 'Angular'],
    ['id="svelte"', 'Svelte'],
    ['data-reactroot', 'React'],
];
const likelyClientRendered = wordCount < 100 && (markers.length > 0 || /<script\b[^>]*src=/i.test(html));

The fix depends on the stack: server components in the Next.js App Router, SSR mode in Nuxt, pre-rendering or static generation for a React SPA. You can also do the manual test: View Page Source (not Inspect) and look for your headings.

Check 3: llms.txt, and what it does not do

llms.txt is a Markdown file at the root of your site that summarises it and links the key pages. Jeremy Howard proposed the format on September 3, 2024, at llmstxt.org. The only required part is an H1 with the site name.

curl -s -o /dev/null -w "%{http_code} %{content_type}\n" https://yoursite.com/llms.txt
# 200 text/plain; charset=utf-8   <- what you want

The classic failure is a site that answers every unknown path with its home page: you get a 200, but with text/html. The checker treats that as "no llms.txt".

Now the honest part. No major AI engine has confirmed using llms.txt to decide which brands to cite. Google says you don't need to create new machine readable files, AI text files, or markup to appear in AI Overviews and AI Mode. OpenAI points site owners to robots.txt and OAI-SearchBot for ChatGPT search. Where llms.txt is used is inference on documentation: OpenAI publishes its own docs index as an llms.txt file, and AI coding tools read these files to land on the right page.

File	What it does	Effect on AI visibility
robots.txt	Allows or blocks crawlers	Direct: a blocked crawler cannot cite you
sitemap.xml	Lists the URLs you want indexed	Indirect: helps pages get indexed
llms.txt	Summarises the site, links key pages	None confirmed by a major AI engine

So: write one if you have docs, an API or many pages (it takes minutes), and check robots.txt first.

Run all of it on many URLs

I packaged these checks as a free Apify Actor, AI Visibility Checker. Plain HTTP requests, no AI calls, no third party login. You paste up to 100 URLs per run and get a score out of 100 with one sentence per fix:

Check	Points
AI search and assistant crawlers in robots.txt	25
Text readable without JavaScript	20
Structured data (JSON-LD)	15
XML sitemap	7
AI training crawlers in robots.txt	5
Indexable, title, meta description, canonical, llms.txt	5 each
<html lang>	3

Training crawlers weigh little on purpose: blocking them is a valid choice. robots.txt, llms.txt and the sitemap are fetched once per domain, even if you check many pages of the same site.

{
  "urls": ["https://example.com", "https://example.com/pricing"],
  "timeoutSecs": 15
}

A useful pattern is to schedule it after each deploy, so a robots.txt change or a rendering regression shows up the same day instead of months later.

For the per-bot details (user agent strings, published IP ranges, how to spot a fake GPTBot in your logs), see the AI crawlers robots.txt guide.

---

See what AI says about your brand: howseen.ai