Is your site invisible to ChatGPT? robots.txt, llms.txt and JavaScript checks for AI crawlers

Ed Grows — Sun, 25 Oct 2026 17:22:03 +0000

If ChatGPT, Claude or Perplexity never mention your product, the reason is often boring and fixable: the assistant cannot read your homepage in the first place. Before you think about content strategy for AI search, check three things on the technical side. Can AI crawlers fetch the page? Does the page contain real text without JavaScript? Does it say clearly what the product is?

I run website.show, a small directory of software built by indie makers, and every listing there gets an automated check of exactly these signals. After reading a lot of these reports, the same handful of mistakes keeps showing up. Here they are, with the fix for each.

Which AI crawlers should you allow in robots.txt?

Short answer: allow the search bots if you want to be quoted in AI answers. Decide separately about the training bots.

There are two families of AI crawlers, and they do different jobs:

Purpose	OpenAI	Anthropic	Perplexity / Google
Fetch pages for answers	OAI-SearchBot	Claude-SearchBot	PerplexityBot
Collect pages for training	GPTBot	ClaudeBot	Google-Extended

Blocking a search bot can keep your pages out of that assistant's answers. Blocking a training bot keeps new pages out of future models and leaves the answers alone. Google says Google-Extended does not affect Google Search, and Googlebot itself should never be on a block list: it is how pages reach Google Search, AI Overviews included.

If you are not sure what your robots.txt actually does, paste your domain into an AI crawler checker. It reads only robots.txt and shows which bots get in.

The robots.txt group trap most people fall into

This one surprised me the first time. Under RFC 9309, the robots.txt standard, a crawler follows only the group that names it. It falls back to User-agent: * only when no group names it.

So this looks harmless:

User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: OAI-SearchBot
Allow: /

But now OAI-SearchBot ignores the * group completely, which means /admin/ and /api/ are open to it. You wanted to welcome one bot and you exposed your private paths to it instead.

The safer pattern: let the bots you allow fall through to *, and give a group only to the bots you block:

User-agent: *
Disallow: /admin/
Disallow: /api/

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

That file keeps AI search bots able to read the site and keeps the training bots out. If you would rather not write it by hand, a robots.txt generator for AI crawlers builds this structure for you and runs in the browser, so nothing you type is sent anywhere.

Two more rules from the standard worth knowing: when an Allow and a Disallow both match, the longer rule wins and Allow wins a tie. A missing robots.txt means every crawler may read the site, while a server error on robots.txt makes crawlers treat the whole site as closed. That last one bites people with misconfigured hosting.

Does your page have text without JavaScript?

Short answer: if your homepage builds its text on the client, some crawlers see an empty page.

Not every crawler runs JavaScript. A single-page app that ships a <div id="root"></div> and fills it later can look blank to them. Quick test:

curl -s https://yoursite.com | grep -i "your main headline"

If the headline is not in the raw HTML, server-render or pre-render at least the homepage. For most frameworks that is a config flag, not a rewrite.

robots.txt is not the whole story

Even with a clean robots.txt, a bot can be turned away earlier:

CDN or firewall rules. Bot protection can block crawlers before they ever reach the file.
User-triggered fetches. OpenAI says robots.txt rules may not apply to ChatGPT-User, and Perplexity says Perplexity-User generally ignores them. These are fetches a person asks for in a chat, and they behave differently from the crawlers.

The guide Is your site blocking ChatGPT? goes through each case with links to the vendors' own documentation.

Should you add llms.txt?

Short answer: it is cheap, so yes, but do not expect miracles.

llms.txt is a proposed file at the root of a site that gives AI assistants a short Markdown map: a title, a one-line summary, then links to the pages worth reading. It is a proposal, not a standard, and it does not grant or refuse access. robots.txt still decides that.

A minimal one:

# Your Product

> One sentence that says what the product does and for whom.

## Docs
- [Getting started](https://yoursite.com/docs/start): install and first run
- [Pricing](https://yoursite.com/pricing): plans and limits

You can write it by hand in five minutes, or start from a draft made from your homepage with an llms.txt generator and then trim it. More background is in What llms.txt is.

Say what the product is, in a way a machine can parse

Assistants answer questions like "what is a good tool for X". If your homepage hero says "Build the future of work" and nothing else, there is nothing to quote. Two fixes:

One plain sentence near the top: what it is, who it is for.
Structured data. A SoftwareApplication JSON-LD block with name, category, price and URL gives crawlers the facts in a predictable shape. There is a walkthrough in this SoftwareApplication schema guide.

A five-minute checklist

robots.txt: search bots allowed, training bots by your choice, no accidental Allow groups.
Homepage text present in raw HTML.
No CDN rule blocking known AI bots.
One clear sentence about what the product does.
SoftwareApplication structured data.
llms.txt with your key pages.
The usual search basics: indexable page, title, meta description, H1, canonical, sitemap, link previews, image alt text.

If you want all of that in one pass, the free site check for AI search runs fifteen checks (seven for AI assistants, eight for search engines) and gives a score out of 100 with what to fix first. No AI model is involved in the check itself, so the same page always gets the same result. It does not tell you whether an assistant actually mentions you; it tells you whether it can read and understand you, which has to come first.

FAQ

Will blocking GPTBot remove me from ChatGPT answers?
No. GPTBot collects training data. Answers in ChatGPT search rely on OAI-SearchBot. Block GPTBot if you do not want to be in training data, and keep OAI-SearchBot allowed if you want to be quoted.

Does Google-Extended affect my Google rankings?
Google says it does not affect Google Search.

Is llms.txt required?
No. It is a proposal. It helps assistants find your important pages, but robots.txt and readable HTML matter more.

My robots.txt is fine but nothing changes. Why?
Check your CDN or firewall bot rules and whether the homepage text exists without JavaScript. Those two cover most of the remaining cases.

I build website.show in the open. If you shipped something recently, you can list it there for free and its page will run these checks every week: https://www.website.show

DEV Community: Ed Grows