DEV Community: Suparank

Can AI Assistants Actually Read Your Website? A 15-Minute Access Check

Suparank — Wed, 04 Nov 2026 06:29:46 +0000

When a buyer asks ChatGPT, Claude, Gemini or Google's AI Overviews which tool to use, the assistant can only describe your product from what it was able to fetch, or from what other sites say about you. Plenty of teams spend weeks on "AI SEO" content and never check the basics underneath. Is the site blocked, does the content only appear after JavaScript runs, and is there one clear page that explains what the product does?

This is a 15-minute check you can run on your own site with nothing more than a browser. None of it needs a paid tool.

1. Read your robots.txt like an AI crawler would

Open https://yourdomain.com/robots.txt and look for rules aimed at AI user agents. The ones you will most often see are:

OpenAI: GPTBot (model training), OAI-SearchBot (ChatGPT search results) and ChatGPT-User (pages fetched when a user asks).
Anthropic: ClaudeBot, plus the search and user-fetch agents Anthropic documents.
Perplexity: PerplexityBot.
Google: Google-Extended is a control token, not a separate crawler. AI Overviews and AI Mode are built on regular Google Search crawling, so a Disallow for Googlebot hurts you everywhere.
Microsoft: Copilot answers lean on the Bing index, so Bingbot access matters.

Vendors rename and add agents, so check each one's current documentation before you edit anything.

Three patterns cause most of the accidental blocks:

A blanket User-agent: * / Disallow: / left over from a staging site.
A security plugin or CDN rule that added AI bots to a block list, sometimes switched on by default. In this case robots.txt looks fine, but the firewall returns 403.
Blocking the search agent when you meant to block training. Some teams want to opt out of training but still be cited in answers. Those are different user agents with different rules.

Decide on purpose. If you want to be named in AI answers, the agents that fetch pages for answers need access to your public marketing pages.

2. Test what a non-browser client receives

Robots rules are only a request. What matters is what the server actually returns. From a terminal:

curl -s -o /dev/null -w "%{http_code}\n" -A "GPTBot" https://yourdomain.com/

Then run it with a normal browser user agent and compare the two. A 403, a 429 or a challenge page for the bot means a firewall rule is blocking it, whatever robots.txt says.

Then check the page body itself. Run curl -s https://yourdomain.com/pricing/ | grep -i "per month", or grep for any phrase you know is on the page. If the text only shows up after client-side JavaScript runs, many fetchers will see an empty shell. Your pricing, feature list and "what is it" paragraph should be in the HTML the server returns.

3. Make sure there is a sitemap and that it is current

Robots.txt should point to it (Sitemap: https://yourdomain.com/sitemap.xml). Open the sitemap and spot-check that:

the key pages are there (home, product, pricing, docs or guides),
URLs return 200, not redirects or 404s,
staging, tag or search-result URLs are not padding it out.

4. Give machines a short, honest summary: llms.txt

llms.txt is a proposed plain-markdown file at your root. It gives a one-paragraph description of the site and links to the pages that matter most. It is not a ranking switch and not every assistant reads it, but it costs ten minutes and forces you to write the clearest possible description of your product. That description is useful everywhere else too.

Keep it factual. Say what the product is, who it is for and what it costs, then link your best explainer pages. Don't stuff it with keywords.

5. Check that your key facts are extractable

Assistants quote sentences, not layouts. Open your homepage and pricing page and ask:

Is there one plain sentence that says what the product is? ("X is a ... for ...")
Are prices written as text, not only inside images or toggles?
Do headings match the questions buyers ask ("How much does X cost?", "Does X integrate with Y?")?
Is there structured data (Organization, Product or SoftwareApplication, FAQPage where it fits) that matches the visible text?
Do canonical tags point at the right URL, so duplicate versions don't split signals?

If an assistant can't find a fact in one or two clean sentences, it will either skip you or fill the gap with whatever a third-party page says.

6. Write down what you found, then re-check after changes

Make a short table with five columns: check, status, page, fix and owner. Most sites end up with two or three real issues, usually a firewall rule, a JavaScript-only pricing table or a missing sitemap entry. Fix those first. Wording tweaks can wait.

Doing it faster

If you'd rather not run each step by hand, Suparank has a free agent readiness checker. It needs no sign-up and scores a site from 0 to 100 against about 20 AI-agent standards, covering robots rules, sitemaps and the well-known files agents read. For a full crawl, the GEO and technical audit in Suparank runs 63 checks in 11 categories, including AI crawler access and llms.txt, and ranks the fixes by impact.

Either way, fix access first. Content work only pays off once assistants can actually read your pages.

AI crawler access control — robots.txt, user-agents and the trade-off

Suparank — Wed, 16 Sep 2026 21:56:58 +0000

Deciding who may crawl you is now two separate decisions wearing one name, and confusing them is the most common self-inflicted visibility problem on the web.

Published by Suparank · 2026 · suparank.io

A few years ago, crawler policy was a settled subject: let search engines in, keep bad bots out, move on. AI agents broke that because they are not one kind of visitor. Some collect text to train models. Some fetch pages to build the index behind an AI search product. Some fetch a single page because a user asked an assistant to read it. These have genuinely different implications, and a robots.txt written as though they were the same thing will produce an outcome you did not intend. This reference sets out what is publicly documented, how to express it, and where the silent failures live. It is published by Suparank, which audits this surface as part of its GEO work.

The distinction that decides everything

Three purposes, three different consequences when you block them:

Training collection. Text is gathered to be used in model training. Blocking this is a content-licensing decision. It has no immediate effect on whether you appear in answers today, and its long-run effect on how well a model "knows" your brand is real but slow and unmeasurable.
Live retrieval / index building. Pages are fetched to populate the search index an AI product consults while answering. Blocking this removes you from the cited-source answers you are presumably trying to win. The effect is direct, and it arrives quickly.
User-initiated fetch. A specific page is fetched because a user pointed an assistant at it. Blocking this means that when a prospect asks an assistant to read your own page, the assistant cannot — a strange thing to prevent.

Almost every damaging configuration comes from wanting the first and accidentally getting all three: a blanket rule, a security vendor's "block AI bots" toggle, or a copied snippet from an article that predates the split.

Documented agents and what blocking each costs

The table below lists only agents whose purpose is publicly documented by their operator. Crawler documentation changes, so treat this as a starting point and confirm each string against the operator's own current documentation before you deploy a rule. If you encounter a user-agent string in your logs that you cannot find documented anywhere, do not assume its purpose from its name.

User-agent	Operator	Purpose	Blocking it costs you
`GPTBot`	OpenAI	Collects web content for model training	Long-run, unmeasurable influence on what the model knows about you; no immediate loss of answer presence
`OAI-SearchBot`	OpenAI	Builds the index behind ChatGPT's search surface	Direct: your pages become unavailable as sources for ChatGPT's search-backed answers
`ChatGPT-User`	OpenAI	Fetches a page in response to a user's explicit request	A user who asks ChatGPT to read your page gets nothing
`ClaudeBot`	Anthropic	Web crawling for Anthropic's models	Exclusion from Anthropic's crawled corpus
`Claude-User`	Anthropic	Fetches a page on a user's direct request	Claude users cannot have your pages read to them
`PerplexityBot`	Perplexity	Indexing for Perplexity's search and answers	Direct: removal from Perplexity's citable index
`Google-Extended`	Google	Not a crawler — a robots.txt token controlling use of already-crawled content in certain Google AI products	Content-usage restriction only; Googlebot crawling and ordinary Search are unaffected
`Googlebot`	Google	Core Search crawling, which also feeds AI Overviews	Everything: ordinary Search and AI Overviews alike. Never block this as an AI-policy measure

Google-Extended is the one that confuses people

It is worth restating on its own, because it is regularly implemented wrongly. Google-Extended does not fetch anything and has no separate crawl. It is a usage control expressed in robots.txt syntax. Disallowing it does not remove you from Google Search, and it does not make you invisible to Googlebot. Conversely, blocking Googlebot in an attempt to opt out of AI features destroys your ordinary search presence and does not achieve the intended goal. These are different levers with different blast radii.

Syntax

Rules are per-agent groups, matched by the most specific applicable group — an agent that has its own group ignores the wildcard group entirely. That behaviour is the source of most surprises. A configuration that permits retrieval and user fetches while declining training collection looks like this:

User-agent: GPTBot Disallow: /

User-agent: OAI-SearchBot Allow: /

User-agent: ChatGPT-User Allow: /

User-agent: * Disallow: /admin/

Because GPTBot has its own group, it never reads the wildcard group. Every agent you name needs its complete instructions inside its own block.

Two further points. First, robots.txt is a request, not an enforcement mechanism; it is honoured by operators who choose to honour it and ignored by everyone else, so it is a policy statement, not a security control. Second, the file must be served at the root of each host and scheme you care about — subdomains have their own robots.txt, and a rule on the apex domain does not cover them. If you would rather not hand-write these groups, the AI crawler robots.txt generator produces per-agent blocks with the training and retrieval distinction made explicit.

Server-side rules: the invisible failure

This is the part that costs organisations the most and is diagnosed the least, because nothing in robots.txt reveals it. Your edge can be refusing these agents while your published policy welcomes them.

WAF bot-management rules. Managed rulesets often classify unfamiliar automated clients as bots and challenge or block them by default. An AI retrieval agent receiving a JavaScript challenge page fails, and records nothing useful about your site.
CDN and platform toggles. Several providers ship one-click "block AI crawlers" controls. They are legitimate, but they are usually blunt: they catch the retrieval agents alongside the training ones. If someone enabled one during a scraping scare, your visibility problem may date from that day.
Rate limiting. Aggressive per-IP limits can throttle a legitimate crawler into partial coverage. You get indexed inconsistently and it looks like a content problem.
Geographic and ASN blocking. Blanket country or cloud-provider blocks catch datacentre-hosted crawlers indiscriminately.
Consent walls and interstitials. If the substantive content only appears after an interaction, what the agent stores is your consent notice.

Verifying that you are actually reachable

Policy is what you wrote; reachability is what happens. Check the second.

Fetch your own pages with each agent's user-agent string and inspect the status code and the returned body. A 200 that contains a challenge page is a failure dressed as a success.
Compare served HTML with rendered output. If your main content only exists after client-side hydration, an agent that does not execute scripts sees an empty shell. Disable JavaScript and read the page.
Test every hostname and both schemes, including www and non-www, plus any subdomain carrying documentation or product content.
Read your access logs for the documented agents. Absence of any hits over a long period is evidence of blocking, not of disinterest.
Ask an assistant to read one of your URLs directly. A refusal or a summary that describes your cookie banner tells you immediately that something upstream is interfering.

The agent readiness checker runs several of these checks against a live URL, and where the cause turns out to be structural rather than a single rule, a full GEO and technical audit covers rendering, structure and access together.

Default position for most commercial sites: allow retrieval and user-initiated fetch agents everywhere, decide on training collection deliberately as a content-licensing question, never block Googlebot as an AI measure, and audit the edge before concluding that your robots.txt is the whole story.

Making the trade-off

There is a real tension for publishers whose product is the text itself: training collection offers them nothing and may substitute for their traffic. For most software and service businesses the calculation is different, because being described accurately in an AI answer is a distribution channel, not a leak. Either way, the decision should be made once, written down, and expressed as separate rules per purpose — not inherited from a snippet someone pasted in 2024. Review it when operators publish new agents, because they do, and a stale file quietly makes the decision for you. More on the measurement side of this work is at suparank.io.

Suparank is a chat-first AI visibility and GEO platform. It runs your buyer prompts through ChatGPT, Gemini, Claude and Google AI Overviews and reports mentions, citations, sentiment and share of voice against named competitors, alongside GEO and technical audits and connected GA4 and Search Console data. It does not do rank tracking. Free tier available, no credit card required. https://suparank.io/ · marketing@suparank.io · Chase Allen, SEO/GEO Specialist.

Originally published by Suparank.