DEV Community: Ke Gao

Your robots.txt might be hiding you from AI answers - a 5-minute check

Ke Gao — Tue, 10 Nov 2026 05:20:39 +0000

Every audit I run starts the same way, and it is not with content. It is with robots.txt.

The pattern: someone has spent six months publishing comparison pages and wondering why ChatGPT never mentions them, and somewhere in the site config a Disallow: / rule was copied from a staging template, or a security plugin added User-agent: * with a blocklist, or a developer blocked "AI scrapers" in 2023 and never revisited it. Two years of content, invisible to the engines.

Here is the check, and the nuance people get wrong.

1. Which bots matter, and what each one actually does

These are not interchangeable. Blocking one does not block the others:

Bot	Used for	Blocking it costs you
`GPTBot`	OpenAI's crawling for model training	Training signal, but not necessarily ChatGPT search visibility
`OAI-SearchBot`	Building the ChatGPT search index	Appearing in ChatGPT search answers
`ChatGPT-User`	User-initiated fetch ("open this URL")	ChatGPT being able to read your page on demand
`ClaudeBot`	Anthropic's crawler	Claude understanding/citing your pages
`PerplexityBot`	Perplexity's index	Perplexity citations
`Google-Extended`	Gemini / Vertex training controls	Gemini training signal; does not affect Search ranking
`Bingbot`	Bing index — which Copilot runs on	Copilot visibility entirely

The one people get wrong most often: they block GPTBot to "opt out of training", feel principled, and don't realise OAI-SearchBot is a separate switch that governs whether they show up in ChatGPT search. Opting out of training is a legitimate choice. Opting out of being cited is usually an accident.

2. Read your own file

curl -s https://yourdomain.com/robots.txt

Then look for four things:

A blanket Disallow: / under User-agent: * — kills everything below it unless a specific agent overrides it.
Named agent groups with Disallow: / (User-agent: GPTBot followed by Disallow: / is the classic "we blocked AI in 2023" leftover).
Whole-site blocks hidden in a plugin's "security" tab, which you won't see in the file's first ten lines.
A Sitemap: line that points to a sitemap returning 404 — common after a domain or framework migration.

3. The minimum viable allowlist

If you want to be readable by assistants and search engines, you need explicit allows (specific agent groups are matched before *, so being explicit is safer than relying on the wildcard):

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Keep the training-crawler decision (GPTBot, CCBot) as a separate, deliberate choice. Just make it deliberately, not by inheritance.

4. Check the engine that powers the answer

Copilot runs on the Bing index, and a surprising number of sites are simply not in it. Verify with:

https://www.bing.com/search?q=site:yourdomain.com

Zero results means no amount of content work will help that slice of users. Submit the sitemap in Bing Webmaster Tools and wait; it is usually days, not weeks.

5. Then, and only then, work on content

Order matters. Crawler access → indexing → entity clarity → content → earned mentions. If you skip to step five, you are optimising pages nobody is allowed to read.

This is the boring part of AI visibility, and it is also the part that produces the biggest jumps, because it is binary: either the crawler gets a 200 and reads the page, or nothing else you do matters.

If you want to measure the result afterwards, I wrote about auditing whether AI mentions you at all and about the six kinds of pages models actually cite. The check above is what I run before either of them, and it takes five minutes.

(Disclosure: I build bidvix, which measures mention rates across five models — but crawler access is free to fix and I'd do that first either way.)

The six kinds of pages LLMs actually cite (and how to find yours in an afternoon)

Ke Gao — Tue, 10 Nov 2026 05:09:13 +0000

Last time I wrote about auditing whether AI mentions your brand. The part people ask about is always the same: which pages get cited, and how do I get on them?

Here's the short version: run your buyer prompts, collect every URL the answer cites, and cluster them. In every audit I've done, the citations fall into six buckets. Your job is to find the buckets where you're absent but reachable.

Step 1: collect the citations (30 minutes)

Twelve prompts, four families: category, problem, comparison, brand. Run them in ChatGPT (with search), Perplexity and Gemini, logged out. Copy every source URL each answer cites into one column. Don't judge yet, just collect — 60 to 100 rows is normal.

Step 2: cluster into six buckets

Every URL will land in one of these:

Bucket	What it looks like	How you get in
Directories	"Top 40 AI tools", niche tool indexes	Free/paid submission forms
Listicles & comparisons	"Best X for Y in 2026", "A vs B"	Pitch the author, or publish your own comparison page
Docs & changelogs	Product docs, release notes, pricing pages	Yours to write — make one page answer one question
Community threads	Reddit, Quora, HN, Indie Hackers, forums	Answer real questions, disclose you're the maker
News & editorial	Trade press, launch coverage	Harder; usually needs a story, not a product
Code & lists	GitHub repos, awesome-lists, package docs	A PR, if your tool is developer-adjacent

Step 3: read the distribution, not the individual URLs

Count rows per bucket. Two patterns come up constantly:

Directories + listicles dominate. That means the model isn't reading your site, it's reading the pages about your category. Ten good listings beat one beautiful landing page.
Community threads show up more than people expect, especially for "what tool should I use" style questions. One useful Quora answer can be pulled into answers for months.

Now compare with where you actually exist. The gap list is your roadmap — and it's usually shorter than it feels.

Step 4: fix the boring blockers first

Before any of this works, two things are often quietly broken:

Crawler access. Check robots.txt for GPTBot, ClaudeBot, PerplexityBot, Google-Extended. Plenty of sites block them by accident via a copied template. Two years of content behind a crawler block is a common, embarrassing finding.
Indexing in the engine that powers the answer. Copilot runs on Bing. If you have no Bing index, you don't exist for a slice of users regardless of content quality.

Step 5: pick three buckets, not six

The failure mode is trying all six at once. I'd pick: one directory pass (10 submissions), one community pass (3 thoughtful answers, disclosure included), and one content pass (5 answer-shaped pages — one question per page, answer in the first 40 words).

Then re-run the same twelve prompts two to four weeks later and count the same buckets. If the distribution shifted toward sources you control, it's working. If it didn't, you were probably working on the wrong bucket — which is exactly the thing measurement is for.

If you'd rather not build the spreadsheet, I made bidvix for the measurement half (mention rate + cited URLs across five models). The clustering above is the part that actually changes what you do, and it works fine in a notebook.

Also in this series: The 20-minute AI visibility audit · Your robots.txt might be hiding you from AI answers

Your product doesn't exist to ChatGPT — a 20-minute AI visibility audit you can run yourself

Ke Gao — Tue, 10 Nov 2026 00:46:09 +0000

Ask ChatGPT "what's the best tool for X" and you'll get three or four brand names. If yours isn't one of them, you have a problem Google Analytics will never show you: the user never reached your site, so there's no traffic to attribute.

I've been chasing this for my own tool. Here's the checklist I use, the data I got, and the fixes that actually moved the needle.

Why LLMs ignore your product

LLMs don't "browse" a category. They answer from:

Parametric memory — whatever was in the training data. You can't change this directly.
Retrieval — the pages that get pulled in for a query (docs, listicles, directories, Reddit threads, comparison pages).
Entity consistency — does the model have a stable idea of what you are? One-sentence description, same everywhere, with a category.

Most indie products fail on #2 and #3, not #1. The model doesn't know you because nothing on the open web says the same, boring sentence about you ten times.

The audit (20 minutes)

Pick 4 prompt families, and 3 variants each — 12 prompts total:

Family	Example
Category	"best AI visibility tools", "tools to check if ChatGPT mentions my brand"
Problem	"how do I know if AI search engines recommend my product"
Comparison	"brand monitoring vs AI visibility trackers"
Brand	"what is bidvix" (baseline: does the model know you at all?)

Run each prompt 3 times in ChatGPT, Gemini, Claude, Perplexity, and DeepSeek. Score:

Mention rate — how often you appear (out of runs).
Position — first three brands mentioned, or buried in a list.
Sentiment — recommended, neutral, or described wrongly.
Cited sources — which URLs the model actually read. This is the gold. Those pages are your real competitors for attention.

Track it in a sheet. Repeat weekly. A single check is noise; a trend is a signal.

What my own audit showed

For bidvix, brand prompts were fine — the model knew the name. Category prompts were the problem: my tool was absent from "best AI visibility tools" answers, while the cited sources were almost entirely directory listings and listicles, not vendor sites.

Two conclusions that changed my plan:

Being findable on Google is not the same as being quotable by a model. Directories and comparison pages get retrieved; your beautiful landing page often doesn't.
Consistency beats cleverness. When 6 sources describe you with 6 different categories, the model hedges and leaves you out.

Fixes, in order of effort-to-impact

One canonical sentence. Write it once: X is a [category] that [does the thing] for [who]. Use it verbatim on your site, your profiles, and every directory submission. Don't get creative — get repetitive.
Answer pages, not blog posts. One page per question your users actually ask ("does ChatGPT mention my brand", "how to track AI search visibility"). Lead with a 40-word answer, then details. These are the pages retrieval likes.
Get into the sources that get cited. Directories, "best of" lists, comparison pages, community threads. Not 60 of them — 10 that the models already read for your prompts.
Keep a public changelog / docs page that states what the product does and doesn't do. Models respect hedged, factual pages far more than marketing copy.
Measure again after 2–4 weeks. Retrieval-heavy models (Perplexity, ChatGPT with search) move fast; parametric answers move slowly.

The uncomfortable part

You can't A/B test your way into this. There's no dashboard that tells you "you were cited 4 times today" unless you build or buy one — which is, admittedly, why I built mine. If you want to skip the spreadsheet, bidvix runs the same 12-prompt check across five models and gives you a mention rate plus the cited URLs.

But the audit above works with a notebook and 20 minutes. Do that first — the numbers will tell you whether you have a content problem or an entity problem, and those have very different fixes.

If you run it, I'd genuinely like to see your results — especially the cited-URLs column. That list is usually a shock.

Next in this series: The six kinds of pages LLMs actually cite · Your robots.txt might be hiding you from AI answers