A page that ranks in Google is not necessarily readable by GPTBot or ClaudeBot. Your robots.txt can allow one crawler and reject another, and JavaScript-only copy can leave a fetcher staring at an empty shell.
That means an assistant can name a competitor with a worse product simply because it has more retrievable evidence that the competitor exists.
Product quality is not a retrieval signal
When someone asks an assistant for the best tool in a category, the assistant has to construct a shortlist from material it can retrieve: pages it managed to fetch and public text where people discuss that category by name.
Your product quality is real, but it is not directly available as an input. If nobody has written about your product in the places the system can retrieve, and your own pages are inaccessible, there is little for it to assemble into an answer.
Competitors that appear in both places become the safe default. Their own sites explain what they do. Other people mention them in discussions, comparisons, comments, and links. That record gets repeated because it is already there.
AI answers behave like consensus with a lag. If public discussions repeatedly say, "we ended up using X," and none mention your product, you should expect X to appear in category answers more often. Watching an AI visibility score does not alter that underlying record. It is a thermometer, not treatment.
robots.txt can exclude you without affecting Google
A site can work perfectly in ordinary search while remaining inaccessible to named AI crawlers. Those crawlers use separate user-agents and can face separate rules in robots.txt.
This is easy to miss because a rule may have arrived through a template, a hosting configuration, or a change someone made for a reason nobody documented. A founder sees search traffic, assumes the site is available everywhere, and never checks the agents that matter for assistant retrieval.
The practical audit is not complicated: ask whether each named crawler is allowed to fetch the domain, then inspect what it receives.
| Source an answer may draw on | What it needs | What can block it |
|---|---|---|
| Your own pages | HTML a named crawler can fetch |
robots.txt disallow rules, JavaScript-only rendering |
| Public threads on Reddit, LinkedIn, X, Threads, DEV | Your product named in a useful reply | Never appearing where the question is asked |
| Third-party pages and links | Someone describing your product in their own words | No outreach, nothing worth citing |
| Paid ads | Budget | Ad impressions are not text an engine retrieves |
The table is deliberately unglamorous. Most visibility problems are. Before publishing another page, verify that the existing pages can be fetched by the crawlers you expect to find them.
JavaScript rendering can turn your marketing page into a div
Crawler access is only half the question. The other half is whether the useful explanation exists in the response a crawler receives.
Many sites ship an HTML shell and paint their copy after JavaScript runs. A human browser sees the product description. A fetcher that does not run those scripts sees a mostly empty document.
The same failure mode appears when the actual explanation is behind a login, a demo gate, or a chat widget. The site may look complete to someone clicking around, while the retrievable page contains almost none of the category language a buyer would search for.
This is why a crawler audit should inspect actual HTML, not just page screenshots or a green deployment check. The question is not whether the page looks right in your browser. The question is whether a named agent can fetch a page containing the answer.
Public replies still matter because a buyer is reading them now
There is a temptation to treat thread replies as obsolete whenever citation behavior changes. That misses the immediate audience: the person who asked the question is in the thread today, looking for an answer.
A helpful reply also remains public for later searchers and can be retrieved by more than one system. Reddit is one field, not the field. The same buying questions show up on LinkedIn, X, Threads, and DEV, often with different norms and different levels of tolerance for promotion.
The useful unit of work is not "post on Reddit." It is: find the live question your buyer asks, answer it in the place where they asked it, and make sure your own site has a durable version of that answer too.
A good reply survives after removing the product line. It should answer the question, name the tradeoff you would make in the reader's position, and mention your product once only if it genuinely belongs in the answer.
If deleting the product reference makes the reply useless, it was an ad. Readers notice. Moderation systems and account-standing filters often notice too.
Paid reach cannot repair missing retrievable text
Paid ads may help once you know an organic message converts, but ad impressions are not text an engine retrieves when assembling a category answer.
That makes ads a poor first fix for the problem of never being named. Spending can buy attention, but it does not create a fetchable product explanation, a public answer to a buying question, or a third-party mention in someone else's words.
The first moves are less exciting and more durable:
- Check what
robots.txtdoes to GPTBot, ClaudeBot, and other relevant agents. - Confirm that the page's actual answer exists in HTML rather than appearing only after a script runs.
- List the five questions a buyer asks immediately before choosing a tool in your category.
- Find live threads asking those questions.
- Answer them properly, then publish the durable version on your own domain.
This does not mean a crawler allow-list guarantees a recommendation
Allowing a crawler to fetch your site is necessary for this problem, but it is not sufficient to solve it. Fixing robots.txt can happen the same day. Changing the public record that an assistant may retrieve accumulates over time.
There is no fixed clock for when an engine starts naming a product, because retrieval changes when the underlying record changes. It depends on how fetchable the pages are and how often the product appears in public answers people actually read.
Some publishers may intentionally block AI crawlers to protect their content. That is a valid trade if being included in assistant answers is not a goal. For a founder who wants assistants to describe their category and name their product, the same rule is a decision to stay out of the answer.
Have you checked whether GPTBot or ClaudeBot can fetch your production HTML, and did the result match what you expected?
Top comments (0)