DEV Community: Visibility100x

How to rank in Google AI Overviews when ranking first no longer gets you in

Visibility100x — Wed, 12 Aug 2026 23:26:46 +0000

You do not get into an AI Overview by ranking first any more. In the most recent large study of the question, 38% of the pages cited in AI Overviews ranked in the top 10 for that same query, against 76% seven months before. The rest were quoted from page two, page ten, or from nowhere in the top 100. That single change is what makes this a different job from the one it looked like last year.

Where cited pages actually rank
Ahrefs published the numbers on 2 March 2026, from 863,000 keyword SERPs and 4 million AI Overview URLs collected in January 2026.

Where the cited page ranks for that query Share of citations
Top 10 37.9%
Positions 11 to 100 31.2%
Not in the top 100 31.0%
Filtered to plain organic listings only, with ads and snippet features excluded, the picture gets slightly worse for the received wisdom: 37.1% in the top 10 and 36.7% outside the top 100.

One caveat belongs here rather than in a footnote, and Ahrefs raise it themselves. They improved how their parser detects citations between the July 2025 run and this one. Some part of a fall from 76% to 38% is therefore better detection rather than changed behaviour, and anyone quoting the drop as a pure measurement of Google is overreaching. The direction survives the caveat. The precision does not.

Query fan out is the mechanism
The reason a page that ranks nowhere gets quoted is that Google is not answering your question. It is answering several questions it wrote itself.

Given one query, Google generates a set of narrower related searches, runs them concurrently against its index, and composes the overview from what comes back. Your query was the prompt. The searches that actually retrieved the sources were never shown to you. A page can therefore be cited on a query it has never ranked for, because it ranks for a sub question inside it.

Work an example. Somebody searches for the best project management tool for small teams. The fan out underneath is closer to a list: which tools have free tiers, what do teams under ten people complain about, how do the pricing models compare per seat, which integrate with the tools they already run. A comparison page optimised for the head phrase competes with everybody. A support page that answers the fourth question precisely, on a domain nobody would call an authority on project management, wins that retrieval outright.

This is why the old instinct to consolidate everything into one long definitive page now works against you in this specific channel. Fan out rewards a surface that answers many adjacent narrow questions well. It does not reward one page that answers the broad one beautifully.

If you want to see the fan out for your own topic before you write anything, the query fan out prompt reconstructs a plausible set and marks which of them you currently answer.

What Google says, which is less than people claim
It is worth reading the official position, because a large amount of advice sold in this category contradicts it.

Google’s documentation on AI features states that there are no additional requirements to appear in AI Overviews or AI Mode and no other special optimisations necessary. It goes further on the point people most want to be wrong: you do not need to create new machine readable files, AI text files, or markup, and there is no special schema.org structured data that you need to add.

Take that at face value and two things follow. There is no secret to buy. And the eligibility floor is ordinary: the page has to be indexed and allowed to show a snippet. A noindex, a nosnippet, an over aggressive max-snippet, a data-nosnippet wrapped around the part worth quoting, or a robots rule blocking the crawl, and no amount of content work matters. Those are the first things to check and they are quick: the indexability checker and the robots.txt checker between them cover the whole floor.

Believe Google on what is not required. Do not read it as a claim that nothing you do makes a difference, because the citation data plainly shows some pages getting quoted far more than their rankings explain.

The four things that actually move it
Be legible before you are clever
An assistant that cannot parse your page cannot quote it. If the content only exists after JavaScript runs, a retrieval pass may see an empty document, which is the failure we find most often and the one nobody notices because the page looks fine in a browser. Check it with the render gap checker, and run the whole page through the AI content readiness checker to see whether a clean passage can be lifted out of it at all.

Write the answer as a liftable unit
The shape that gets quoted is a self contained passage of forty to sixty words that answers one question without depending on the paragraph above it. That is not a style preference. A retrieval system is choosing a chunk, and a chunk that needs three sentences of context to make sense is a worse candidate than one that does not. Sub headings phrased as the actual question, answered immediately underneath, are the cheapest structural change available.

Cover the sub questions, not just the topic
Once you accept fan out, the content plan changes shape. Instead of one page targeting the head term, you want genuine coverage of the adjacent narrow questions, each one answerable in a paragraph, each with its own heading. A frequently asked questions block is the crudest version of this and it works. A set of properly separated pages is the better version, provided they are actually different pages and not the same page with the noun swapped.

Be somewhere other than your own site
The single most useful detail in the Ahrefs data is buried in the tail. Of the citations that came from outside the top 100, 18.2% were YouTube URLs, and YouTube accounted for 5.6% of every AI Overview citation in the sample. Fan out retrieves across source types, so a brand with a decent presence across a video, a forum thread, a documentation page and a review platform is quoted more than a brand with one excellent article and nothing else. Our own reading of the wider evidence, written up in where AI answers really come from, points the same way.

What the click is worth now, stated honestly
Three studies, three methods, one direction.

Pew Research Center installed a browsing tracker on more than 900 consenting adults and recorded 68,879 real Google queries in March 2025. Users clicked a traditional result on 8% of visits where an AI summary appeared, against 15% where none did. They clicked a link inside the summary itself on 1% of visits.

Seer Interactive analysed 3,119 informational queries across 42 organisations, covering 25.1 million organic impressions from June 2024 to September 2025, and published on 4 November 2025. Organic click through rate on queries with an AI Overview fell from 1.76% to 0.61%.

And the number that should temper the panic, from the same Seer dataset: queries with no AI Overview at all fell 41% year over year. Something larger than this one feature is moving, and attributing every lost click to AI Overviews is the kind of reasoning that produces a bad strategy.

Against that, the finding that justifies the work: Seer measured that being cited in the overview was associated with a 35% higher organic click through rate than not being cited on the same query. The pie is smaller. Being named is how you keep a slice of it.

How often does this even happen to you
Published prevalence figures for the first half of 2026 range from roughly a quarter of queries to roughly 60%, depending on whose keyword set, which country, which devices and how the tracker detects the feature. The spread is the finding. Averages assembled from somebody else’s keyword list tell you nothing actionable about yours.

Your own Search Console data does. Look for queries where impressions are steady and clicks have fallen, and you have found where this is happening to you specifically, without needing anybody’s industry benchmark. The Search Console keyword analysis prompt does that split on an export.

What not to bother with
An llms.txt file, for this purpose. It has its arguments and Google’s stated position is explicit that no AI text file is required. We wrote up the wider evidence in llms.txt, and whether it works.

Blocking Google-Extended and expecting to stay in AI Overviews. That directive governs training and other Google AI products, and getting the robots directives confused here is a way to remove yourself from a surface you were trying to win. The AI crawler checker shows what each agent is currently allowed.

Any product promising placement. There is no placement to sell. There is eligibility, which is technical and checkable, and there is being the best available source for a question you never see, which is editorial.

Publishing twenty near identical pages to cover twenty sub questions. Fan out rewards coverage, and a template with the noun swapped is not coverage. It is the doorway pattern, it is a known penalty, and it is the most likely way this advice gets misapplied.

The order to do this in
Confirm eligibility. Indexable, crawlable, no snippet suppression on the part worth quoting. Nothing else matters until this passes.
Fix the render gap if there is one, because it invalidates everything downstream.
Reconstruct the fan out for your five most valuable queries and list the sub questions you do not currently answer.
Restructure your existing strong pages so each section answers one question in a liftable passage. This is editing, not writing, and it is the highest return hour on the list.
Only then write new pages, one per genuinely distinct question.
Establish presence on at least one surface you do not own.
A free audit covers steps one and two in a couple of minutes, which is the part that is mechanical. Steps three onwards are the actual work and no tool does them for you.

GPTBot, ClaudeBot, PerplexityBot: which AI crawlers can read your site

Visibility100x — Thu, 06 Aug 2026 08:07:29 +0000

If your robots.txt has no Disallow lines that apply to them, every AI crawler can already read your site. The ones that matter are GPTBot and OAI-SearchBot for ChatGPT, ClaudeBot for Claude, PerplexityBot for Perplexity, and the Google-Extended control token for Gemini and AI Overviews. Blocking any of them means that assistant cannot quote your pages. Everything below is the detail behind that sentence.

The full list, and what each one actually does
Checked 3 August 2026. Operators add and rename tokens without announcement, so treat this as a snapshot and re run the check on your own file rather than trusting any list, including this one.

GPTBot OpenAI · General crawling for OpenAI, including model training · Crawler
OAI-SearchBot OpenAI · The search index behind links shown in ChatGPT · Crawler
ChatGPT-User OpenAI · A single fetch when a user asks ChatGPT to open your page · User triggered
ClaudeBot Anthropic · Crawling for Claude · Crawler
Claude-User Anthropic · A fetch made on behalf of a person using Claude · User triggered
PerplexityBot Perplexity · The Perplexity index · Crawler
Perplexity-User Perplexity · A fetch triggered by a user question · User triggered
Google-Extended Google · Whether Gemini and AI Overviews may use what Googlebot fetched · Control token
Googlebot Google · Google Search, and therefore AI Overviews at all · Crawler
Bingbot Microsoft · Bing, which is the index behind Copilot · Crawler
Applebot Apple · Siri and Spotlight · Crawler
Applebot-Extended Apple · Whether Apple may use that content to train its models · Control token
CCBot Common Crawl · An open dataset used as training input by many labs · Crawler
meta-externalagent Meta · Meta AI training and products · Crawler
Amazonbot Amazon · Alexa and Amazon shopping assistants · Crawler
Bytespider ByteDance · TikTok search and Doubao · Crawler

Google-Extended and Applebot-Extended are not crawlers
Nothing ever arrives at your server calling itself Google-Extended. It is a token you put in robots.txt that answers a policy question: may the content Googlebot already fetched be used for Gemini and for grounding AI Overviews. Applebot-Extended works the same way for Apple.

This matters because it makes them safe to reason about. Disallowing Google-Extended costs you nothing in Google Search. Your rankings, your indexing and your crawl budget are untouched, because the fetch that already happened was Googlebot's. What you lose is presence in the answers Gemini generates and in AI Overviews. That is a real cost and it is a clean, separable one, which is more than can be said for most decisions in this area.

The reverse is worth stating too. If Googlebot is blocked, Google-Extended is irrelevant. A page nobody fetched cannot be used for anything.

Crawlers and user triggered fetchers are different questions
GPTBot crawls the web on OpenAI's schedule. ChatGPT-User fetches one page because a person in a chat asked for it. Anthropic and Perplexity draw the same line with Claude-User and Perplexity-User.

The distinction is the whole basis of the nuanced position many publishers want: refuse bulk crawling for training, allow the fetch that happens when a real person is reading you right now. That is expressible in robots.txt, and it is the configuration most businesses arrive at once they understand the two are separable.

User-agent: GPTBot
Disallow: /

User-agent: ChatGPT-User
Allow: /
The rule that silently breaks most robots.txt files
A crawler obeys exactly one group. It picks the group whose user agent token is the longest match for its own name, and it then ignores every other group in the file, including the wildcard.

That single rule is behind most of the accidental blocks we see. Consider this file:

User-agent: *
Disallow: /private/

User-agent: GPTBot
Disallow: /
Every crawler except GPTBot stays out of /private/. GPTBot is blocked from everything, and is also free to crawl /private/, because the wildcard group does not apply to it at all. Now consider the more common accident:

User-agent: GPTBot
Allow: /

User-agent: *
Disallow: /
The author intended to welcome GPTBot. What they actually published blocks every other crawler on the internet, Googlebot included.

Within the group that applies, the longest matching path wins, not the first line in the file. Allow: /blog/ beats Disallow: / for a URL under /blog/. Our checker implements both rules, group selection by longest matching token and then longest match path evaluation, which is why it sometimes disagrees with a quick read of the file.

The two configurations worth copying
Allow everything, which is what a business that wants to be recommended should publish:

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
That is it. No per bot allow lines are needed, because permission is the default and every crawler falls into the wildcard group.

Refuse training, keep retrieval, which is the publisher position:

User-agent: GPTBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Bytespider
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: Applebot-Extended
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: *
Allow: /

Sitemap: https://example.com/sitemap.xml
Read that second file carefully before deploying it. It is a real editorial position, not a safe default, and it does cost you some AI visibility on purpose.

So should you block them?
There is an honest case on both sides and anyone who tells you otherwise is selling something.

Block, if your revenue depends on the visit itself. News, recipes, reference content and anything monetised by advertising all share a shape: an assistant that answers the question has taken the transaction and left you the cost of producing the answer. Training crawlers take content for a product you are not paid for and are not credited in. Blocking is a coherent business decision and plenty of large publishers have made it.

Allow, if you sell something the answer leads to. For software, services, professional practices and most business to business companies, being named in an answer is the point. A buyer asking an assistant which tool to use is further down the funnel than a buyer typing a keyword into a search box, and being absent from that answer is a lost sale with no compensating upside. Nobody bought less software because ChatGPT was allowed to read the documentation.

The uncomfortable middle is a business that is both, and there the crawler and fetcher split above is usually the honest answer.

robots.txt is a convention, not a fence
Every major operator publishes documentation stating that their crawlers respect robots.txt. Independent research has repeatedly found fetches from paths that a robots.txt disallowed, sometimes from agents identifying themselves differently or from addresses not published as belonging to the operator.

Both things are true at once and it is worth being precise about the consequence. robots.txt is a request that well behaved software honours. It has no enforcement behind it. If a block is a preference, robots.txt is the right tool and it is sufficient. If a block has to hold, enforce it where it can be enforced: firewall rules, bot management at your CDN, or authentication in front of the content. Use robots.txt alongside that to state the intent clearly, because an operator that finds no stated preference has been given no preference to ignore.

How to check your own file
Reading robots.txt top to bottom is exactly how the group precedence mistake survives review. Evaluate it the way a crawler does, once per user agent.

Our AI crawler checker fetches your live robots.txt and does this for seven named bots, reporting which assistants are shut out by name rather than telling you something is wrong somewhere. It runs on a free account with no card. If the result surprises you, the fix is one commit and it takes effect on the crawler's next visit.

Two things worth doing in the same sitting. Confirm the crawlers can actually read the page once they arrive, because most AI crawlers do not run JavaScript and a client rendered site can be wide open and still effectively empty. Then check whether your brand is currently named in AI answers at all, which is the question all of this was ever in service of.

We wrote this article to the same standard our engine scores: the answer in the first sixty words, sections bounded by headings a model can chunk on, a table where a table is clearer than prose, and every claim either verifiable in a published document or marked as a snapshot with a date on it. You can run that check on this page.

Sources

Every token in the table above comes from the operator's own documentation. Check these rather than us, because they change and we are a snapshot.
OpenAI, bots and user agents, for GPTBot, OAI-SearchBot and ChatGPT-User.
Anthropic, does Anthropic crawl the web, and how can site owners block it, for ClaudeBot and Claude-User.
Perplexity, bots, for PerplexityBot and Perplexity-User.
Google, common crawlers and the overview of Google crawlers, for Googlebot and Google-Extended.
Apple, about Applebot, for Applebot and Applebot-Extended.
Common Crawl, CCBot.
IETF, RFC 9309: Robots Exclusion Protocol, for the group selection and longest match rules described above.

FAQ
Does blocking GPTBot remove my site from ChatGPT?
It stops OpenAI fetching new pages from your site, so anything you publish from that point on cannot be quoted. It does not erase what a model already absorbed during earlier training, and it does not remove pages on other sites that describe you. The effect is a slow fade rather than a disappearance.

What is the difference between GPTBot and OAI-SearchBot?
GPTBot crawls pages for OpenAI generally. OAI-SearchBot exists to build the search index that surfaces links inside ChatGPT. If you want to be cited with a clickable link but you do not want your content used for training, OAI-SearchBot is the one to allow and GPTBot is the one to disallow.

Does Google-Extended affect my Google Search rankings?
No. Google-Extended is a control token, not a crawler. It governs whether content Googlebot already fetched may be used to ground and train Gemini and AI Overviews. Disallowing it changes nothing about ordinary Search indexing or ranking.

Do AI crawlers actually obey robots.txt?
The major operators publish documentation saying their crawlers do, and independent research has repeatedly found fetches from disallowed paths. robots.txt is a request, not a fence. If a block has to hold, enforce it at the edge with firewall or bot management rules and use robots.txt to state the intent.

How do I check which AI crawlers are blocked on my site?
Fetch your robots.txt and evaluate it once per user agent, applying real group precedence rather than reading it top to bottom. Our free AI crawler checker does exactly that against seven named bots and tells you which assistants are shut out.

Originally published at https://visibility100x.com/blog/ai-crawlers-gptbot-claudebot-perplexitybot/