DEV Community: Costin Gheorghe

AI Crawlers Are Now a Line Item: What They Actually Cost Wikimedia and Small Sites

Costin Gheorghe — Wed, 16 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

Most of what gets written about AI crawlers is about visibility — can ChatGPT
see your page, will it cite you, should you welcome the traffic. None of that
is the problem for a growing list of sites. Their problem is that the traffic
showed up, kept showing up, and the bill came due.

The clearest documented case is Wikimedia. The Wikimedia Foundation's own
engineering blog reported that bandwidth consumption for multimedia downloads
from Wikimedia Commons has surged 50% since January 2024, and it did not
hedge about the cause: the surge was not coming from human readers, but from
"automated, data-hungry scrapers looking to train AI models." TechCrunch's
reporting on the post picked up the same figures. This is not a monitoring
question — we covered that one already —
it's a cost and reliability question, and it deserves its own answer.

Why bots cost more per request, not just more in total

The intuitive model is that bot traffic is a scaled-up version of human
traffic — more requests, same shape, so the cost scales linearly with volume.
Wikimedia's explanation is that this isn't what happened, and the mismatch is
the actual story.

Human traffic clusters around popular pages, the ones a CDN or edge cache
already has warm. Crawler bots, per Wikimedia's own account, tend to "bulk
read" — they work through large numbers of pages indiscriminately, including
the long tail of obscure pages that were rarely requested before and were
never worth caching. Serving those requests means going back to origin
infrastructure instead of an edge cache, every time. A bot doesn't just add
volume; it adds volume in the expensive shape.

The numbers back this up directly. Bots accounted for roughly 65% of the
most resource-intensive traffic Wikimedia serves, despite representing only
around 35% of overall pageviews. Read that gap carefully: the group causing
two-thirds of the expensive work is a third of the requests. If you only
looked at request counts, you would badly underestimate what's driving your
bill.

Wikimedia was blunt about what this does to infrastructure built for a
different kind of spike: its systems, in the Foundation's own words, were
"built to sustain sudden traffic spikes from humans during high-interest
events, but the amount of traffic generated by scraper bots is unprecedented
and presents growing risks and costs." That's a nonprofit that has spent two
decades engineering for viral human traffic, saying the thing it wasn't built
for is a bot.

It isn't only Wikimedia

Wikimedia has the size and the engineering culture to publish a number. Most
sites carrying the same problem don't, which is a reason to take the
anecdotal reports seriously rather than dismiss them for lacking a published
percentage.

The same body of reporting that surfaced Wikimedia's figures also names
several smaller, independent infrastructure projects describing the identical
pattern in their own words: the git-hosting service SourceHut, Diaspora
developer Dennis Schubert, the repair-guide site iFixit, and the
documentation host Read the Docs have all separately and publicly reported
bandwidth spikes and inflated infrastructure costs they attribute to
AI-crawler traffic — in some cases severe enough to affect site reliability
for real users. Unlike Wikimedia's figures, none of these came with a formal
published percentage, so treat them as real, corroborating, but less precise
than Wikimedia's own numbers. The pattern showing up independently across a
git host, a documentation site, a repair-guide site, and an individual
developer's project is still worth something even without a shared
methodology behind it.

Wikimedia's response is measurable too: its 2025/2026 annual plan sets an
explicit goal of cutting crawler-generated request rate by 20% and bandwidth
usage by 30%. That's a nonprofit budgeting engineering time specifically
against bot load, not against human growth — which tells you how large the
line item got.

What this means if you're not running Wikimedia-scale infrastructure

Almost nobody reading this runs a site at Wikimedia's scale, and that's the
point worth making rather than skipping past: the mechanism doesn't require
that scale. A "bulk read" crawler that hits your long tail of rarely-visited
pages, bypassing your cache the same way, produces the same disproportionate
cost on infrastructure a hundredth the size. The absolute dollar figure is
smaller. The ratio — request share versus cost share — is the same shape.

If you're behind a CDN, you already have more leverage here than you might
realize, and most of it costs nothing to turn on.

Start with rate-limiting, not blocking. LetsLaunch runs behind
Cloudflare, the same setup we've described before
for watching crawler traffic — the same dashboard that shows you who's
visiting can also throttle them. Cloudflare's rate-limiting and bot-management
rules can target a specific crawler's request rate directly, which is a much
narrower tool than an outright block. A crawler capped at, say, one request
per second still finishes crawling your site — just slower, and without
hammering your origin in a burst. That's usually the actual goal: less load,
not zero bots.

Know what blocking actually trades away. This is the part worth being
honest about, because it points in the opposite direction from most of what
this blog has argued elsewhere. We've spent several other posts
on the case for staying crawlable, because citation eligibility depends on a
crawler being able to reach your page at all. Blocking a retrieval crawler
outright — OAI-SearchBot, PerplexityBot, Claude-SearchBot — solves a
bandwidth problem by creating a visibility problem. If your actual goal is
citation, rate-limiting is almost always the better trade: a slower crawl
still completes, and you stay in whatever index that crawler feeds. Reserve
an outright block for a crawler that's demonstrably causing reliability
problems for real users right now, not for one you're merely annoyed at
seeing in the logs.

Proof-of-work challenges are a real, separate category. Some smaller
open-source infrastructure projects have adopted tools that put a lightweight
computational challenge in front of a request before serving the page — cheap
enough that a normal browser clears it invisibly, expensive enough that
sustaining a bulk-scraping pattern across thousands of pages gets
computationally costly for whoever is running the scraper. This is a real and
growing defense worth knowing about by mechanism, whether or not you adopt
it — it's a different lever from a robots.txt rule or a firewall block,
because it changes the cost calculus for the crawler operator rather than
just refusing the request.

Match the response to the actual problem. The uncomfortable truth is that
"should I block AI crawlers" doesn't have one answer, because it's two
different questions wearing the same sentence. A site chasing citation
eligibility should not blanket-apply advice written for a bandwidth crisis —
"just block everything AI-labeled" throws away exactly what you're trying to
get. A site with a genuine bandwidth or reliability problem should not feel
obligated to stay maximally open just because citation matters to sites in
general — if the traffic is actively degrading service for real users, that's
a legitimate reason to throttle or block specific offenders, full stop. Check
our crawler cheat sheet for what each bot
actually is before deciding which category you're in.

Is AI crawler traffic actually costing sites money?

Yes, documented directly. The Wikimedia Foundation's own engineering blog
reported a 50% surge in bandwidth for multimedia downloads from Wikimedia
Commons since January 2024, attributed to AI-training scrapers rather than
human readers, with bots responsible for roughly 65% of its most
resource-intensive traffic despite being about 35% of pageviews. Smaller
sites — SourceHut, iFixit, Read the Docs, and independent developer Dennis
Schubert among them — have separately reported the same pattern, without
Wikimedia's level of published detail.

Should I block AI crawlers to save bandwidth?

Only if you have a genuine, demonstrated bandwidth or reliability problem,
and rate-limiting is usually the better first move over an outright block.
Throttling a crawler's request rate reduces load while letting the crawl
still complete, which keeps you eligible for citation in that crawler's
index. A hard block trades your bandwidth problem for a visibility problem —
worth it if the traffic is actually degrading your site for real users, not
worth it if you're just trying to reduce a number in a dashboard.

Why does bot traffic cost more than the same amount of human traffic?

Because of what gets requested, not just how much. Wikimedia's own
explanation is that crawler bots "bulk read" through large numbers of pages,
including rarely-visited ones that were never popular enough to be cached —
so each of those requests goes back to origin infrastructure instead of
being served from a CDN edge cache. Human traffic clusters around popular,
already-cached pages; bot traffic doesn't, which is why a smaller share of
requests can still account for the majority of the most expensive ones to
serve.

Cloudflare Says Perplexity Ignored Robots.txt. We Checked the Record.

Costin Gheorghe — Wed, 16 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

Our own AI crawler cheat sheet says the named
crawlers "have an established track record of compliance" with robots.txt.
That line needs a footnote, and the footnote is Perplexity.

In a blog post published in August 2025, Cloudflare accused Perplexity of
"stealth crawling" — disguising its crawler's identity to keep pulling content
from sites that had explicitly blocked Perplexity's declared crawler, either
through robots.txt or a network-level block. This is not a vague industry
rumor. Cloudflare published a specific experiment, specific traffic numbers,
and a specific response. Perplexity disputes the characterization. Both of
those things are true at once, and this post is about being precise on which
parts are documented fact and which part is a live dispute.

What Cloudflare says it found

Cloudflare's test was simple enough to describe in two sentences. It stood up
brand-new domains that were never linked from anywhere and never submitted to
any search engine or index, so the only way to know what was on them was to
actually fetch and read them. Each domain's robots.txt disallowed every
crawler, with no exceptions.

Then, according to Cloudflare's blog post, they asked Perplexity's own AI
assistant about the content on those exact test domains — and got back
detailed, accurate answers about pages the assistant should have had no
legitimate way to see. Nothing was indexed. Nothing was linked. The only path
from "page exists" to "assistant can describe it" was a crawl that ignored the
disallow rule.

Cloudflare says the traffic behind this broke down into two distinct patterns:

Perplexity-User, Perplexity's openly declared, user-triggered agent, running at roughly 20-25 million requests a day. This is the crawler named on our own cheat sheet and it isn't the accusation — a declared agent fetching a page a user asked about is normal and expected.
A second, undeclared crawler, at roughly 3-6 million requests a day, identifying itself as an ordinary Chrome browser on macOS. According to Cloudflare, this traffic used IP ranges outside Perplexity's official published ranges and rotated across different network providers — different ASNs — in a pattern consistent with avoiding detection as a single identifiable source.

That second pattern is the entire allegation. Not that Perplexity crawls the
web — every retrieval-based assistant does that, openly, and we cover the
mechanics of it elsewhere. The allegation is that when the declared crawler
was blocked, a second, disguised one kept going anyway.

What Cloudflare did about it

Cloudflare didn't just publish a blog post and leave it there. It removed
Perplexity from its "verified bots" program, added detection rules for the
disguised traffic pattern to its managed rule sets, and extended the ability
to block that traffic to every customer — including free-tier sites that
previously had no way to act on this kind of pattern even if they noticed it.

That's the detail worth sitting with. A robots.txt disallow line is
something any site can write, for free, unilaterally. Detecting a crawler that
is actively trying not to look like a crawler — rotating IP ranges and network
providers to defeat pattern-matching — is not something most site owners can
do themselves. It needs traffic volume and infrastructure at Cloudflare's
scale to even notice the pattern, let alone block it. According to Cloudflare,
over 2.5 million websites added AI-crawler blocking rules within a month of
the related tooling becoming available — a number worth reading as
approximate, since it's Cloudflare's own reporting on adoption of its own
feature, not an independently audited count.

What Perplexity says

Perplexity disputes the characterization. Multiple outlets reported a company
spokesperson pushing back, disputing that the undeclared crawler belonged to
Perplexity at all, or disputing that any blocked content was actually
accessed. We're stating that as what it is: a public rebuttal, not a
concession, and not something we can independently verify from here.

We're not going to adjudicate that dispute for you. What we can say plainly is
that Cloudflare's evidence is unusually concrete for this kind of claim — a
controlled experiment on domains nobody could have found any other way,
compared to the more common pattern of "our traffic logs look suspicious,"
which is much easier to wave away. That doesn't make the allegation settled.
It makes it a specific, checkable claim rather than a vibe, which is more than
most accusations like this get.

This isn't how OpenAI's crawlers have been reported to behave

Worth one paragraph of context: multiple reports have described OpenAI's
crawlers — the same GPTBot and OAI-SearchBot covered on our
cheat sheet — actually stopping when a site
disallows them via robots.txt. That's a reported contrast, not a claim that
OpenAI's crawlers are flawless everywhere at all times; we've said elsewhere
that robots.txt compliance across the industry is a norm the major crawlers
have generally followed, not a legal guarantee that binds any of them. This
incident is the sharpest documented exception to that norm we're aware of. It
is not proof the norm doesn't exist.

What this actually changes for a site owner

Nothing about this changes the mechanics we've written about elsewhere — see
why ChatGPT can't see your site for
the separate, much more common problem of crawlers simply not rendering your
JavaScript. What it changes is how much you can trust a robots.txt disallow
line to actually be the whole story.

The honest takeaway is narrow. A robots.txt rule is a request you cannot
verify is being honored by every crawler that might read it, and this incident
is the concrete case for why — not a reason to assume the file is theater,
since the record shows most named crawlers do follow it. If you want to check
whether your own site is actually being blocked the way your robots.txt
claims, our AI crawler check fetches your page as
each named crawler and shows you what came back, which tells you what's
happening to declared traffic. It cannot tell you whether an undeclared
crawler is also getting through — that's the part this whole story shows a
text file was never going to catch, and why Cloudflare's fix lived at the
network layer instead of in anyone's robots.txt.

And a separate reminder, because it's easy to conflate the two: none of this
tells you anything about whether being crawled — declared or not — makes an
assistant more likely to cite you. That's a different, unresolved question,
and we've written about what actually correlates with citation in
what makes AI engines cite a source.
Getting crawled and getting cited are not the same problem, and this story is
only about the first one.

Did Perplexity admit to stealth crawling?

No. Cloudflare published detailed findings — including a controlled experiment
on unlinked test domains — alleging that an undeclared, disguised crawler kept
accessing sites that had blocked Perplexity's declared crawler. Perplexity
publicly disputed the characterization through a company spokesperson. This is
a documented, disputed claim, not a settled admission.

Does this mean robots.txt doesn't work against AI crawlers?

No. The major named crawlers, including OpenAI's, have a general track record
of respecting robots.txt, which is what our cheat sheet
is built around. This incident is the clearest publicly documented exception
to that track record, not evidence that compliance across the industry is
fake. Treat it as a reason to verify rather than a reason to give up on the
file entirely.

How did Cloudflare detect the alleged stealth crawling?

By creating test domains with a robots.txt disallowing every crawler, never
linking or indexing them anywhere, and then asking Perplexity's AI assistant
about content on those exact domains. According to Cloudflare, the assistant
answered with accurate detail about pages it had no legitimate way to know
about, which pointed to a crawler ignoring the disallow rule rather than any
public index.

Is AI Referral Traffic Actually Worth Having? What the 2026 Numbers Say

Costin Gheorghe — Tue, 15 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

Everything we've written about AI citations on this blog so far has been
about the mechanism: can a crawler see your page, what correlates with being
cited, which platform is worth your time. All of it stops short of the one
question a founder actually cares about once traffic starts showing up in
the logs as chatgpt.com or perplexity.ai in the referrer field: is it any
good?

That's a different kind of question, and it has a different kind of answer.
Nobody has shown a reliable way to cause more AI citations — see what
actually gets you cited by ChatGPT and
Perplexity for why we won't
pretend otherwise. But whether the traffic that does show up is worth having
is a separate claim, and it's one several named analytics firms have actually
measured, on real production traffic, not a lab benchmark. We're not going to
blend their numbers into one tidy statistic — they don't agree with each
other closely enough for that — but they agree on the direction, and that
agreement is worth taking seriously.

The numbers, by source

Seer Interactive, a digital-marketing analytics firm, measured
conversion rates by referral source and found ChatGPT-referred traffic
converting at roughly 15.9%, against roughly 1.76% for Google organic search
traffic in the same measurement — call it roughly a 9x difference. That's
specifically ChatGPT compared against organic search, not "AI traffic" as a
category. The same analysis measured Perplexity at around 10.5% and Claude at
around 5.0%. All three beat the organic baseline by a wide margin, but they
don't match each other — ChatGPT, Perplexity and Claude converted at three
meaningfully different rates in the same study, which is consistent with
something we've said elsewhere on this blog about AI platforms generally:
they are not interchangeable, and treating "AI referral traffic" as one
undifferentiated bucket erases a real spread in the underlying data.

Adobe Analytics, in an April 2026 report covering March 2026 e-commerce
data, found AI-referred shoppers converting 42% better than non-AI traffic,
spending 48% more time on product pages, and generating 37% higher revenue
per visit. This is worth flagging clearly as a retail-specific measurement —
Adobe's data comes from shopping behavior, not from SaaS sign-ups, content
engagement or lead forms. A founder running a B2B tool should read this as
"directionally consistent with the pattern," not as a number that transfers
to their own funnel.

Conductor, an SEO and content platform, reported in its own 2026
benchmark that ChatGPT accounts for roughly 87.4% of all AI referral traffic
across the industries it measured, with every other assistant splitting the
remainder. We already made a version of this point in our comparison of
ChatGPT, Google AI Overviews and
Perplexity using reach
and user counts — Conductor's number reinforces it from a different angle,
using actual referred visits rather than platform-level user counts. Either
way you measure it, "AI traffic" is, for most sites in most categories, in
practice overwhelmingly a ChatGPT question. Building a strategy evenly split
across four assistants is optimizing for a distribution that doesn't match
what shows up in the logs.

Here's all three side by side, with what each one actually measured:

Source	What it measured	Headline figure	Scope
Seer Interactive	Conversion rate by referral source	ChatGPT ~15.9% vs. organic ~1.76% (~9x); Perplexity ~10.5%; Claude ~5.0%	Cross-industry conversion comparison
Seer Interactive	Pages per session by referral source	ChatGPT ~2.3 vs. organic ~1.2; Perplexity/Gemini trend lower and more decisive	Engagement depth, not conversion
Adobe Analytics	E-commerce shopper behavior, March 2026	+42% conversion, +48% time on product pages, +37% revenue per visit	Retail/shopping specifically
Conductor	Share of AI referral traffic by platform	ChatGPT ~87.4% of AI referrals	Referred-visit volume, not user counts

They don't agree, and that's worth saying plainly

A 9x conversion lift and a 42% conversion lift are not the same claim, and we
are not going to pretend they triangulate to some cleaner "real" number in
the middle. They come from different firms, measuring different traffic in
different time windows, using different methodologies and almost certainly
different definitions of "conversion" — a purchase in Adobe's retail data
means something different from a completed signup in Seer's cross-industry
sample. Multiplying a nine-fold conversion difference against a
forty-two-percent one and reporting an average would manufacture precision
neither source claims to have.

What's consistent, and what we're comfortable stating plainly, is the
direction: every named source that measured this in 2026 found AI-referred
traffic converting meaningfully better than organic search or non-AI
traffic. The size of that gap varies a lot — by platform, by industry, by
what "conversion" was defined to mean. Treat the specific multiples as
illustrative of a real and repeated pattern, not as a number you can plug
into your own forecast.

Engagement isn't the same story on every platform

Seer's data on pages-per-session is worth reading carefully rather than as
one more "AI traffic is better" data point, because it complicates a simple
story. ChatGPT-referred visitors browsed roughly 2.3 pages per session
against roughly 1.2 for organic — more exploration, more time in the site.
But Perplexity- and Gemini-referred visitors trended the opposite way:
fewer pages, more focused, more decisive visits. Both patterns are
consistent with higher-value traffic; they just look different. A founder
looking at a low pages-per-session number from Perplexity referrals
shouldn't read that as underperformance against the ChatGPT number — it may
be the opposite, a visitor who already knew exactly what they wanted and
went and did it. "More engaged" doesn't mean the same shape of behavior
across every platform, any more than the conversion numbers mean the same
percentage.

A plausible reason, stated as a guess

None of the sources above measured why this traffic converts better — they
measured that it does. But there's a reasonable explanation that's
consistent with all of it, and it's worth naming as long as it's labeled for
what it is: an inference, not a finding.

A visitor who clicks through from a citation inside a generated answer has
typically already had part of their question answered, and their intent
sharpened, by the assistant before they ever land on your page. They didn't
arrive from a broad, exploratory query the way a lot of organic search
traffic does — a meaningful share of organic clicks are early-stage research
that was never going to convert regardless of what the landing page says.
AI referral traffic, by construction, has already passed through one round
of qualification a search click hasn't. That's a plausible mechanism
consistent with a 9x lift and a 42% lift both pointing the same direction.
It is not something Seer, Adobe or Conductor directly measured, and we're
not going to present it as more than the reasonable guess it is.

What this actually changes

It doesn't change the honest answer to "how do I get more AI citations,"
which remains: nobody outside these companies can currently tell you a
reliable way to make that happen on purpose. What it changes is the answer
to a different, more modest question — is it worth doing the low-cost things
anyway, on the chance some AI traffic shows up. Before this data, the honest
case for being crawlable, describing your product consistently across
independent pages, and picking up a free listing somewhere like a LetsLaunch
submission rested entirely on cost: these things are free or
close to it, so there's no real argument against doing them even with an
unproven mechanism.

Now there's a second argument, and it's a real one even though it's
narrower than it sounds: if any of that effort does eventually produce a
citation and a click, multiple independent, named sources measuring real
production traffic in 2026 found that click converting meaningfully better
than an average organic visit. That's not a promise you'll get more AI
traffic. It's a reason to think the traffic is worth having if it comes.

How much better does AI referral traffic convert than organic search?

It depends heavily on the source and the platform, and the studies don't
agree on a single multiple. Seer Interactive measured ChatGPT referrals
converting at roughly 15.9% against roughly 1.76% for organic — about 9x —
with Perplexity around 10.5% and Claude around 5.0% in the same comparison.
Adobe Analytics, measuring e-commerce specifically, found a 42% conversion
lift for AI-referred shoppers. Both point the same direction; neither
number should be treated as the other's substitute.

Is AI referral traffic conversion data reliable, or is this the same kind of unproven claim as AI citation tactics?

These are different kinds of claims. Whether a specific action causes more
AI citations is unproven — see our review of the citation research. Whether
AI-referred traffic converts better once it arrives is an observed outcome,
independently measured by multiple named firms (Seer Interactive, Adobe
Analytics, Conductor) on real production traffic rather than a lab
benchmark. The exact multiple varies by study, but the direction is
consistent across all of them, which is a meaningfully stronger evidentiary
position than the citation-mechanism claims we're skeptical of elsewhere.

Is most "AI traffic" really just ChatGPT?

For most sites in most categories, yes. Conductor's 2026 benchmark found
ChatGPT accounted for roughly 87.4% of all AI referral traffic across the
industries it measured, with the remainder split among other assistants.
That matches what reach and user-count data already suggest — see our
comparison of ChatGPT, Google AI Overviews and Perplexity — but Conductor's
number is based on actual referred visits rather than platform-level user
counts, which makes it a stronger version of the same point.

Every AI Crawler That Hits Your Site, and What to Do About Each One

Costin Gheorghe — Tue, 15 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

Every AI crawler identifies itself with a distinct user-agent string, and the
strings do not map cleanly onto "AI" as one category. Some train models. Some
power a live answer with a citation. Some only fetch a page because a person
pasted its URL into a chat window a minute earlier. A robots.txt rule written
against "AI bots" as if they were one thing usually does something other than
what its author intended.

This is the reference list: every crawler we could verify, who operates it,
what it actually does, and the specific robots.txt line for it. Use it to
write the rule you actually mean, not the closest wildcard you found in
someone else's blog post.

The three jobs a crawler can be doing

Before the table, the distinction that the whole list hangs on. A crawler
identifying as AI-related is doing exactly one of three jobs:

Training. It fetches your pages to become part of a dataset a model is trained on, sometime later, in a batch you have no visibility into. Blocking it has no effect on anything that happens today.
Retrieval and citation. It fetches your pages to build or refresh an index that a live answer engine — ChatGPT Search, Perplexity, Copilot — reads from at answer time. Blocking it removes you from those answers, full stop.
User-triggered fetch. Nobody crawls anything. A person pasted your URL, or asked about it by name, and the assistant fetched that one page, once, live, to answer that one question. This isn't a crawl in the ordinary sense — it's the assistant acting as the user's browser for a single request.

We laid out the training-versus-retrieval split in more depth in
why ChatGPT can't see your site; this
post exists to give every crawler its own row instead of two examples.

OpenAI

Crawler	Job	Recommendation
`GPTBot`	Trains OpenAI's models. Respects `robots.txt`, per OpenAI's own published crawler documentation.	Block if you want out of training; costs nothing in ChatGPT's answers. Allow if you're fine being training data.
`OAI-SearchBot`	Powers ChatGPT Search's retrieval and citation index. A separate crawler from `GPTBot`, launched alongside ChatGPT Search in October 2024.	Allow, unless you want to be invisible to ChatGPT Search entirely.
`ChatGPT-User`	User-triggered: someone inside ChatGPT pasted or asked about your specific URL and the assistant fetched it live. Not a routine crawl.	Allow. Blocking it means a real person who linked to you gets nothing back, for no gain — there's no dataset being built to opt out of.

Anthropic

Crawler	Job	Recommendation
`ClaudeBot`	Anthropic's primary training crawler, functionally parallel to `GPTBot`.	Block for the same reason as `GPTBot`: opts out of training, changes nothing about Claude's live answers.
`Claude-SearchBot`	Retrieval and citation crawler, parallel to `OAI-SearchBot`.	Allow if you want to be citable in Claude's answers.
`Claude-User`	User-triggered fetch, parallel to `ChatGPT-User`.	Allow, same reasoning as `ChatGPT-User`.

OpenAI and Anthropic both explicitly support the training/retrieval split —
that's the entire reason two separate crawlers exist per company instead of
one. If you want out of training but still want to be citable, that's not a
compromise you're forcing on the system; it's the exact use case the split
was built for.

Perplexity

Crawler	Job	Recommendation
`PerplexityBot`	Perplexity's primary crawler, builds its retrieval and citation index.	Allow if you want to appear in Perplexity's answers. Block only if you want out entirely — there's no separate training-only variant to opt out of instead.
`Perplexity-User`	User-triggered fetch, parallel to `ChatGPT-User`.	Allow, same reasoning as the other user-triggered fetchers.

Google

Crawler	Job	Recommendation
`Googlebot`	Ordinary search crawler. Required to rank in Google Search at all.	Allow. This is not an AI-crawler decision — blocking it removes you from Google Search, not from an AI feature.
`Google-Extended`	An opt-out signal specifically for Google's AI training uses (Gemini, AI features). Separate from `Googlebot`.	Block if you want out of Google's AI training. Blocking it does not affect Google Search ranking — that's the entire point of it existing as its own token.

This is the pair people mix up most often, because both names start with
"Google" and only one of them touches your search rankings. Googlebot and
Google-Extended are read by Google's systems for entirely different
purposes, and disallowing the wrong one either does nothing or costs you your
search visibility, depending on which way you get it backwards.

Apple

Crawler	Job	Recommendation
`Applebot`	Ordinary crawler behind Siri and Spotlight search features.	Allow, unless you specifically want out of Apple's search surfaces.
`Applebot-Extended`	Apple's AI-training opt-out signal, parallel in purpose to `Google-Extended`. Separate from ordinary `Applebot`.	Block if you want out of Apple's AI training without touching Siri/Spotlight visibility.

Microsoft

Crawler	Job	Recommendation
`Bingbot`	Microsoft's ordinary search crawler — and also the retrieval backbone behind Copilot's web-grounded answers. Plays double duty the way `Googlebot` does for Google, except Microsoft hasn't split it into two tokens.	Allow. Blocking it costs you both Bing Search ranking and Copilot citations at once, because there's no separate opt-out for one without the other.

Crawler	Job	Recommendation
`Meta-ExternalAgent`	Meta's AI-training crawler, for Llama and Meta AI features.	Block if you want out of training for Meta's models.
`Meta-ExternalFetcher`	User-triggered fetch, parallel to `ChatGPT-User`, for links a user shares with Meta AI.	Allow, same reasoning as the other user-triggered fetchers.

Everyone else on the list

Crawler	Operator	Job	Recommendation
`Amazonbot`	Amazon	Crawls for Amazon's AI/Alexa-related features.	Allow unless you specifically want out of Amazon's AI features — no training/retrieval split is documented for it.
`DuckAssistBot`	DuckDuckGo	Powers DuckDuckGo's AI-assisted answer feature.	Allow if you want to be citable in DuckDuckGo's AI answers; this is a retrieval crawler, so blocking it removes you from those answers.
`Bytespider`	ByteDance (owner of TikTok)	Crawls to power TikTok search, content recommendations, and ByteDance's own AI features.	Your call — bundles search, recommendations and AI training into one crawler with no separate opt-out for any one piece.
`CCBot`	Common Crawl	A nonprofit that crawls the open web and republishes the dataset for anyone to use — researchers and AI labs included. Not itself an AI company, but its dataset is a common ingredient in many models' training data.	Block if your goal is keeping content out of AI training generally, since the dataset feeds more than one lab at once. Allow if you're fine contributing to open research datasets.
`Diffbot`	Diffbot (commercial web-data extraction)	Crawls on behalf of its own customers to supply structured data for AI training, search, and retrieval-augmented generation products.	Treat like `CCBot`: it's a pass-through to unknown downstream customers rather than one named consumer product, so block it if you want a hard boundary on who resells your content.
`Timpibot`	Timpi	Builds a decentralized search engine's index and collects training data for LLMs. Smaller and less well-known than the crawlers above, but real and documented.	Allow or block on the same logic as any other combined search/training crawler — there's no split to take advantage of.
`cohere-ai`	Cohere	Training/retrieval crawler for Cohere's models.	Allow if you want to be usable by Cohere's products; block to opt out. LetsLaunch's own `robots.txt` names this one explicitly rather than relying on a wildcard.
`MistralAI-User`	Mistral AI	Training/retrieval crawler for Mistral's models.	Same logic as `cohere-ai`. LetsLaunch's `robots.txt` names this one too.

What LetsLaunch actually does

Our own robots.txt (src/app/robots.ts in this codebase) names sixteen
agents explicitly in an AI_SEARCH_AGENTS group and allows all of them,
including cohere-ai and MistralAI-User alongside the bigger names — a
listing cited inside any of these answer engines is worth as much to a maker
as an ordinary search result, so we chose visibility over opting out. The code
comment on that group is direct about the user-triggered fetchers too:
ChatGPT-User, Claude-User and Perplexity-User ignore robots.txt by
design, so naming them there is a statement of intent, not a control that
actually stops anything.

That's our call for a directory that exists to get products seen. It is not
a universal recommendation. A publisher with a subscription paywall, or
content it doesn't want folded into a model's training data regardless of
citation upside, has a legitimate reason to block the training crawlers on
this list while leaving the retrieval and user-triggered ones alone — that
split is exactly what separate tokens like GPTBot/OAI-SearchBot and
ClaudeBot/Claude-SearchBot are for.

Writing the actual rule

Robots.txt groups by user-agent are independent — a crawler that finds a group
naming it by name ignores the * group entirely, so a specific agent needs
its own disallow lines if you want to block just that one:

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: *
Allow: /

That example opts out of OpenAI and Anthropic training specifically while
staying allowed everywhere else, including for OAI-SearchBot's retrieval
index. Copy the shape, not the specific agents — decide per crawler, from the
tables above, rather than pasting a blanket AI-blocking snippet that treats
every name on this page as the same decision.

Checking whether it's actually working

A robots.txt rule is a request, not an enforcement mechanism — the polite
crawlers on this list respect it, but the file itself can't verify anything.
Once you've written the rule you mean, two different checks matter and they
answer different questions. Our AI crawler check
fetches your page as each named crawler once and shows you the status code and
the content each one received right now — useful for confirming a rule change
took effect. For whether a crawler actually shows up over time, on its own
schedule, we wrote a separate guide to
grepping logs and reading Cloudflare's AI Crawlers dashboard — that's an
ongoing question a one-time fetch can't answer.

If you're launching a product and want it reachable by name across these
answer engines in the first place, a LetsLaunch listing is one more
independent page describing it the same way your own site does.

Does blocking GPTBot remove me from ChatGPT's answers?

No. GPTBot only collects training data; OAI-SearchBot is the separate
crawler that builds the index behind ChatGPT Search's citations. Blocking
GPTBot opts you out of training runs and has no effect on whether ChatGPT
can cite you — only blocking OAI-SearchBot does that.

Does blocking Google-Extended hurt my Google ranking?

No. Google-Extended is a distinct opt-out token for Google's AI training
uses, separate from Googlebot, which is the ordinary search crawler required
for ranking in Google Search at all. Blocking Google-Extended leaves
Googlebot and your search ranking untouched.

Should I block the user-triggered fetchers like ChatGPT-User?

Usually not. ChatGPT-User, Claude-User, Perplexity-User and
Meta-ExternalFetcher don't crawl your site on a schedule — they fetch one
page, once, because a real person pasted your link or asked about it by name
inside that assistant. Blocking them doesn't opt you out of any training
dataset; it means that specific person gets nothing back.

ChatGPT vs Google AI Overviews vs Perplexity: Where to Put Limited Effort

Costin Gheorghe — Mon, 14 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

Every "how to get cited by AI" article treats "AI" as one thing with one
mechanism. It isn't. ChatGPT, Google AI Overviews and Perplexity are three
different products, built by three different companies, with wildly different
user counts and — more importantly for a founder deciding where to spend
time — genuinely different ways of deciding what to put in front of a reader.

This isn't another post about the mechanism of getting cited. We've written
those; see what actually gets you cited by ChatGPT and
Perplexity and why ChatGPT can't
see your site if that's what you're
after. This one is about the products themselves — who uses each, how big
each actually is, and what each one draws from when it answers — so a small
team can decide where limited effort has any chance of paying off before
spending it.

The three, side by side

	ChatGPT (search/browsing)	Google AI Overviews	Perplexity
Reach	1B+ weekly active users (OpenAI, reported mid-2026)	Appears in an estimated ~25-45% of Google searches, depending on measurement firm and query type	~230M monthly active users, 1B+ queries/month (Perplexity's own reported figures, early 2026)
What it's built on	Its own retrieval + browsing layer on top of a general model	Google's existing web index, enriched with authority/freshness signals	Its own retrieval layer, tuned toward recent and community content
Leans on	Wikipedia heavily among top cited sources	The same web index that ranks ordinary Google Search	Reddit and recent content heavily
Separate destination?	Yes — its own product, its own habit	No — a feature inside Google Search	Yes — its own product, smaller but growing fast

The numbers are worth taking one at a time, because none of them are as clean
as a single headline figure suggests.

How big is each one, honestly

ChatGPT is the outlier by scale. OpenAI reported ChatGPT crossing 1
billion weekly active users by July 2026, a figure that's been widely
reported since. That's a number for the whole product, not just its
search/browsing mode specifically — but it establishes the ceiling nothing
else here gets close to.

Perplexity is a different order of magnitude. Per Perplexity's own
reported figures, it passed roughly 230 million monthly active users in
early 2026 and processes over a billion queries a month. That's real and
growing fast — the same reporting put the company's valuation at over $20
billion — but it's a fraction of ChatGPT's reach. Treat Perplexity as a
smaller, more specific audience, not a peer-sized alternative.

Google AI Overviews is the hardest of the three to pin to one number, and
we're not going to pretend otherwise. Different measurement firms report
different shares of Google searches that trigger an AI Overview in 2026 —
estimates run from roughly a quarter to nearly half of searches, depending on
methodology and query type. That's a wide range, and anyone quoting one exact
percentage as settled fact is smoothing over a real disagreement between
measurement methods. What isn't in dispute: Google Search's total query
volume dwarfs ChatGPT's or Perplexity's own native search by a wide margin.
That's the actual reason AI Overviews matters as much as it does — not
because it's a large percentage of a small number, but because it's a
meaningful percentage of the largest number in this comparison, applied
inside a feature most people never consciously decide to use. Nobody opts
into AI Overviews the way they open the ChatGPT app; it just shows up above
the results they were already going to scroll past.

How each one actually decides what to show you

This is the part that should drive where you spend effort, because the three
don't source answers the same way at all.

ChatGPT leans heavily on Wikipedia among its top cited sources. That
doesn't mean Wikipedia is the only thing it cites, but it's a consistent
pattern across citation studies, and it means an encyclopedic, neutral,
well-structured presence — the kind of thing Wikipedia itself rewards —
correlates with showing up in ChatGPT's answers more than flashier marketing
copy does.

Perplexity leans the opposite direction: heavily toward Reddit and recent
content. Perplexity is tuned to surface what real people are currently
saying, not what a canonical reference source says. A product with an active,
recent presence in relevant subreddits and forums has a genuinely different
shot at a Perplexity citation than the same product with only a polished
homepage — see why Wikipedia and Reddit are different, mostly-unreachable
bets for what that actually takes.

Google AI Overviews doesn't have an independent retrieval system at all
in the way the other two do — it's built on top of Google's own existing web
index, enriched with authority and freshness signals layered on top of
ordinary ranking. Practically, that means the things that get a page ranked
in ordinary Google Search — indexation, authority, freshness, structure — are
the same things that get a page pulled into an AI Overview. It's not a
separate contest with separate rules; it's the same contest with an extra
layer on top.

The consequence, worth stating plainly: only about 11-14% of domains cited by
ChatGPT also get cited by Perplexity, per independent 2026 citation-tracking
analyses. Whatever gets you into one of these products' answers is not
reliably the same thing that gets you into the other's. Treat them as
separate bets, because the data says they behave like separate bets.

Where to actually spend effort

Given all of that, here's a decisive ordering for a small team that can't
chase all three equally.

Start with being crawlable and readable — this is non-negotiable for all
three. If the AI crawlers behind ChatGPT and Perplexity can't get your
actual content out of your page (client-side rendering with nothing in the
initial HTML is the classic failure — see why ChatGPT can't see your
site), none of the sourcing-behavior
differences above matter, because you were never a candidate to begin with.
This is the one universal prerequisite. Check yours with the AI-readable
site tool before anything else on this list.

If you can only do one more thing, do ordinary SEO — for AI Overviews
specifically. Because AI Overviews is built on Google's existing index
rather than a separate retrieval system, classic SEO fundamentals — the
things that get you ranked in ordinary Google Search — carry more weight here
than they do for ChatGPT's or Perplexity's citation behavior. This is the one
case on this list where "just do normal SEO" is a legitimate, specific
AI-visibility strategy rather than a vague deflection. Given that AI
Overviews sits on top of the search engine with by far the largest total
query volume of the three, this is usually the highest-leverage single move
available to a small team.

Treat ChatGPT and Perplexity as separate, smaller bets — not extensions of
your SEO work or of each other. If you have the bandwidth to chase citation
specifically inside these products, know what you're chasing: a Wikipedia-style,
structured, neutral presence tends to correlate with ChatGPT citation more
than pure marketing copy does; an active, recent presence in relevant
community discussion (Reddit foremost) tends to correlate more with
Perplexity. But given the 11-14% overlap figure above, doing well in one is
not a reasonable basis for assuming you'll do well in the other, and neither
is a reasonable basis for assuming your Google AI Overview presence will
follow. A team with genuinely limited time should pick based on where its
actual audience already is — Reddit-heavy technical audience, lean toward
Perplexity's sourcing habits; broad consumer audience, ChatGPT's reach alone
probably makes it worth the baseline effort regardless.

Don't mistake any of this for a guarantee. Nothing here is a mechanism
proven to make any of these three name your product. It's a way to prioritize
limited effort given what's actually observable about how each product finds
its answers — reach, sourcing pattern, and how much they overlap with each
other. That's a real basis for a decision. It is not a promise.

One thing that helps regardless of which of these three you prioritize: an
independent listing that describes your product consistently with how you
describe it everywhere else. It won't cause a citation on its own — nobody
can currently show that any single action does — but a free LetsLaunch
listing is a low-cost way to add one more consistent, crawlable
description of your product to the web, which is the one thing that
correlates (not causes) with being named, across all three products discussed
here.

Frequently asked questions

Should a founder pick just one of ChatGPT, Google AI Overviews or Perplexity to focus on?

Not entirely, but the priority order should reflect the numbers above. Get
crawlable first, since it's a prerequisite for all three. Then lean toward
ordinary SEO, because it's the one strategy that specifically helps with
Google AI Overviews, which sits on top of by far the largest search volume of
the three. Only after that does it make sense to chase ChatGPT- or
Perplexity-specific citation habits, and those two should be treated as
separate efforts from each other, not one combined push.

Is Google AI Overviews the same kind of thing as ChatGPT or Perplexity?

Not really, and that distinction matters for where you spend effort. ChatGPT
and Perplexity are both independent products with their own retrieval
systems and their own habit-forming destinations. Google AI Overviews is a
feature inside ordinary Google Search, built on Google's existing web index
rather than a separate retrieval system — which is why classic SEO carries
more weight there than it does for the other two.

Does doing well in ChatGPT citations mean I'll also do well in Perplexity?

No, and the data says the opposite is closer to true. Independent 2026
citation-tracking analyses put the overlap between domains cited by ChatGPT
and domains cited by Perplexity at only about 11-14%. ChatGPT leans on
Wikipedia-style sourcing; Perplexity leans on Reddit and recent content.
Treat them as separate bets rather than assuming one strategy transfers to
the other.

llms.txt in 2026: We Publish One. Here's What It Actually Does.

Costin Gheorghe — Mon, 14 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

We publish an llms.txt and an llms-full.txt on this site. If you read the
rest of this post, you might reasonably ask why we bothered.

Short version: it costs nothing, it does no harm, and it is not the thing
doing any real work. If you are deciding whether to spend an afternoon on it,
the honest answer is spend five minutes instead — and then go read our guide
on what actually makes a site readable to AI crawlers,
because that part is not optional the way this one is.

What llms.txt was supposed to do

The idea, proposed in 2024, is a plain-text file at your domain root that
hands a language model a curated, markdown-formatted summary of your site —
the parts worth reading, without the navigation chrome and cookie banners a
normal crawl picks up. It reads like robots.txt for meaning instead of
access: not "you may crawl this," but "here is what matters if you do."

It's a reasonable idea. The problem is not the idea. The problem is that
essentially nobody who could act on it decided to.

What Google has actually said

At Google Search Central Live, Google's Gary Illyes confirmed Google does not
support llms.txt and has no plans to. He compared it directly to the
keywords meta tag — a field search engines stopped reading over a decade
ago because it's written by the site operator and therefore tells you nothing
they didn't already choose to claim about themselves.

John Mueller went further in a public reply: "Google doesn't use llms.txt or
llms-author.txt. I don't know of any other crawler / llm confirming they're
using these." Asked about the related content-signal robots.txt directive,
he added that no crawler or LLM uses it either — "it was made up by a CDN,
afaik it has no effects whatsoever" — and that shipping unsupported
directives is maintenance burden with no upside.

That is about as direct as a platform statement gets. Not "we're evaluating
it." Not "support is coming." Nobody is using it, and nobody we know of is
planning to.

What actually happens to a published llms.txt file

Google's disinterest would matter less if some other assistant had adopted the
file instead. So the more useful question is behavioural: when a domain
publishes one, does anything actually request it?

Ahrefs answered this at scale in May 2026, analysing server logs and bot
traffic across 137,210 domains. The numbers:

28% of domains in the sample publish a valid llms.txt.
97% of those files received zero requests for the entire month.
Of the 3% that got any traffic at all, 96% of requests came from bots — but a mix of crawlers, agents and scrapers, not specifically the assistants the file is meant for. Named AI retrieval bots like PerplexityBot and OAI-SearchBot accounted for only about 1% of the requests these files did receive.
On domains with no llms.txt at all, AI bots made zero speculative requests for one. They don't go looking. A crawler that respects llms.txt on the sites that have it would probe for it everywhere, the way every crawler probes for robots.txt on every domain it touches. That probing does not happen, which is a stronger signal than the low fetch rate on its own: the file isn't being ignored occasionally, it isn't part of any crawler's routine at all.

Put together: publishing the file gets you into a small minority (28%) who
did the extra work, and of that minority, 97% got nothing for it.

Why we still have one

None of this makes llms.txt harmful, and the cost of publishing one is close
to zero — it's a static file, not an integration. What we actually rely on to
be readable by ChatGPT, Claude and Perplexity is entirely different, and it's
the same lesson we keep landing on when we check whether a page is visible to
AI crawlers at all: the thing that
matters is whether the page itself — the actual HTML a crawler receives at
the actual URL — contains real content, unblocked, with no JavaScript
required to render it.

That's a robots.txt that names AI crawlers explicitly instead of leaving
them to guess, and a page that returns full content on the first request, no
script execution needed. llms.txt is neither of those things — it's a
side-channel a wildly small handful of crawlers might, unverified, glance at
once. We keep ours published because it costs one file and documents intent.
We do not tell anyone it is doing the job that server-rendered HTML is
actually doing.

If you are prioritising a Friday afternoon between the two, do the HTML.

Does llms.txt help with SEO or AI search rankings?

No measurable evidence that it does either. Google has stated directly it
doesn't use the file for search, and no AI assistant provider has confirmed
using it for anything. It has no known effect on classic search rankings or
on whether an assistant cites you.

Do ChatGPT, Claude or Perplexity actually read llms.txt files?

Essentially never, based on the available data. A May 2026 Ahrefs study of
137,210 domains found 97% of published llms.txt files received zero
requests in a month, and named AI retrieval bots made up roughly 1% of the
traffic the remaining 3% did receive. Domains without the file saw zero
speculative requests for one — crawlers aren't checking for it the way they
check for robots.txt.

Should I still publish an llms.txt file?

If it takes you five minutes, there's no harm in it — it costs nothing to
serve a static file, and it's a reasonable record of intent. Just don't treat
it as a fix for AI visibility, and don't spend real engineering time on it
ahead of the thing that's actually load-bearing: making sure the HTML your
server sends, with no JavaScript required, actually contains your content for
the crawlers that matter.

GPTBot Says It Visits Your Site. Here's How to Check, Every Week.

Costin Gheorghe — Sun, 13 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

We wrote before about whether an AI
crawler can read your page at all — rendering, robots.txt mistakes, a 403 from
your own firewall. That is a one-time question. You fetch the page as
OAI-SearchBot, you look at what comes back, you fix it or you don't, and the
answer stays true until you change something.

This post is about a different question: whether these crawlers actually show
up, on an ongoing basis, once the one-time check passes. A page that was
readable to GPTBot last month tells you nothing about whether GPTBot has
fetched it since. Readability is a fact about your server. Traffic is a fact
about their behavior, and it changes without you touching anything.

What you're actually looking for

Every major AI crawler identifies itself with a distinct user-agent string.
The ones worth watching for:

GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-User,
Claude-SearchBot, PerplexityBot, Perplexity-User, Google-Extended,
Applebot-Extended, Bingbot, Amazonbot, DuckAssistBot, Bytespider,
CCBot.

LetsLaunch's own robots.txt (src/app/robots.ts in this codebase) names
several of these explicitly rather than relying on a wildcard rule, for the
same reason you'd want to grep for them by name: a rule or a log line that
just says "bot" tells you nothing about which one.

Two of these strings are worth separating in your head before you start
counting anything, because they answer different questions.

Training crawlers and retrieval crawlers are not the same signal

We made this distinction in the readability post and it matters just as much
for monitoring: GPTBot collects training data. OAI-SearchBot and
PerplexityBot fetch pages to serve as citations in an actual answer. They
run independently, on their own schedules, for their own purposes.

If you only track GPTBot, a busy week of hits tells you your content might
end up in a future training run — and tells you nothing about whether you can
be cited in an answer today. If you want to know about citation potential, you
need to watch OAI-SearchBot, Claude-SearchBot and PerplexityBot
separately, not lump every AI-labeled agent into one counter. A dashboard that
reports "247 AI bot hits this week" without splitting training from retrieval
is reporting a number nobody can act on.

Grepping your own logs

If you run your own server or have access to raw access logs, this is a grep
away. Standard combined log format puts the request path, status code, and
user-agent on one line, so a single pipeline gets you counts per crawler:

grep -E "GPTBot|OAI-SearchBot|ClaudeBot|PerplexityBot|Claude-SearchBot" \
  access.log \
  | awk -F'"' '{print $6}' \
  | sort \
  | uniq -c \
  | sort -rn

awk -F'"' splits the line on quote characters, and in combined log format
the user-agent is the sixth quoted field — the request line and the referer
are the fourth and there's no fifth field before it. sort | uniq -c turns
that into a count per exact user-agent string, which is enough to tell you
who's showing up and how often.

To see what a specific crawler is actually requesting, rather than just how
often it shows up:

grep "PerplexityBot" access.log | awk '{print $7, $9}' | sort | uniq -c | sort -rn

Field 7 is the request path and field 9 is the status code in the default
combined format (exact field numbers shift if your log adds extra fields, so
check one line by hand before trusting the count). This tells you two useful
things at once: which pages a crawler actually cares about, and whether it's
getting 200s or something else on the way in.

Run this weekly, or wire it into whatever log-shipping you already have, and
you get a trend instead of a single data point. A one-time curl check answers
"can it read this page." A weekly grep answers "does it come back."

The version with no log access required

Most sites don't sit in front of raw logs — they sit behind Cloudflare, which
is also LetsLaunch's own setup. Cloudflare's dashboard has an AI Crawlers
section, under its AI Bots reporting, that shows this without any grep at all:
total requests from AI crawlers, a breakdown of allowed versus blocked
requests, a per-crawler split naming the major players — Google, OpenAI,
Anthropic, Perplexity, ByteDance, Common Crawl, DuckDuckGo among them — and
the paths those crawlers hit most.

If your site is already on Cloudflare, this is the first place to look before
building anything yourself. It's the same information the grep gives you,
already split by crawler, already trending over time, with nothing to
instrument. The tradeoff is that it only sees what passes through Cloudflare's
edge — if you serve some paths from elsewhere, those don't show up here.

Cloudflare's own traffic data shows AI bot traffic rising sharply year over
year. We won't put a number on that, because the precise figure varies by
which report and which time window you're reading, and a specific percentage
attributed loosely is worse than no percentage at all. The direction is the
useful part: this is traffic worth watching, not a rounding error, and it is
worth setting up a recurring check rather than a one-time one.

If you just want a fast answer today

Setting up log monitoring or checking a Cloudflare dashboard is worth it if
you want a trend. If you just want to know right now whether a specific
crawler can reach a specific page, that's a narrower and faster question, and
it's what our AI crawler check does: it fetches your
page as each crawler, once, and shows you the status code and the text each
one received. It won't tell you whether ClaudeBot visited last Tuesday. It
will tell you, in the next thirty seconds, whether it could visit at all. Pair
that with our guide to making a page AI-readable if
the answer comes back wrong.

What a crawler hit does and doesn't prove

Here's the part worth being honest about, because it's the same caution we
applied to citations in the readability post. Seeing PerplexityBot in your
logs is necessary evidence. It is not sufficient evidence.

A hit tells you the crawler reached your page and (assuming a 200 and
readable HTML) received your content. It does not tell you that content was
used, indexed, or ever surfaced to a single user. You cannot infer citation
frequency from crawl frequency — a crawler can fetch a page a hundred times
and cite it never, and there's no dashboard, ours included, that closes that
gap.

What monitoring does give you is the other direction, which is just as
useful: absence of hits is real information. If OAI-SearchBot has never
touched your domain, you cannot be cited from that index, full stop — no
amount of good content changes that if the crawler was never there to read
it. Monitoring won't tell you you're winning. It will tell you, reliably,
when you've been shut out before you've done anything else.

Treat crawler-hit counts as a diagnostic, not a growth metric. They rule
things out. They don't rule things in.

Do crawler visits mean my product will be cited by ChatGPT or Perplexity?

No. A crawler hit only confirms the page was fetched and, if it returned a
readable 200, that the content was received. Whether it's later cited in an
answer is a separate step nobody can currently observe from the outside, and
we know of no reliable way to predict it from crawl frequency alone.

Do I need Cloudflare to monitor AI crawler traffic?

No, but it removes the setup work. Cloudflare's AI Crawlers analytics shows
per-crawler request counts, allowed-versus-blocked splits, and top crawled
paths for any site behind its network, with no log access needed. Without
Cloudflare, the same information is available by grepping your raw access
logs for the relevant user-agent strings.

What's the difference between monitoring GPTBot and monitoring OAI-SearchBot?

GPTBot hits reflect training-data collection. OAI-SearchBot hits reflect
retrieval for citations in ChatGPT's search answers. They're separate
crawlers with separate purposes, so tracking only GPTBot tells you nothing
about whether you can currently be cited — you have to watch the
search/retrieval agents separately to answer that question.

Does Schema Markup Help You Get Cited by AI? What Google Actually Says

Costin Gheorghe — Sun, 13 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

Somewhere between "add schema markup" and "get cited by ChatGPT" a lot of
advice quietly inserts a step that doesn't exist. The pitch is usually some
version of: implement the right structured data and AI Overviews will start
citing you. It's a tidy claim, it sounds technical enough to be true, and it
isn't what Google's own documentation says.

What Google actually requires for AI Overviews

Google's current public documentation on AI Overviews and AI Mode is direct
about eligibility: there are no special extra technical requirements beyond
being indexed and eligible to appear in ordinary Google Search with a snippet.
Not a schema type. Not a markup vocabulary. Not a checklist item labeled "AI
Overviews" anywhere in Google's structured data guidelines.

That's worth sitting with, because it forecloses a specific, common claim.
There is no "AI schema" to add. If a vendor is selling you a structured-data
package specifically for AI Overviews eligibility, they are selling you
something Google has never said exists.

So why does structured data get talked about in the same breath as AI answers at all

Because it's genuinely useful for something adjacent, and the two things get
merged in a lot of marketing copy until they sound like one thing.

Schema markup's documented job is helping a search engine parse a page's
meaning more precisely — this is a product, here is its price, this
organization is who it claims to be, this block of text answers this
question, this article has this structure. That precision is what makes a
page eligible for ordinary rich results in classic Google Search: star
ratings, price ranges, FAQ dropdowns, breadcrumbs. None of that is new or
AI-specific — it's the same mechanism structured data has always served.

Google's actual stated requirement isn't that the markup exists. It's that
the markup is accurate and consistent with what's visibly on the page. A
Product schema block claiming a 4.8-star rating on a page that shows no
reviews at all isn't a growth hack, it's a policy violation, and it's treated
as one.

The inference worth making carefully

Here's the part that actually justifies writing a post about this, because
if the answer were simply "schema does nothing for AI," there'd be nothing
else to say.

We've already established, in our comparison of ChatGPT, Google AI Overviews
and Perplexity, that AI
Overviews doesn't run its own independent retrieval system. It sits on top of
Google's existing web index — the same one that ranks ordinary search
results — with authority and freshness signals layered on top. It isn't a
separate product reading a separate corpus; it's a feature inside Google
Search, drawing on the index Google Search already built.

Follow that through and a reasonable inference falls out: anything that
helps Google's systems parse and trust a page more precisely — including
accurate, consistent schema markup — plausibly supports the same underlying
signals AI Overviews draws from, even though no AI-specific markup exists to
add. Not because schema has an AI mode, but because AI Overviews is drawing
on the same index that schema was always helping Google understand.

Notice the hedging in that paragraph, because it's load-bearing. "Plausibly
supports" is not "causes." We have no controlled study showing that adding
schema markup increases how often AI Overviews cites a given page. What we
have is a documented mechanism (schema improves parsing) and a documented
architecture fact (AI Overviews sits on the index that parsing feeds), and a
chain of reasoning between them that we have not seen tested. That's a
different, weaker claim than "schema gets you cited by AI," and marketing
copy that skips the hedge is overstating what's known.

Where this breaks down: FAQPage schema specifically

This is the case where the temptation to conflate the two things is
strongest, so it's worth being explicit rather than implicit about it.

FAQPage schema is real and it does something documented: implemented
correctly, it can earn an expandable FAQ dropdown directly in Google's
search results, next to your listing. That's a legitimate, specific,
well-documented rich-result feature. Nobody is disputing it.

What it is not is a citation tactic for AI assistants. We've said this
plainly elsewhere on this site, in our free AI-readable-site
tool: we have seen no evidence that adding FAQ
markup produces a lift in how often an assistant cites you, and we're not
going to imply otherwise here either. The FAQ dropdown in classic search
results and being quoted by an AI assistant are two different outcomes, and
a lot of "schema for AI visibility" content treats getting one as evidence
you're working toward the other. It isn't. Implement FAQPage schema if you
want the search-results dropdown — that's a real, achievable thing. Don't
implement it believing it's an AI-citation lever, because nothing we've seen
supports that being true.

A concrete example of taking schema seriously without the AI angle

We run structured data on every product page on this site, and the reason
has nothing to do with AI citation. Product names, taglines and descriptions
on LetsLaunch are supplied by makers — user-generated content we don't
control the contents of. If you serialize that straight into a
<script type="application/ld+json"> tag with a plain JSON.stringify, a
maker whose product name happens to contain a literal </script sequence
breaks out of the script block, because an HTML parser closes a <script>
tag at the first literal </script it sees regardless of what the
surrounding JSON says. That's not a hypothetical — it's the kind of thing
that shows up the first time somebody pastes a tagline with the wrong
characters in it.

That's why every piece of JSON-LD on this site goes through a function
called serializeJsonLd, which escapes <, > and & (and a couple of
line-terminator characters that are legal JSON but not legal
mid-statement in older JavaScript) before anything reaches the page. It's
defensive engineering against user-supplied content breaking a page, not a
citation-boosting tactic. We mention it here because it's the honest
version of "we take structured data seriously" — the reason is correctness
and security, and any downstream SEO benefit is a side effect of doing the
markup right, not the reason we did it.

The actual sequence, in order

Structured data should be accurate and match what's visibly on the page. That's Google's stated requirement, and it's also just not lying to a parser that other systems (possibly including AI ones) may lean on later.
Implement schema for what it's actually documented to do — rich results in classic search — not for an AI-specific eligibility gate that doesn't exist.
Treat any AI-citation benefit as a plausible side effect of a page being better understood by the same index AI Overviews sits on top of, not as a measured, provable outcome you can point to.
If you want an assistant to describe your product accurately, the more defensible lever remains describing it the same way everywhere it appears — your own site, and any independent page that lists you, including a LetsLaunch listing if you want one more. We've written about why an assistant needs to actually receive that text in the first place before any of this matters at all.

Does adding schema markup get you cited by ChatGPT or AI Overviews?

There's no evidence of a direct, measured effect, and Google's own
documentation names no AI-Overviews-specific markup to add in the first
place. What's defensible is a chain of inference: schema helps Google parse
a page more precisely, and AI Overviews sits on top of the same index that
parsing feeds — so accurate schema plausibly supports the same underlying
signals, without anyone having measured a specific AI-citation lift from it.

Is FAQPage schema worth implementing?

Yes, for what it actually does: it's a real, documented way to earn an
expandable FAQ dropdown in ordinary Google search results. It is not a shown
way to get an AI assistant to cite you more often — we've stated that
directly elsewhere on this site and we're repeating it here rather than
softening it.

Do I need special schema markup to be eligible for Google's AI Overviews?

No. Google's current documentation states there are no special extra
technical requirements for AI Overviews or AI Mode eligibility beyond being
indexed and eligible to appear in ordinary Google Search with a snippet.
There is no AI-Overviews-specific schema type or vocabulary to add.

Should You Block GPTBot? The Decision, Not the Panic

Costin Gheorghe — Sat, 12 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

Somebody on a founder Slack posts a robots.txt snippet that blocks GPTBot,
someone else says "wait, doesn't that get you removed from ChatGPT," and the
thread ends in fifteen replies of confident, contradictory advice. It shouldn't
take fifteen replies. This is a narrow, mechanical question with a narrow,
mechanical answer.

The one fact that resolves most of the confusion

Blocking GPTBot has no effect on whether ChatGPT can cite you in an answer.

That's it. That's the fact everyone in that thread needed. ChatGPT's search
and citation feature is powered by a different crawler, OAI-SearchBot, and
we've covered the split in detail in
why ChatGPT can't see your site:
GPTBot collects training data, OAI-SearchBot fetches pages for the index
behind ChatGPT's citations, and OpenAI operates them as independent crawlers.
Disallowing one in robots.txt says nothing to the other. If your actual goal
is "I want ChatGPT to be able to find and quote my page," this entire post is
irrelevant to you — go read that one instead, because the crawler you care
about isn't the one this post is about.

If your actual goal is "I don't want OpenAI training a model on my content,"
keep reading, because that's a real and separate question, and it deserves an
honest answer about what blocking GPTBot actually buys you.

What blocking GPTBot actually does

Per OpenAI's own published crawler documentation, and consistent with
independent server-log analyses of how major AI crawlers behave, GPTBot
generally respects a Disallow rule in robots.txt that names it specifically.
The same is true of the other big named crawlers — Anthropic's ClaudeBot,
Google's Google-Extended. This is worth being precise about, because it is
an industry norm, not a legal requirement. Nothing compels a crawler operator
to honor robots.txt; a rule aimed at a named user-agent can in principle be
ignored, or the crawler could lie about its own identity. But the major, named
crawlers have an established track record of compliance, and it's checkable —
you don't have to take anyone's word for it. Pull your own server logs and
look for the user-agent string after you make the change; if requests from it
stop, the rule is being honored. If you want the full list of every named
crawler and the rule that matches each one, see our
AI crawler cheat sheet — this post is the
decision framework for one crawler on that list.

Given that it works, what you actually get is narrower than most people
assume going in:

You opt your pages out of being used as training data going forward, starting whenever OpenAI's crawler next respects the rule.
You get nothing retroactive. If your content was already fetched in a past crawl and used in a training run that already happened, blocking the crawler today does not undo that. There is no unlearning button on the other end of a robots.txt edit.
You get no change to ChatGPT search or citations — covered above, worth repeating because it's the thing people actually came here worried about.
You get no change to Google Search — GPTBot has nothing to do with Google, and blocking it doesn't touch your rankings there.
You get no change to any other AI product's retrieval — Perplexity, Claude's web search, anything else that fetches your page live to answer a question is a separate crawler making a separate request, governed by whatever rule you have (or don't have) aimed at it.

So the actual decision on the table is small and specific: do you want this
one company not training on this content from this point forward. Everything
else people worry about when they think about this — search visibility, being
"in ChatGPT," getting delisted somewhere — is not actually affected by this
switch.

The case for blocking it

The underlying reason opt-out mechanisms like this exist at all is a real,
ongoing disagreement in the industry: some publishers and rights-holders have
raised copyright and compensation concerns about AI companies training models
on their content without consent or payment. That's been a live, publicly
reported point of contention between publishers and AI companies, and it's
the actual origin of both the training/retrieval split and the
robots.txt-based opt-outs that OpenAI, Anthropic and Google all support.
Whether training on public web content should require consent or payment is
not a question this post is going to answer for you — it's genuinely
unsettled, reasonable people land on different sides, and your view on it is
a fine reason to block the crawler on its own, independent of anything else
in this post.

If that's your reason, block it. You don't need a further business case. It's
a values call about your own content, and it's yours to make.

The case for not blocking it

The case against is more speculative, and worth stating as speculative rather
than dressing it up as a strategy.

Some founders and publishers want their product to be part of what a model
"knows" about when it's later asked general questions — the idea being that
training data shapes a model's baseline familiarity with a topic or brand,
which is a different thing from live retrieval and citation. That's a
plausible-sounding motivation. It is not a proven one. We've written before,
in what makes AI engines cite a source,
about how thin the actual evidence is for anything that reliably causes a
model to recommend or cite a specific product — and being present in a
training run is a different, further-upstream bet than the citation question
that post covers, with even less established evidence behind it. Nobody has
shown that a product being in GPTBot's training data causes ChatGPT to
mention that product later. It might contribute something to how the model
talks about a category in general terms. It might do nothing measurable at
all. There's no study to point at either way.

So "leave it open in case it helps the model know about us" is a bet, not a
fact, and it should be weighed as one — against a concern (uncompensated
training use of your content) that, for some site owners, is not speculative
at all.

The middle path both companies explicitly support

Most people asking this question don't actually want an all-or-nothing
choice, and they don't have to make one. OpenAI and Anthropic both explicitly
support splitting training from retrieval at the crawler level, because
they operate them as separate user-agents in the first place:

OpenAI: disallow GPTBot (opt out of training), explicitly allow OAI-SearchBot (stay eligible for citation in ChatGPT's answers) and ChatGPT-User (stay reachable when a person pastes your link into a chat).
Anthropic: the same shape — disallow ClaudeBot, allow Claude-SearchBot and Claude-User.
Google: disallow Google-Extended if you want to opt out of Gemini/AI feature training, but never touch ordinary Googlebot — that one indexes you for Google Search itself, and blocking it removes you from Google entirely, which is a completely different and much larger consequence than anything else in this post.

This is the setting most sites end up at once they understand the split:
opted out of training, still eligible to be cited, still reachable by a
person with a link. It isn't a compromise so much as just correctly stating
what you want, at the level of granularity the crawlers actually offer.

How to decide, in practice

Skip the philosophy question for a second and ask two narrower ones:

Do I have an actual objection to my content training a model, independent of any effect on my visibility? If yes, disallow GPTBot — that's the whole ask, and blocking it delivers exactly that with no side effects on search or citation.
Am I blocking it because I think it'll somehow hurt me if I don't, or help me if I do, in terms of getting found? If that's the actual reason, go re-read the mechanics above — there isn't one. Training and retrieval don't talk to each other. Decide on the merits of the first question instead, because the second one isn't actually load-bearing here.

If you're not sure which of those you are, that uncertainty is itself useful
information: it usually means the real answer is "block GPTBot, allow the
search and user-agent crawlers," and move on. It costs you nothing you were
actually using, and it settles the one part of this that was a genuine,
values-based decision rather than a guess about SEO mechanics.

Getting found in the first place is a separate problem from any of this — one
more listing that names your product consistently is a small, low-cost input
to that, and submitting to LetsLaunch takes a few minutes if you
want one.

Does blocking GPTBot remove my site from ChatGPT?

No. GPTBot collects training data; ChatGPT's search and citation feature is
powered by a separate crawler, OAI-SearchBot. Blocking GPTBot opts you out
of training runs and has no effect on whether you can be cited in ChatGPT's
answers, because that depends entirely on OAI-SearchBot staying allowed.

Does blocking GPTBot undo past training?

No. A robots.txt rule only affects future crawls. If your content was
already fetched and used in a training run before you added the disallow
rule, blocking the crawler today does not remove it from a model that has
already been trained.

Is GPTBot legally required to respect robots.txt?

No. Respecting robots.txt is an industry norm, not a legal obligation —
nothing compels a crawler to honor a disallow rule aimed at it, and a rule
naming a specific user-agent could in principle be ignored or spoofed. In
practice, the major named crawlers, including GPTBot, ClaudeBot and
Google-Extended, have a documented track record of compliance, and you can
verify it yourself by checking your own server logs for the user-agent after
adding the rule.

What Actually Gets You Cited by ChatGPT and Perplexity? Reading the Research

Costin Gheorghe — Sat, 12 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

Search "how to get cited by ChatGPT" and you'll find no shortage of people
selling a method. Almost none of them link to a study. The ones that do
usually link to the same one, and usually overstate what it found.

We went and read it, plus what's been published since. This is the honest
version: one real result worth knowing, a real and important gap in what it
proves, and a short list of things that correlate with being named by an
assistant — stated as correlations, because that's all any of this currently
is.

The one study doing real work here

In November 2023, researchers from Princeton, the Allen Institute for AI,
Georgia Tech and IIT Delhi published GEO: Generative Engine Optimization —
the paper that put a name to this whole field. It later appeared at KDD 2024.

They built GEO-BENCH, a set of roughly 10,000 queries across nine domains, and
tested nine content-editing strategies inside a generative-engine prototype to
see which ones changed how often a source got cited in the answer. Two
strategies stood out clearly above the rest: adding quotations from
authoritative sources, and adding relevant statistics. Both produced
visibility gains in the range of 22-41% depending on domain. Citing sources
within the content helped too, by a smaller margin. Keyword stuffing — the
one tactic everyone already knew how to do from a decade of classic SEO —
measurably hurt, performing worse than making no changes at all.

That's a genuinely useful finding, and it's the most rigorous data point in
this entire field as of writing.

The gap almost nobody mentions

Read that paragraph again and notice what it does not say. It does not say
"we changed a real webpage and ChatGPT started citing it." It says a
benchmark, built by the researchers, running against a generative-engine
prototype, showed a visibility lift when content included quotes and
statistics.

That's a controlled measurement of a mechanism, not a field result. It tells
you that language models, when generating an answer, respond to the presence
of quotable, specific claims in candidate source text — which is a
believable and useful thing to know about how these systems work. It does not
tell you that adding a statistic to your homepage will get you cited by the
ChatGPT and Perplexity people actually use, which run different retrieval
pipelines, different ranking layers, and different citation logic than a 2023
research prototype, and which nobody outside those companies can fully
observe.

Most of the "GEO checklist" content published since 2024 quietly drops that
distinction and presents the benchmark's percentages as if they were measured
on production ChatGPT. We're not going to do that. The mechanism is real and
worth designing content around. The specific percentage is not a promise
about your specific page.

What's observable about real platforms, separate from the study

A few things about how ChatGPT, Perplexity and Google AI Overviews actually
pick sources are visible from the outside, without needing lab access:

They don't all pull from the same pool. Independent analyses of what gets cited show ChatGPT leaning heavily on general reference sources like Wikipedia, while Perplexity leans toward forum and discussion content like Reddit, with a strong preference for recent material. These are different retrieval habits, not one "AI search" behaviour.
Ranking well in Google is not the same contest. Multiple 2026 cross-platform studies put the overlap between an assistant's cited sources and Google's own top-10 results well under half, with estimates ranging roughly from one in eight to just over one in three depending on methodology and query type. The studies disagree on the exact number because they measure different platforms and query sets — but they agree on the direction: doing well in classic search does not reliably predict getting cited by an assistant. It's a related contest, not the same one.
Being readable is a precondition, not a strategy. None of the above matters if the crawler behind the assistant can't retrieve your content in the first place — see our breakdown of what actually makes a page invisible to AI crawlers if you haven't confirmed yours can be read.

What we're not going to tell you

Some things routinely get claimed in this space with no study behind them at
all, and we'd rather say so than repeat them:

That FAQ schema increases citation rate. We've seen no evidence for it —
structured data has real, separately-documented uses in classic search, but
nothing published shows it moving AI citation.

That a directory listing, ours or anyone else's, causes an assistant to cite
you. It's an obvious thing for a directory to want to be true, which is
exactly why it deserves suspicion instead of a marketing page. G2 and
Capterra carry more domain authority than almost any product page that could
ever link to them, and receive close to no AI citations — if raw authority
were the mechanism, they'd dominate every category. They don't.

That an "AI visibility score" from some tool measures anything stable. Asking
a model whether it knows a product and recording the answer isn't
measurement — between 9% and 28% of such answers flip on a repeated identical
prompt, even at temperature zero. A single check dressed up as a score out of
100 is closer to a coin flip than a metric.

What's actually worth doing with limited certainty

Given all of the above, here's what survives: make sure your content is
retrievable at all — no JavaScript-gated text, correct robots rules, real
HTML, checkable in about a minute. Where you do
write about your own product, prefer specific, quotable claims and real
numbers over adjectives — the one mechanism with an actual controlled
measurement behind it. Describe the product the same way across every page
that mentions it, your own site included, since consistent description
correlates with being named, even though nobody has shown which particular
page causes it. And be skeptical of anyone selling you a guaranteed
mechanism, including us, when we don't have receipts for the claim.

What makes ChatGPT or Perplexity cite a source?

The most rigorous evidence is a 2023 Princeton study (GEO, published at KDD
2024) that measured a 22-41% visibility lift inside a research benchmark from
adding quotations and statistics to content, with keyword stuffing performing
worse than no change at all. That's a real, controlled finding about how
generative engines weigh content — but it was measured on a lab benchmark, not
on production ChatGPT or Perplexity, so treat the mechanism as real and the
exact percentage as not transferable to your page.

Does ranking well in Google get you cited by AI assistants?

Not reliably. Multiple 2026 studies comparing AI-cited sources against
Google's top-10 results found overlap well under half, though the exact
figure varies by study and platform. Classic SEO and AI citation appear to be
related but distinct contests — being crawlable and readable matters to both,
but ranking highly in one doesn't predict success in the other.

Is generative engine optimization (GEO) a proven discipline?

Partially. One mechanism — content with specific quotes and statistics
getting cited more than vague or keyword-stuffed content — has real,
peer-reviewed measurement behind it, from a controlled benchmark. Most other
claims circulating under the "GEO" label, including FAQ schema and directory
submissions as citation drivers, currently have no published evidence behind
them at all.

Wikipedia and Reddit Aren't One Strategy — They're ChatGPT's and Perplexity's Different Bets

Costin Gheorghe — Fri, 11 Sep 2026 17:20:00 +0000

Originally published on LetsLaunch.

Ask ChatGPT about almost anything and there's a decent chance the answer leans
on a page you didn't write, on a site you don't control, that has never heard
of your product. Wikipedia and Reddit show up so often in these systems'
citations that the pattern isn't really a curiosity anymore — it's closer to
structural.

We've written before about what the actual research says about getting
cited by AI engines, and the honest
answer there was mostly "less than the industry claims, and mostly
correlational." This post is narrower. It's not about how to write content
that gets cited. It's about where these systems are pulling from in the first
place — and what that structural fact does and doesn't leave open to an
independent founder with no Wikipedia page and no Reddit history.

The numbers, and who's reporting them

5W, an AI communications firm, published a synthesis called "The State of AI
Citations 2026," covered by PR Newswire as analyzing more than 680 million
tracked AI citations across ChatGPT, Claude, Perplexity, Gemini and Google AI
Overviews. Two findings from it are worth sitting with.

First: Wikipedia and Reddit together account for more than 25% of all
ChatGPT citations in the US. Not a niche source. More than a quarter of
everything the most widely used assistant cites, concentrated in two
platforms neither of which is a news outlet, a review site, or anything a
product team would normally target.

Second, and more pointed: none of the Wall Street Journal, the New York
Times, or Bloomberg appear in ChatGPT's top 20 cited domains. Three of the
most authoritative news organizations in the English language, by any
conventional measure of editorial rigor or domain authority, don't crack the
top 20. If you've been operating on the assumption that getting into a major
outlet is the top of the AI-citation food chain, this is worth updating on.

Other 2026 citation-tracking analyses — methodologies vary enough between
them that we're not going to force a single number onto a single source — have
separately put Wikipedia's share of ChatGPT's own top-10 cited sources at
roughly 48%. Call it directionally consistent with 5W's finding rather than
the same measurement: however you slice it, Wikipedia is not one source among
many for ChatGPT. It's the biggest one, by a wide margin, in more than one
independent look at the data.

Wikipedia and Reddit are not the same bet

Here's the part that a lazy read of the above skips past, and it matters more
than the headline numbers.

Wikipedia's dominance is not uniform across assistants. One comparative
analysis found ChatGPT cites Wikipedia for roughly 12% of its own citations,
Claude cites it for roughly 0.1%, and Perplexity doesn't cite Wikipedia in any
meaningful volume at all. Perplexity's citation behavior looks almost
inverted: the same analysis put Reddit at roughly 47% of Perplexity's own
top-10 source share — Perplexity's Wikipedia-equivalent, structurally, is
Reddit.

So "get on Wikipedia" and "get into Reddit" are not the same advice wearing
two names. They're bets on two different platforms that happen to both
concentrate around user-generated, non-brand-controlled content, and a tactic
aimed at one does close to nothing for the other.

This lines up with something we said plainly in the GEO research
post: these are different
retrieval systems, not one "AI search." Independent analyses found only about
11% of domains get cited by both ChatGPT and Perplexity. If you optimize for
one assistant's citation habits, you have done close to nothing for the
other's, and Wikipedia versus Reddit is the clearest illustration of that
split we've seen yet.

It's also worth separating this from a different problem entirely: none of
this matters if the crawler behind the assistant can't retrieve your page's
content to begin with. That's a rendering and access problem, not a source-mix
problem, and we've covered it separately in why ChatGPT can't see your
site. Fix that first regardless of
anything below — it's the one part of this that's fully mechanical and fully
in your control.

Why you can't just "get on Wikipedia"

You cannot get your own product a Wikipedia page on demand, and this isn't a
process gap that a clever workaround fixes — it's the intended design.
Wikipedia's notability and neutral-point-of-view rules exist specifically to
resist self-promotion. A page created, or heavily edited, by someone connected
to the subject is routinely flagged and deleted, and the volunteer editors who
patrol new pages are reasonably good at spotting the pattern: a founder,
employee, or paid editor writing about their own product in the neutral third
person is one of the most common deletion categories on the site.

The honest path to a legitimate Wikipedia mention runs the other direction
entirely. You get covered by independent, reliable secondary sources — real
press, not a paid placement, not a sponsored post — and then, sometimes, a
Wikipedia editor with no connection to you decides that coverage is enough to
justify an entry, and writes it. The Wikipedia mention is a lagging effect of
real independent coverage existing, not something you pursue directly. If
someone offers to get your product onto Wikipedia for a fee, what they're
actually offering is a page that gets deleted once a volunteer editor notices
the conflict of interest — which is most of them, most of the time.

Reddit is reachable, but only slowly and only if you mean it

Reddit is the more directly reachable one of the two, but the qualifier
matters: reachable as participation, not as marketing.

What actually produces the kind of discussion thread that later becomes
citation fodder is answering real questions in relevant subreddits, over
time, from an account with a real history of doing that — not one created
the week your product launched. A first post that reads as a product pitch
gets read as spam, both by the subreddit's human moderators and, per Reddit's
own well-documented moderation norms, often by automated filters before a
human even sees it. It gets removed before anything resembling a citable
thread ever exists, let alone before a crawler could find it.

There's no shortcut here that survives contact with how Reddit actually
moderates itself. Genuine participation compounds slowly, over months, and
looks like actually being present in a community rather than treating it as a
distribution channel with a login page.

What this doesn't change

None of the above is a mechanism you can point at and expect a result from.
It's the same caveat we made in the companion post on citation research: this
is the honest, sourced shape of the problem as it currently stands, not a
guarantee about what happens to your specific product if you do these things.
Anyone offering to get you cited on Wikipedia or Reddit as a paid, one-off
deliverable is not offering a real mechanism — they're offering something that
either gets deleted (Wikipedia) or removed by moderators (Reddit) once anyone
looks at it closely.

What's actually within reach, and costs you nothing beyond the time to do it,
is the same low-effort thing we've pointed to before: describing your product
consistently across whatever independent pages do exist about it, including a
LetsLaunch listing if you want one more page that names your
product the same way your own site does. We're not claiming that causes a
citation — nobody has shown that, and we said so plainly in the last post too.
It's unproven, but it's also free, which is a different bar than the one
Wikipedia and Reddit ask you to clear.

Why does Wikipedia show up so often in ChatGPT's answers?

Because ChatGPT's own citation mix leans heavily on it. A 5W synthesis of over
680 million tracked citations found Wikipedia and Reddit together make up more
than 25% of ChatGPT's US citations, and separate analyses have put Wikipedia
alone at around 48% of ChatGPT's top-10 cited sources — making it the single
largest source by a wide margin. That's a property of how ChatGPT's retrieval
currently weighs general reference content, not a signal that Wikipedia
coverage is easy to obtain or that it transfers to other assistants.

Does the same Wikipedia-and-Reddit pattern apply to Perplexity?

Not evenly. One comparative analysis found ChatGPT cites Wikipedia for
roughly 12% of its citations while Claude cites it for roughly 0.1%, and
Perplexity doesn't cite Wikipedia in meaningful volume at all — Perplexity
instead leans on Reddit, reported at roughly 47% of its own top-10 source
share. Wikipedia and Reddit are two separate, platform-specific patterns, not
one rule that applies across every assistant.

Can I get my product added to Wikipedia to improve AI citations?

Not directly, and not on request. Wikipedia's notability and neutral-point-of-
view rules are built to resist self-promotion, and pages created or heavily
edited by someone connected to the subject are routinely deleted once a
volunteer editor notices. The realistic path is being covered by independent
press first — a Wikipedia mention, where it happens at all, follows real
outside coverage rather than preceding it.

How to Write a Paragraph an AI Engine Can Actually Quote

Costin Gheorghe — Fri, 11 Sep 2026 08:20:00 +0000

Originally published on LetsLaunch.

We wrote up the actual research
on what correlates with an AI engine citing a source, and the honest version
is shorter than most "GEO checklist" posts want it to be: one controlled
benchmark found that quotations and statistics got cited more than vague
content, and keyword stuffing measurably hurt. That's a real mechanism,
observed in a lab, and it's the most solid data point that exists.

This post is not about that research. It's about what to actually do with it
— the writing-craft side, for someone sitting down to write a paragraph today.
Not a proven-percentage technique, because we don't have one to sell you. Two
mechanical, checkable habits that follow honestly from what's known, plus one
you should adopt for a much older reason: it makes your writing better for
the person reading it, independent of whatever ChatGPT does with it.

Why structure matters even without a proven number

An extraction system — whether it's Google building an AI Overview,
Perplexity assembling an answer, or ChatGPT summarizing a page it retrieved —
is doing something specific: pulling a passage out of your page and presenting
it with reduced or no surrounding context. It might show one paragraph. It
might show one sentence. Whatever it shows has to stand on its own, because
the rest of your page usually doesn't travel with it.

That's true regardless of what the exact citation-rate lift is, which nobody
outside a handful of labs has measured on production systems. What you can
reason about, without needing a study, is whether a given passage survives
being extracted. Either it still makes a complete, correct claim once it's
alone on the screen, or it doesn't. That's a property of the text, checkable
by reading it, not a promise about traffic.

The test: copy it into a blank document

Here's the mechanical version, and you can run it on anything you've already
published.

Take one paragraph. Copy it alone into a blank document — nothing before it,
nothing after it. Read it as if you'd never seen the rest of the page. Ask:
does this still make a complete, correct, unambiguous claim?

If yes, the paragraph is structurally extractable. If it depends on "as
mentioned above," a pronoun with no visible referent, or a claim the previous
paragraph set up but this one doesn't restate, it isn't — and an extraction
system either garbles it, skips it, or quotes something that reads as
confused or wrong when separated from its context. None of that requires
knowing anything about how any particular model works. It's the same test a
human editor would run on a pull-quote.

A before-and-after

Here's a paragraph that fails the test, from a page describing a fictional
product:

It handles this automatically, which saves teams a lot of time compared to
the old way of doing it. That's part of why customers tend to stick around
longer once they switch.

Read alone, this says almost nothing. What does "it" do? What's "the old
way"? What's "a lot of time"? "Tend to stick around longer" than what, by how
much? Every noun refers outward to a sentence that isn't there anymore. A
person skimming a summary would come away with no actual information, and an
extraction system quoting it verbatim would be quoting a sentence that reads
as vague at best and misleading at worst — it implies a specific comparison
without stating one.

Here's the same claim, rewritten to stand alone:

Acme's scheduling tool re-assigns a missed shift automatically within two
minutes, instead of a manager finding a replacement by phone. Teams that
switched from manual scheduling report needing about 30% less manager time
per week on shift coverage.

Now it names the product, states the mechanism, gives a concrete number, and
makes a specific comparison. Someone could read only this paragraph, with
nothing before or after it, and walk away with a correct and complete
understanding of the claim. That's the whole test — not length, not keyword
density, just whether the sentence still works with everything around it
removed.

Notice this rewrite also happens to be the kind of claim GEO-BENCH found
correlated with more citations in its benchmark: specific, quotable, backed
by a number rather than an adjective. That's not a coincidence — a claim that
stands alone and a claim that's specific enough to quote tend to be the same
sentence. But we're stating that as a plausible connection between two
documented properties, not as a guarantee that this exact rewrite will get
you cited anywhere.

Put the answer before the build-up

The second habit: when a section implicitly answers a question — "how much
does this cost," "does it support SSO," "how long does setup take" — state
the answer in the first sentence of the section, then explain or qualify it
afterward. Not the reverse, where the section opens with context and works
toward a reveal at the end.

This isn't from a measured percentage. It's a documented, widely-recommended
practice in web writing generally, for a plain reason: both a human skimming
your page and a system scanning for a relevant passage are doing the same
thing — looking for the sentence that answers the question, not reading in
order for a payoff. A reader who wants to know if you support SSO and finds
that answer buried after three paragraphs of company background has a worse
experience whether or not any AI ever touches the page. Front-loading the
answer serves the actual reader first. That it also gives an extraction
system a clean, early sentence to lift is a second, plausible benefit — not
the reason to do it.

Compare:

Our platform was built from the ground up with enterprise needs in mind,
and over the years we've worked closely with security teams at companies of
every size to refine an authentication experience that meets a wide range
of compliance requirements. SSO is supported.

against:

Yes, this supports SSO, via SAML and OIDC. It's configured in Settings →
Security, and works with Okta, Azure AD and Google Workspace out of the
box.

The second version answers the implied question in its first four words. Everything
after that is detail a reader can keep going for or stop reading, having
already gotten what they came for.

What this doesn't promise

Neither habit above comes with a measured lift on real AI engines, and we're
not going to invent one. You may come across posts claiming a specific
percentage improvement from "answer-first" writing, sometimes down to one
decimal place, or a claim that answers need to be a precise word count to
count as citable. We looked. We couldn't trace either kind of number back to
a named study, a named researcher, or any data anyone could check — just
marketing content citing other marketing content. A blog built on "verified,
not estimated" doesn't get to make an exception for a number just because
it's specific-sounding. So we're leaving both out, and we'd suggest treating
any post that states one with unusual precision and no source the same way.

What we will say: self-contained paragraphs and answer-first sections are
good writing by any standard that predates language models entirely. They
make content easier to skim, easier to excerpt in an email, easier to quote
in a meeting, easier for a tired reader at 11pm to get what they need from
without reading the whole page. If a generative engine also finds that shape
easier to extract cleanly — which the one real study we trust suggests is at
least plausible, since specific and self-contained claims are close cousins —
that's a reasonable bonus on top of writing that was already worth doing. It
is not, and we're not going to pretend it is, a guaranteed technique with a
number attached.

None of this works if the system pulling the passage can't reach your page in
the first place — see why ChatGPT can't see your
site if you haven't confirmed yours is
readable by a crawler before worrying about how the sentences are shaped.

And if you're publishing a product description anywhere public, including
a LetsLaunch listing, the same test applies to that paragraph too
— read it alone, with no page around it, and see if it still holds up.

Does writing self-contained paragraphs actually increase AI citations?

Nobody has published a study measuring that specific effect on production AI
engines, so we can't claim it does. What's documented is a related, narrower
finding: content with specific quotes and statistics got cited more than
vague content inside a Princeton benchmark. Self-contained, specific writing
is a reasonable, checkable habit that's consistent with that finding and
also makes content easier for a human to skim — not a proven technique with
a measured percentage behind it.

What's the quickest way to check if a paragraph is AI-citable?

Copy it alone into a blank document, with nothing before or after it, and
read it as a stranger would. If it still states a complete, correct,
unambiguous claim with no missing context, it's structurally extractable. If
it depends on a pronoun, "as mentioned above," or a fact the previous
sentence supplied, it isn't — rewrite it to carry its own context.

Should I put the answer first or build up to it?

Put it first, then explain. This isn't based on a measured citation
percentage — it's a documented practice grounded in how both readers and
extraction systems scan text: looking for the sentence that answers the
question, not reading in narrative order. A reader who wants to know if you
support SSO benefits from the answer in the first sentence whether or not any
AI system ever touches the page.