DEV Community: Mike

How to appear in ChatGPT answers (the full 2026 playbook)

Mike — Thu, 04 Jun 2026 18:13:41 +0000

Originally published at brandswarm.io/blog/how-to-appear-in-chatgpt-answers/.

ChatGPT (and Claude, Perplexity, Gemini, AI Overviews — but most of this
applies to all of them) picks brands the way a journalist picks sources:
by what shows up in its retrieval layer, how that content is structured,
and how often other respected pages back it up. There is a playbook, and
unlike a lot of SEO advice, this one actually works in 90 days if you do
the work.

This is the long version. If you want the 60-second version, the
high-leverage ranked list is:

Stop blocking AI crawlers at your CDN.
Get your domain indexed by Bing.
Publish category-positioning content (comparison, alternatives, listicles).
Get cited by third parties in your space.
Ship structured data (Organization, Product, FAQ).
Win the wedge prompts before the head terms.
Track + iterate every week.

We'll walk through each of those, in order.

The mental model: training vs retrieval

Every modern AI assistant pulls answers from two places: its training
data (a frozen snapshot of the internet from a year or so ago) and
real-time retrieval (a live search at the moment of the
query). The mix differs by engine:

ChatGPT: retrieval via Bing when the question seems factual or recent; training data otherwise.
Claude: training data primarily; web tool when explicitly invoked.
Perplexity: retrieval almost always (this is the whole product).
Gemini: retrieval via Google Search; training data fallback.
AI Overviews: retrieval via Google Search, then synthesis.
Google AI Mode: retrieval-heavy, often with Reddit + community sources weighted.

The practical implication: if you want to appear in answers this
quarter, you're optimizing for the retrieval layer. Training-data
inclusion happens on a 6–18 month lag and is worth playing the long game on,
but the high-leverage work is retrieval.

Part 1: Make sure you're crawlable

1.1 Audit your robots.txt

Pull https://yourbrand.com/robots.txt and look for these
user-agents:

GPTBot — OpenAI's crawler. Allow.
ClaudeBot — Anthropic's crawler. Allow.
Google-Extended — Google's AI/Gemini training crawler. Allow if you want training-data inclusion; doesn't affect AI Overviews (those use the regular Googlebot).
Bytespider — ByteDance's crawler. Allow if you want Doubao / TikTok-side coverage.
PerplexityBot — explicitly allow.
CCBot — Common Crawl. Allow (used by many models as a fallback training source).
Amazonbot, Applebot-Extended, meta-externalagent — allow if you want full coverage; lower priority than the above.

If any of these are Disallow: /, that's almost certainly your
Cloudflare zone applying its "AI Audit" template by default. Override it in
the Cloudflare dashboard (Bots → AI Audit → Allow).

1.2 Get into Bing's index

ChatGPT's browsing tool uses Bing. If you're not in Bing's index, ChatGPT can't
pull from you in real time.

Verify your domain at bing.com/webmasters.
Submit your sitemap.
Use Bing's URL submission API to push your most important pages (pricing, comparison pages, top blog posts).
Watch "Discovered but not indexed" in Bing's Index Explorer. The "Why?" column tells you the exact reason for non-indexing.

1.3 Get into Google's index

This is your AI Overviews + Gemini channel. Verify at Google Search Console,
submit sitemap, monitor Coverage. Most teams already have this dialled in;
if you don't, this is table stakes.

Part 2: Be the kind of content AI engines retrieve

AI engines retrieve content that discusses categories, not just
brand-promotional content. Three page types disproportionately get pulled:

2.1 The "Best X for Y" listicle

Write the listicle for your own category. Be honest about it. List
competitors. Position yourself somewhere in the middle. Add a comparison
table at the end. Update it quarterly.

Yes, this feels self-defeating ("why would I write about my competitors?").
Two reasons: (1) AI engines will quote you when users ask the comparison
question, and (2) writing about a category fairly is the strongest
authority signal in 2026 retrieval models. Pages that read like sales
pitches get retrieved less than pages that read like reviews.

2.2 The "[Competitor] alternatives" page

One page per major competitor. Each page lists 3–5 alternatives, including
you, with honest comparison. These rank for high-intent
"[competitor] alternative" searches in Google and get cited heavily in
Perplexity and AI Overviews because they answer a literal common
question with structured comparison data.

2.3 The "X vs Y" head-to-head

Pick your top 3 competitors. Write a head-to-head for each one, with you
as one side. Be specific (features, pricing, who-it's-for, what each
one is bad at). These convert in three ways: rank in Google's traditional
search, get cited in AI engines, and convert directly when a buyer is
comparing options.

2.4 The buyer's guide

"How to choose a [category]" — a 2,000–4,000 word evaluation framework
for buyers in your category. Don't recommend yourself in the guide; just
give a real framework. AI engines retrieve these heavily for "what should
I look for in X" questions.

2.5 Original-data posts

AI retrieval layers reward novel data. If you have a unique dataset (we have
AI-visibility scores; you might have product-usage stats, industry
benchmarks, survey results), publish it. A single original-data post
attracts 2–4× the backlinks of an ordinary blog post, and those backlinks
feed back into retrieval ranking. This is the highest-leverage content
asset you can build if you have any unique data to publish.

Part 3: Be cited by third parties

Your own content gets you partial coverage. Third-party citations get you
the rest. AI engines weight third-party mentions heavily — it's the same
reason they read like a Wikipedia-style synthesis: they're synthesizing
multiple sources, and brands that appear across multiple sources get
weighted higher.

3.1 The eight high-weight citation sources

In rough order of impact for most B2B SaaS:

Wikipedia — single highest-weight source. Get there when you meet notability guidelines (covered by independent reliable sources, multiple). This is a long-term play and you can't shortcut it, but it's the single biggest needle-mover.
Reddit — increasingly weighted, especially in Google AI Mode and Perplexity. Authentic Reddit threads in your category drive direct citations.
Hacker News — high-trust source for technical audiences. A successful Show HN can move your retrieval-layer presence in days.
G2 / Capterra / Product Hunt — category aggregators routinely retrieved for "best X" queries.
Industry publications — pick yours (TechCrunch, The Verge, SearchEngineLand, Backlinko for SEO tools, Lenny's for PM tools, etc.).
Comparison sites — AlternativeTo, SaaSworthy, Slant — niche per-category aggregators.
Substacks + newsletters with engaged audiences — sponsorships and earned mentions in respected newsletters become citation sources.
YouTube transcripts — yes, really. The transcripts of well-watched YouTube videos in your category are crawled and feed into retrieval. A single featured spot in a major reviewer's video can drive AI citations for months.

3.2 The minimum-viable citation campaign

For a brand-new SaaS in 2026, the realistic 90-day citation goal is:

One feature in a category-relevant publication (earned, not paid).
Three substantive Reddit threads in your category subs where you're named in the discussion (organically — never pitch yourself; participate helpfully for months first).
One Show HN / one Product Hunt launch.
Listings on the three biggest aggregators in your space.
10–20 backlinks from category-relevant sites you've earned through guest posts, sponsorships, or being a useful interview subject.

This is enough citation breadth to start showing up consistently in
retrieval-based AI answers.

Part 4: Make your structured data unmissable

Structured data is the difference between an AI engine guessing what your
page is and knowing what your page is. Three blocks of JSON-LD do most of
the work.

4.1 Organization schema (on every page, or at least the homepage)

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "YourBrand",
  "url": "https://yourbrand.com",
  "logo": "https://yourbrand.com/logo.png",
  "description": "One-sentence category positioning.",
  "foundingDate": "2024-01-15",
  "sameAs": [
    "https://twitter.com/yourbrand",
    "https://linkedin.com/company/yourbrand",
    "https://github.com/yourbrand",
    "https://en.wikipedia.org/wiki/YourBrand"
  ]
}

The sameAs array is doing the heaviest lifting here — it
associates your domain with your other web presences and helps the AI
retrieval layer build a consistent identity graph.

4.2 Product / SoftwareApplication schema (on product pages)

{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "YourBrand",
  "applicationCategory": "BusinessApplication",
  "applicationSubCategory": "AI Brand Monitoring",
  "operatingSystem": "Web",
  "offers": {"@type": "Offer", "price": "49", "priceCurrency": "USD"}
}

4.3 FAQPage on pages with question/answer pairs

Your billing FAQ, your "how does this work?" doc, your blog posts with
H3 questions — all of these benefit from FAQPage JSON-LD.
It's the most-retrieved structured-data type in AI Overviews and a
surprisingly significant share of Perplexity citations.

Part 5: Win the wedge prompts

Don't try to win "best CRM" if you're a 6-month-old CRM startup. Win
"best CRM for Israeli accountants" or "cheapest CRM with Zapier integration"
first. These prompts have lower competition, higher conversion intent, and
give you a foundation of wins that compound.

How to find your wedge prompts:

Run a free AI-visibility scan on your category's head term. See who wins and who doesn't.
Brainstorm 20 "[role-of-buyer] looking for [niche-need-in-category]" queries.
Test each one in ChatGPT and Perplexity manually. Note which ones already mention you weakly (where you can push to #1 with a content push) and which ones don't mention anyone (where you can be first).
Pick 5 wedges. Concentrate content + citation outreach on those for one quarter.

Part 6: Track + iterate weekly

Without weekly tracking, you have no idea what's working. AI search shifts
fast (model updates can reshuffle answers overnight), and the only way
to know whether your last content push moved your score is to measure.

Use any of the AI-visibility tools we covered in our
Best AI Visibility Tools 2026
roundup. The free instant scan in the footer of this post is a fine starting
point; the daily-monitoring tools are worth their cost if you're
actively optimizing.

The 90-day execution plan

Days 1–7: Foundation

Audit robots.txt + unblock AI crawlers (CDN dashboard).
Verify domain in Bing Webmaster Tools + Google Search Console.
Ship Organization + FAQPage + Product schema on home + pricing.
Run a baseline AI-visibility scan to measure starting point.

Days 8–30: Content

Publish one "Best X for Y" listicle for your category.
Publish one "[top-competitor] alternatives" page.
Publish one "X vs Y" head-to-head against your closest competitor.
Pick 5 wedge prompts. Publish content targeting each.

Days 31–60: Citations

List on G2, Capterra, Product Hunt, AlternativeTo, Slant (and your category's specific aggregator).
Pitch one feature in a category-relevant publication.
Begin participating helpfully in 3–5 Reddit subs in your space (no self-promotion yet — build credibility for a quarter first).
Launch on Show HN or Product Hunt (when ready).

Days 61–90: Iterate

Re-scan weekly. Track which wedge prompts you've won and which haven't moved.
Double down on what's working. Replace what isn't.
Begin building toward Wikipedia notability (long-term play, but starts here).

FAQ

How long until I see results?

Crawlability fixes: 1–2 weeks. Content + structured-data wins on
long-tail prompts: 30–60 days. Authority gains on head terms: 90+ days.
Training-data inclusion in next model generation: 6–18 months.

Do I need to do all of this?

No. Crawlability (Part 1) is non-negotiable; without it nothing else
works. Beyond that, prioritize based on your situation: brand-new
startups should focus on Part 3 (citations) and Part 5 (wedge prompts).
Established brands with crawlability problems should focus on Part 1
and Part 4 (structured data).

What if my product is plain bad?

No AI-visibility playbook will fix that. The retrieval layer reflects
the discourse around your brand — if the discourse is "this product
doesn't work," the AI will surface that, and good. Fix the product
first.

Is there a paid shortcut?

Sponsored content in respected category newsletters is the closest
thing — pay for placement in a newsletter the retrieval layer trusts,
and you get a high-weight citation in days, not months. Avoid
PR-distribution wires (Newswire, PRWeb) — those are flagged as
low-quality by most retrieval models. Avoid pay-for-inclusion in
"best of" listicles run by content farms; AI engines specifically
downweight these sources.

One last thing

Most teams who follow this playbook for 90 days move from
"not mentioned anywhere" to "mentioned consistently on 3–4 surfaces
for our wedge prompts and sometimes on head terms." That's a
realistic outcome. Bigger jumps (head-term dominance) take longer,
but they're built on the same foundation. The honest answer to "how
do I appear in ChatGPT answers?" is the same honest answer for any
marketing channel: do the unglamorous work consistently, measure
weekly, and act on what the measurement tells you.

Why isn't my brand in ChatGPT? 7 reasons (and exact fixes)

Mike — Thu, 04 Jun 2026 18:12:10 +0000

Originally published at brandswarm.io/blog/why-isnt-my-brand-in-chatgpt/.

Most teams who walk in asking "why isn't my brand in ChatGPT?" have already
Googled the answer and found something vague about "increasing authority" or
"creating quality content." Useless. The answer is almost always one of seven
specific causes, and each one has a specific fix. Here they are, in roughly the
order we find them when we audit a brand for the first time.

If you'd rather diagnose your own brand in 60 seconds before reading further,
run a free Brandswarm scan — you'll see exactly which surfaces mention you and
which don't. Then come back here and look up the fix.

Cause #1: Your robots.txt or CDN is blocking AI crawlers

This is the most common cause by a wide margin, and the most embarrassing
one when teams discover it. In 2025, Cloudflare turned on its "AI Audit" /
"Block AI Bots" feature for many zones by default. The result is that
robots.txt on your site is silently disallowing GPTBot
(OpenAI), ClaudeBot (Anthropic), Google-Extended
(Gemini's training crawler), Bytespider (TikTok / Doubao),
Applebot-Extended, and others.

These crawlers don't all matter equally. ChatGPT's retrieval, for example, uses
Bing's index plus its own internal training data — so blocking GPTBot affects
ChatGPT less than you'd expect. But Perplexity, Claude, and Gemini all do
significant real-time retrieval from their own crawlers, and AI Overviews use
Google's regular index too. Blocking these will make you disappear from at
least some surfaces, sometimes all of them.

Fix:

Curl your own robots.txt. Look for Disallow: / under any of the AI user-agents above.
If you're behind Cloudflare: open the Cloudflare dashboard → your zone → AI Audit / Bots → toggle off the "Block AI Bots" rule, or override the "Cloudflare Managed Content" template entries.
Confirm by re-fetching robots.txt and looking for the user-agents above.
Wait 1–2 weeks for the AI retrieval layer to re-index you.

Cause #2: Your domain isn't in Bing

ChatGPT's web-browsing retrieval is powered by Bing. If your site isn't in
Bing's index, ChatGPT's browsing tool won't find it when a user asks
about your category. (This is separate from ChatGPT's training data, which
is a different pipeline and updates much less frequently.)

Even a brand with strong Google rankings can be invisible in Bing — Bing's
crawler is less aggressive, and a few common signals (parameters in URLs,
missing canonicals, conflicting hreflang) can keep an entire site out of the
Bing index.

Fix:

Verify your domain in Bing Webmaster Tools.
Submit your sitemap.
Use Bing's URL Submission API to push your most important pages immediately.
Check the Index Explorer for "Discovered but not indexed" — these are pages Bing has found but is choosing not to keep. The "Why?" column will tell you the exact reason.

Cause #3: Your content doesn't answer category questions

AI engines pick brands the way a journalist picks sources: they synthesize an
answer from the sources that already discuss the topic. If a user asks
"what's the best CRM for startups?", the AI doesn't crawl your homepage and
decide if your CRM is good — it looks for pages that discuss "CRM for startups"
and pulls the names that come up there.

Translation: if you don't have content that names your category and
positions you within it, you don't exist in AI search. This is the
most common content-side cause we see. Brands write a homepage like "we make
the best [product]" and then complain that AI doesn't mention them. AI engines
don't read marketing pages well; they read content that discusses the
category, including comparison pages, listicles, alternatives pages,
and third-party reviews.

Fix: Publish at least one piece of content for each of:

"Best X for Y" where X is your category and Y is your customer.
"[Competitor] alternatives" for each major competitor.
"X vs Y" head-to-head comparisons with named competitors.
"How to choose a [category]" buyer's-guide-style content.

Then make sure other publications in your space have similar content that
mentions you by name. AI engines weigh third-party citations heavily; getting
named in someone else's listicle is often a bigger lift than publishing your own.

Cause #4: No (or wrong) structured data

Schema.org structured data — Organization, Product,
FAQPage, HowTo, Article — helps AI
retrieval layers understand what your page is, not just what words
appear on it. Pages with strong schema get picked up more consistently in
AI Overviews and in Perplexity citations.

Most brands have either no JSON-LD at all, or thin WebSite +
BreadcrumbList markers and nothing semantically meaningful.
Some have incorrect JSON-LD that Google's structured-data testing
tool flags as broken, which can be worse than none at all.

Fix:

Add Organization schema with a sameAs array linking to your LinkedIn, X/Twitter, GitHub, and Wikipedia (if applicable).
Add SoftwareApplication or Product schema on your product pages, including name, description, applicationCategory, and pricing if appropriate.
Add FAQPage schema on any page with question/answer pairs.
Validate everything with Google's Rich Results Test before shipping.

Cause #5: You're a "new" brand from the model's perspective

Language-model training cutoffs lag reality by 6–18 months depending on the
model. If your brand launched in the last 18 months and you haven't built
significant retrieval-layer presence (third-party mentions, Wikipedia entry,
trusted-publication coverage), the model just doesn't know about you. The
browsing/retrieval layer can fill in the gap, but only if the
retrieval-layer cause #1 and #2 are clean.

Fix:

Get at least one citation from a high-authority publication in your category (TechCrunch, The Verge, Hacker News front-page Show HN, Lenny's, SearchEngineLand — pick yours).
Aim for a Wikipedia entry when you cross significant signal thresholds (notability under WP:NCORP or WP:NSOFT) — it's the single highest-weight citation source for most LLMs.
Get listed on category-aggregator sites (G2, Capterra, Product Hunt, AlternativeTo) — these are routinely retrieved when users ask comparison questions.

Cause #6: Your category competitors are dominating the prompts you care about

Even with great content and good citations, if a category has a clear top-three
(think Stripe / Square / Adyen in payments, or Notion / ClickUp / Asana in
productivity), it can be hard to break through on the most generic queries.
The fix isn't to compete on "best payment processor" — it's to compete on the
prompts you can actually win.

Fix:

Identify long-tail and "[role-of-buyer]" prompts you have a real shot at: "best Stripe alternative for crypto", "cheapest payment processor for nonprofits", "Adyen for marketplaces". These are searched less but won more easily, and they convert better.
Track them in your AI-visibility tool over time. As you accumulate wins, the model's overall sense of who you are improves.
Pick five "wedge" prompts and concentrate content + outreach on those rather than trying to win the head term.

Cause #7: Your sentiment is bad enough to suppress you

Less common but worth knowing: if your brand has accumulated enough negative
sentiment in the retrieval layer (a viral outage post-mortem, a notorious
G2 review, a Reddit thread), the model may technically know you but choose
to omit you or to mention you with a caveat. Sentiment-induced suppression
is real for a small number of brands.

Fix:

Track sentiment per surface in your AI-visibility tool. If one surface is consistently negative, find the source content driving it.
Where the source is correctable (an old support thread you can resolve, an outdated competitor comparison the writer would update), do the work.
Where it's not, drown it out: publish a higher volume of positive citation-worthy content that the retrieval layer will preferentially weight.

The diagnostic playbook

If you don't yet know which of the above is your problem:

Run a free scan across the five AI surfaces. See which ones mention you and which don't.
If no surfaces mention you → most likely cause #1 (crawlers blocked) or #2 (not in Bing). Check both first.
If only some surfaces mention you → cause #1 or #4 (structured data) per surface. The pattern of which surface fails will tell you which.
If you're mentioned but never on the head terms → cause #6 (competitors winning the head); pivot to wedge prompts.
If you're mentioned but the language is unflattering → cause #7 (sentiment); track the source.
If you're a young brand with sparse mentions → cause #5 (model doesn't know you yet); citation campaign.

FAQ

How long does it take to fix this?

The crawler/Bing fixes can show in 1–2 weeks. Content and citation work
takes 1–3 months to compound. Wikipedia entries (when applicable) can take
6+ months to be picked up by training cycles. Plan for a quarter.

Should I bother if I'm a B2C consumer brand?

Yes, but the prompts that matter are different. For B2C, watch
"what is [your brand]", "is [your brand] legit", "[your brand] vs
[competitor]", and category-level recommendation queries ("best running
shoes for flat feet"). Sentiment matters more in B2C than in B2B.

What if I'm a brand-new startup with no content yet?

Start by getting on category aggregators (Product Hunt, G2, Capterra,
AlternativeTo) — those are the first sources AI engines retrieve from when
asked about a category you're new to. Then build your own comparison /
category content and seek third-party citations.

Is paying for an AI-visibility tool worth it?

Only if you intend to act on the recommendations. Measurement without
action is theater. If you're going to publish content, fix schema, and
do citation outreach based on what the tool tells you, then yes. If you
just want a dashboard to glance at, save your money.

One more thing

We built Brandswarm because we kept watching teams chase phantom causes
("Google must hate us!") when the real problem was something boring — a CDN
rule, a missing schema block, three weeks of letting their G2 page drift.
The fixes above aren't glamorous. They are, however, the ones that actually
move the needle.

Reddit's outsized weight in AI search answers (and how to earn it)

Mike — Thu, 04 Jun 2026 18:07:39 +0000

Originally published at brandswarm.io/blog/reddit-weight-ai-search/.

Of all the citation sources we see when we analyze where AI engines pull
their answers from, one is more lopsidedly important than every prediction
we made when we started: Reddit. Not your homepage. Not your blog. Not your
Wikipedia entry (when you have one). Reddit threads — the recent ones, in
the subs your buyers actually read — disproportionately shape what ChatGPT,
Perplexity, and especially Google AI Mode say about your category.

This post is what we've learned about why Reddit weighs so heavily, what
kinds of threads carry the most signal, and how to earn that signal without
getting banned from r/SEO.

The pattern

When we analyzed citation sources across the five major AI surfaces (ChatGPT,
Claude, Perplexity, Gemini, AI Overviews) for category-comparison queries
like "best CRM for nonprofits" or "alternatives to Notion," Reddit shows up
in the top three cited sources roughly 60% of the time. For Google AI Mode
specifically, Reddit shows up in the top three more than 80% of the time —
Google has been very public about giving Reddit conversations heavy weight
since the 2024 partnership.

Why? A few overlapping reasons:

Reddit is opinion-dense. A r/SEO thread titled "What's everyone using to track AI visibility?" has thirty different humans saying twenty different things. AI engines synthesize this into "here's the range of what people use" — far more useful for the user than any single vendor page.
Reddit is anti-spam-policed by humans. Bad answers get downvoted; obvious shilling gets called out. The remaining content has high authenticity signal. AI engines trust upvote ratios as a proxy for "is this answer actually good."
Reddit is recent. A thread from last month outweighs a blog post from 2022. AI engines that prioritize freshness (Perplexity, Google AI Mode) lean on Reddit specifically because the median answer is timely.
Reddit is structured. Comments are threaded, replies are linked to parents, upvotes provide a quality score. AI retrieval layers can parse this structure cleanly — much cleaner than parsing a 5,000-word blog post with no semantic markup.

Which Reddit threads carry the most signal

From the pattern we see, in rough order:

"What's everyone using for X?" threads — high vendor-comparison density, often cited verbatim.
"Has anyone tried [your competitor]?" threads — when users discuss a competitor, AI engines retrieve those discussions for queries about the competitor's alternatives.
"[Your competitor] vs [other competitor]" threads — direct head-to-head discussions show up when buyers run that exact query.
Negative-experience threads ("Why I left [competitor]") — these become "alternatives to" answers for the competitor.
Tutorial threads ("How I set up X") — high citation rate for how-to queries.

The pattern: buyer-language threads cite better than
technical-deep-dive threads. r/SEO discussions about a topic get
pulled more than r/programming discussions about the same topic. Buyers
search in buyer terms.

How to earn the signal without getting banned

Reddit's communities are intensely allergic to self-promotion. We've watched
multiple SaaS founders nuke their brand presence overnight by posting in
r/SEO with what looked like a casual mention but read as a pitch. The rules
that actually keep you safe:

The 90/10 rule

For every comment that names your product, make nine that genuinely help and
don't name it. Reddit's automod tracks this ratio on a per-account basis.
Drop below 80/20 and your comments start getting auto-removed before anyone
sees them.

The "passing reference" link rule

When you do link your site, it should look like a passing reference, not a
recommendation. "There's a free check at brandswarm.io if you want to see
your own numbers" reads as casual mention. "Check out my SaaS Brandswarm,
which solves exactly this!" reads as pitch. The second gets auto-removed by
most subs' filters; the first usually doesn't.

The age-the-account rule

New Reddit accounts that immediately start posting in their category sub get
shadowbanned within hours. If your account is less than ~6 months old or has
less than ~500 karma, spend 4–8 weeks participating in totally unrelated subs
(cooking, fitness, your hobby) before you touch the SEO/marketing subs at
all. Tedious, but the only path that works.

The "answer the actual question" rule

Read the thread. If someone asks "what's the best free CRM," answer with a
free CRM. Don't pivot to "well, the better question is what CRM gives the
best ROI" — that's marketer-speak and Reddit smells it instantly. Answer the
question first, then optionally add context.

The subs that matter most for AI visibility specifically

If your buyers might be in any of these, get presence (organic, helpful, low-promotion) over a few months:

r/SEO — the big tent, 700K+ members, broad SEO + AI search topics.
r/TechSEO — smaller, more technical, very engaged. Better for genuinely deep takes.
r/bigseo — pros-only flavor; threads are higher-quality on average.
r/marketing — broad marketing audience; AI-search threads land well here.
r/SaaS — founder-friendly; this is where "alternatives to" threads come from.
r/Entrepreneur — 4M+ members but very strict on promo. Pure helpful only.
r/PPC — adjacent; tends to discuss AI search from a paid angle but reads same content.

For each sub, lurk for two weeks before commenting. Read the wiki / pinned
rules thread. The rules vary subtly and breaking them anywhere kills your
presence everywhere (Reddit's automod is account-wide).

What not to do

Don't post your own threads pitching your tool. Reddit users immediately downvote and the thread sinks. Comment, don't post.
Don't comment under multiple accounts. Reddit's IP-fingerprint detection will catch you and zero out all your accounts.
Don't use the same opening line across comments. Boring but: vary the first 30 characters meaningfully so your account doesn't look templated.
Don't link to gated content. "Here's our free report (just enter your email)" reads as marketing; "here's the report PDF" reads as a gift.
Don't ask Reddit users to "DM me" for more info. That's a known spammer move.

What does this look like in practice?

A reasonable cadence for a founder building Reddit presence: 2 helpful
comments per day across the subs in your category, for 60 days. Out
of ~120 comments, maybe 10 will include a link to your site. The rest are
pure helpful. By month three you'll have measurable Reddit traffic to your
site, AI engines will start citing the discussions you participated in, and
your account will be old enough that your contributions carry weight.

This is slow. There's no shortcut that works. The teams that try to compress
it into a week get banned within a week. The teams that play it long get
permanent surface area in the highest-leverage citation source in AI search.

FAQ

Can I pay an agency to do Reddit for me?

Generally no. Reddit's automod and human mods are very good at spotting the
agency-account fingerprint (rapid posting across unrelated subs, mostly
English-only with subtle generic phrasing, link-heavy). Agencies that promise
Reddit growth have a very high account-death rate. Build presence yourself
or invest in someone in-house who genuinely uses Reddit.

How do I measure Reddit's contribution to my AI visibility?

Two ways. One, run a Brandswarm scan or an equivalent tool and watch your
citation sources per surface over time — Reddit URLs showing up in your
cited sources is the leading indicator. Two, watch UTM-tagged traffic from
reddit.com referrer in your analytics. The Reddit-to-organic
AI-citation transition takes about 8–12 weeks; the Reddit-to-direct-traffic
signal shows up within days of well-received comments.

Should I also be on X / LinkedIn / Hacker News?

X and LinkedIn matter for different reasons (network effects, B2B reach) but
carry much less AI-search citation weight than Reddit. Hacker News carries
more weight than X/LinkedIn for the developer/SaaS audience specifically.
Order of priority for AI-search-driven growth: Reddit ≫ Hacker News > X
≫ LinkedIn.

Bottom line

Reddit isn't a nice-to-have for AI search visibility — it's the single
highest-leverage non-paid channel for a brand-new SaaS. Plan to spend a few
months building it. Stay almost entirely helpful. Save your two links per
week for the threads where naming your product would be the right answer
regardless of who you were. Long game, but it compounds.

Why Bing matters more than Google for ChatGPT visibility

Mike — Thu, 04 Jun 2026 18:06:08 +0000

Originally published at brandswarm.io/blog/bing-chatgpt-visibility/.

ChatGPT's browsing tool — the one that activates when a user asks for current
or factual information — fetches its results from Bing. Not Google. Most marketers
haven't internalized this, and it's why some brands with strong Google
rankings find themselves invisible to ChatGPT for queries where they should
obviously dominate.

This post is the practical version of the Bing-vs-Google story for the AI era:
what's different, how to check whether you're affected, and the four fixes
that have moved real brands' ChatGPT visibility in 2–4 weeks.

The architecture, briefly

ChatGPT uses two retrieval paths:

Training data — a frozen snapshot of the internet from ~6–18 months ago, baked into the model. Slow to update; you can't directly influence it short of becoming Wikipedia-famous before the next training cut.
Browsing / web tool — when ChatGPT decides it needs real-time information, it calls Bing's search API, fetches the top results, and synthesizes an answer from them.

Which path gets used depends on the question. "What is [your brand]?" usually
pulls from training. "Best CRM for nonprofits in 2026" — current,
comparative — almost always triggers browsing.

And browsing means Bing. So: the question of whether ChatGPT cites you on
competitive, current queries is largely the question of whether you rank in
Bing.

Why Bing is different from Google

Bing crawls less aggressively than Google, weighs different signals slightly
differently, and applies a couple of editorial heuristics that catch a lot of
sites off-guard:

Smaller crawl budget. Bing's crawler hits each site less frequently than Googlebot. If your site has a slow origin (typical of Render / Vercel free tiers during cold starts), Bing's crawler may give up before fully indexing.
URL parameters tolerated less. URLs with multiple query parameters get more often dropped by Bing as "non-canonical." Clean URLs index more reliably.
Heavier weight on the title tag and H1. If your title and H1 don't agree about what the page is about, Bing will sometimes skip indexing entirely. Google is more forgiving.
Sitemap submission matters more. Google often discovers pages from links alone. Bing relies on the sitemap as the primary discovery signal; pages not in the sitemap take longer or never get crawled.
Less tolerance for JavaScript-rendered content. Bing's renderer is years behind Google's. SPA-style sites that hide content behind hydration often partially or fully fail to index.

None of these are dealbreakers. They're just different. The compound effect:
a brand with strong Google performance can be at ~30–60% of its potential Bing
coverage without realizing it.

How to check if you have a Bing problem

The 30-second version

Go to Bing (bing.com) and type:

site:yourdomain.com

Compare the number of results to the same query on Google. If Bing returns
<50% of what Google shows, you have a Bing coverage gap. If Bing returns
<10%, you have a serious one.

The 5-minute version

Sign in (or sign up) at
Bing
Webmaster Tools with the same Microsoft account you use for everything
else. Add your domain, verify (DNS TXT or HTML meta tag), and submit your
sitemap. Within an hour, Bing's Index Explorer will show you:

Total pages indexed
Pages "discovered but not indexed" (and the specific reason in the "Why?" column)
Crawl errors
Search performance

Bing Webmaster Tools is a strictly better experience than Google Search Console
for this category — the data is more actionable and the rate limits are
friendlier. If you've never used it, you're missing the easiest 30%
improvement in your ChatGPT visibility.

The 4 fixes that actually move the needle

Fix #1: Submit your sitemap to Bing Webmaster Tools today

This is the cheapest move. Verify your domain, add the sitemap, use the
URL Submission API to push your top 50 pages immediately. Bing's URL
Submission limit is 10,000 URLs per day per domain — far higher than Google's
daily quota — so you can effectively force a re-crawl of your whole site.
Most teams see a 2–4× jump in indexed pages within a week.

Fix #2: Fix the title/H1 agreement on key pages

Walk your most important 20 pages and verify that the <title>
and the <h1> are about the same thing. Not identical —
just consistent topic. "Best CRM for Nonprofits in 2026" / "Best CRM tools
for nonprofit organizations" — fine. "Solutions" / "Welcome to FooCRM" —
not fine. Bing's classifier reads both and downweights mismatches.

Fix #3: Speed up Bingbot's crawl path

Bing's crawler times out faster than Google's. If your origin has cold-start
latency (Render free tier, Vercel hobby, Heroku eco), Bing gives up. Two fixes
that work:

Upgrade to a paid tier that keeps the dyno warm ($7/mo on Render Starter; similar elsewhere).
Cache your top 50 pages at the CDN with a long TTL (1 hour+) and let Bing fetch from cache. We did this and saw indexed pages 4× in two weeks.

Fix #4: Submit specifically the pages ChatGPT users are looking for

Comparison pages, "alternatives to X," and "best X for Y" listicles dominate
ChatGPT's browsing answers because those are the queries it's most likely to
call Bing for. If you have ten product pages and one comparison page, your
comparison page is the one ChatGPT will most often cite. Make sure it's the
first one you push to the URL Submission API.

What about Yahoo, DuckDuckGo, and Brave Search?

All three use Bing's index as a primary source (some with their own overlay
rankings). So getting into Bing also gets you into all three. The "fix Bing"
effort is leveraged across the entire non-Google search universe.

FAQ

Does this also help Claude or Perplexity?

Not directly. Claude does very little real-time retrieval and Perplexity uses
its own crawler infrastructure. But strong Bing presence does correlate with
strong Perplexity citation rates because the underlying signal (third-party
authority + clean URLs + structured content) is the same. Fixing Bing tends
to fix Perplexity as a side effect.

How long until I see ChatGPT mention me?

Bing indexing: 1–2 weeks from submission. ChatGPT citation: another 1–2
weeks for browsing to pick up the new pages, and 4–8 weeks for the citation
pattern to stabilize. So 6–10 weeks for a clean read on whether the fix
worked.

Anything else Bing-specific to know?

Two things. First, Bing pays much more attention to noindex,follow
vs noindex,nofollow than Google does — be precise on auth pages
to avoid bleeding link equity. Second, Bing's image search is dramatically
underused as a referral source; if you have a brand-relevant asset (an
infographic, a comparison chart, a screenshot), naming it descriptively and
including ALT text drives 5–10× more traffic from Bing image search than
from Google.

Bottom line

ChatGPT's browsing answers are Bing answers in a different wrapper. Two hours
of work in Bing Webmaster Tools, in most cases, returns more ChatGPT
visibility than two weeks of generic "SEO improvements." If you haven't
verified your domain there, that's the move this week.

If you want to see whether you're currently being cited by ChatGPT and
Perplexity for the queries that matter in your category, a free Brandswarm
scan shows you per-surface visibility in 60 seconds.

Schema markup for AI search: the complete 2026 reference

Mike — Thu, 04 Jun 2026 17:59:00 +0000

Originally published at brandswarm.io/blog/schema-markup-for-ai-search/.

AI engines retrieve structured pages more reliably than unstructured ones. Not
in the abstract — measurably, in side-by-side tests we and others have run. A
product page with proper SoftwareApplication JSON-LD gets cited in
AI Overviews and Perplexity 2–3× more often than the same page without it. A
documentation page with FAQPage gets snippet-quoted by ChatGPT
directly. The marginal effort is 30 minutes; the marginal value is large.

This is the reference. We've put together the four Schema.org types that
actually matter for AI search in 2026, with copy-pasteable JSON-LD, the
validation step that catches 80% of mistakes, and the three structured-data
patterns that quietly tank visibility even when they look right.

The 4 schemas that matter

Of the 800+ types in the Schema.org vocabulary, four cover almost every situation a B2B or B2C brand cares about for AI search:

Organization — your identity. Goes on the homepage; usually one per site.
SoftwareApplication or Product — what you sell. Goes on product pages and pricing.
FAQPage — question-answer content. Goes on docs, billing FAQ, help pages.
HowTo — step-by-step instructions. Goes on tutorials and onboarding guides.

Two more sometimes worth adding: Article / BlogPosting
(for blog posts; reasonable but smaller AI-retrieval lift than the above) and
BreadcrumbList (small lift, near-zero effort if your URLs are clean).
Skip the rest for now.

1. Organization schema (every site needs this)

The Organization block is doing one specific job: connecting your
domain to your other web presences so AI engines can build a coherent identity
graph. The sameAs array is what does the connecting.

{
  "@context": "https://schema.org",
  "@type": "Organization",
  "name": "Brandswarm",
  "alternateName": "Brandswarm.io",
  "url": "https://brandswarm.io",
  "logo": "https://brandswarm.io/static/logo/brandswarm-wordmark-dark-md.png",
  "description": "AI brand visibility tracking across ChatGPT, Claude, Perplexity, Gemini, and AI Overviews.",
  "foundingDate": "2024-09-01",
  "email": "hello@brandswarm.io",
  "sameAs": [
    "https://twitter.com/brandswarm",
    "https://linkedin.com/company/brandswarm",
    "https://github.com/brandswarm",
    "https://www.crunchbase.com/organization/brandswarm",
    "https://en.wikipedia.org/wiki/Brandswarm"
  ]
}

What to omit: aspirational sameAs URLs that
don't exist yet. Google's structured-data validator flags broken URLs as
errors and Google may discount your whole block. Only list what's live.

Where to put it: in the <head> of your
homepage as <script type="application/ld+json">. Repeating
it on every page is allowed but unnecessary — the homepage is enough.

2. SoftwareApplication / Product schema

This block tells AI engines what you sell, what category, and roughly what it
costs. Goes on the product page and pricing page.

{
  "@context": "https://schema.org",
  "@type": "SoftwareApplication",
  "name": "Brandswarm",
  "description": "Track how AI assistants describe your brand. 5 surfaces, daily monitoring.",
  "applicationCategory": "BusinessApplication",
  "applicationSubCategory": "AI Brand Monitoring",
  "operatingSystem": "Web",
  "url": "https://brandswarm.io",
  "offers": [
    {"@type": "Offer", "name": "Starter", "price": "49", "priceCurrency": "USD"},
    {"@type": "Offer", "name": "Growth",  "price": "149", "priceCurrency": "USD"},
    {"@type": "Offer", "name": "Enterprise", "price": "399", "priceCurrency": "USD"}
  ],
  "creator": {"@type": "Organization", "name": "Brandswarm", "url": "https://brandswarm.io/"}
}

About aggregateRating: if you've seen the spec,
you've seen aggregateRating blocks with star averages. Don't add
one unless you have a real source for the rating (Trustpilot, G2 review count,
etc.). Google's algorithm specifically downweights brands that ship fabricated
ratings. Risk-to-reward is bad.

For B2C / physical products: use Product instead
of SoftwareApplication. Same structure; different
@type and include brand, sku,
gtin, and image.

3. FAQPage schema (the highest-lift block per page)

AI Overviews quote FAQ schemas directly. Perplexity cites them. ChatGPT
retrieves them when answering similar questions. Pound for pound, the
best-converting structured-data type for AI visibility.

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "What does Brandswarm track?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Brandswarm tracks how your brand appears in ChatGPT, Claude, Perplexity, Gemini, and AI Overviews. For each surface we capture mention rate, position, sentiment, sample quotes, and citation sources."
      }
    },
    {
      "@type": "Question",
      "name": "Do I need a credit card to start?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "No. The free instant scan runs without signup or card. Paid trial also starts without a card."
      }
    }
  ]
}

What works: 3–10 question/answer pairs per page. Each
question should be one a real user would type. Each answer should be a
complete, standalone sentence (50–250 chars is the sweet spot — long enough
to be useful, short enough to be quotable).

What doesn't: stuffing 30 Q&As to "win more snippets." Google
cracked down on this in 2023; AI engines followed. Quality > quantity.

4. HowTo schema (the most under-used)

Tutorial pages with HowTo schema get retrieved disproportionately
by Gemini and AI Overviews because the structure maps cleanly to a step-by-step
answer format. If you have a "getting started" doc or onboarding guide, this
is your easiest win.

{
  "@context": "https://schema.org",
  "@type": "HowTo",
  "name": "Get your first AI brand-visibility report",
  "description": "Track your brand in ChatGPT, Claude, Perplexity, Gemini, and AI Overviews in under 5 minutes.",
  "totalTime": "PT5M",
  "step": [
    {"@type": "HowToStep", "position": 1, "name": "Run an instant report",
     "text": "Visit brandswarm.io/scan/, enter your domain. Free, no signup."},
    {"@type": "HowToStep", "position": 2, "name": "Claim the report",
     "text": "Enter your email to save the scan and unlock the full report."},
    {"@type": "HowToStep", "position": 3, "name": "Add your prompts",
     "text": "Add the competitive queries your buyers actually search."}
  ]
}

Keep each text field a single, instructive sentence. Avoid links
inside step text (they don't help; sometimes they hurt). Include
image URLs if you have step illustrations — both Google and
Perplexity surface them.

The 3 structured-data mistakes that quietly tank visibility

Mistake #1: Mismatched URL in `url` vs canonical

Your JSON-LD says "url": "https://www.brandswarm.io/" and your
canonical link tag says https://brandswarm.io/. Or vice versa.
Google penalizes the inconsistency; AI engines silently discount the page.
Fix: keep the URL identical to the canonical, including the trailing slash
policy.

Mistake #2: `sameAs` URLs that 404

You listed https://twitter.com/brandswarm in sameAs
but the handle is actually @brandswarm_io, or the LinkedIn page
was deleted, or the GitHub org never existed. Each broken URL nudges your
block toward "untrusted." Audit annually. Use Google's structured-data testing
tool — it'll flag broken sameAs URLs explicitly.

Mistake #3: FAQ schema where the visible page doesn't show the questions

You put FAQPage JSON-LD on a page where the questions and
answers aren't visibly rendered (they're hidden in an accordion that loads
via JS, or they're in the schema but not on the page at all). Google's spam
team called this out specifically in late 2023: schema content must match
what users actually see. AI engines now check the same way. Fix: render the
Q/A pairs as visible <h3> + <p> on the
page, then mirror in JSON-LD.

The validation step that catches 80% of mistakes

Two free tools, in this order:

Google's Rich Results Test (search.google.com/test/rich-results) — paste a URL or the raw JSON-LD. Catches syntax errors and missing required fields.
Schema.org's own validator (validator.schema.org) — stricter than Google's; catches type mismatches that Google tolerates.

Run both before shipping. Then sample 5–10 of your most important URLs in
Google Search Console under Enhancements → check that the structured data
types you expect are reported as discovered. If they're not, Google didn't
recognize what you shipped — usually a syntax issue.

What you don't need

Speakable — niche, voice-search only, AI engines don't use.
VideoObject on every page with embedded video — only useful if the video is the page's main content.
Three different schemas on the same page just to be thorough — pick the most specific type that matches the page's main content. Multiple non-conflicting types are fine, but redundant ones add noise.
Schema generators that produce 200-line blocks — most fields are optional. Ship the minimum and add only when you have content to fill it.

Quick checklist

If you do nothing else, do this in this order:

Add Organization with a complete sameAs array to your homepage.
Add SoftwareApplication (or Product) to your product/pricing page.
Add FAQPage to any page with question-answer content (billing FAQ, docs, support).
Add HowTo to your getting-started or onboarding guide.
Validate everything in Google's Rich Results Test before shipping.
Re-validate quarterly; sameAs URLs especially go stale.

This is two hours of work for most sites. The retrieval-layer lift you get is
measurable within a month. If you want to see where structured data is moving
your AI visibility specifically, the free Brandswarm scan flags structured-data
gaps per surface as part of the recommendations engine.

robots.txt for AI search: the 2026 cheat sheet (GPTBot, ClaudeBot, and the rest)

Mike — Thu, 04 Jun 2026 17:58:57 +0000

Originally published at brandswarm.io/blog/robots-txt-for-ai-search/.

Your robots.txt is the first place AI crawlers look when they
arrive at your site. Get it wrong and you're invisible to ChatGPT, Claude,
Perplexity, Gemini, and AI Overviews regardless of how good your content,
schema, or backlinks are. Get it right and the cost is zero — it's just a
text file.

This is the cheat sheet. Every AI crawler that matters in 2026, whether
to allow them, and a copy-pasteable robots.txt file you can
drop in today.

The user-agents that matter

User-agent	Operator	What it does	Allow?
`GPTBot`	OpenAI	Trains future models. Does NOT do real-time retrieval for ChatGPT.	Yes — visibility, not training
`OAI-SearchBot`	OpenAI	Retrieval for ChatGPT search / SearchGPT.	Yes — direct ChatGPT visibility
`ChatGPT-User`	OpenAI	Used when a user invokes ChatGPT's browsing tool. Fetches a single URL.	Yes — required for browsing
`ClaudeBot`	Anthropic	Crawl for Claude (training + retrieval).	Yes — direct Claude visibility
`Claude-Web` / `anthropic-ai`	Anthropic	Older / alternate user-agent variants.	Yes — same reason
`Google-Extended`	Google	Crawls for Gemini training. Separate from Googlebot.	Optional — yes if you want training inclusion
`Googlebot`	Google	Powers regular Google search + AI Overviews. Do not block.	Always yes
`PerplexityBot`	Perplexity	Retrieval for Perplexity answers.	Yes
`Perplexity-User`	Perplexity	Fetches single URLs when users follow Perplexity links.	Yes
`Bytespider`	ByteDance	Crawls for Doubao / TikTok AI features.	Yes if you have TikTok/APAC audience
`Amazonbot`	Amazon	Powers Alexa / Q / Amazon AI features.	Optional
`Applebot-Extended`	Apple	Crawls for Apple Intelligence training. Separate from Applebot (search).	Optional
`Applebot`	Apple	Powers Spotlight + Siri suggestions.	Always yes
`meta-externalagent`	Meta	Crawls for Meta AI training.	Optional
`CCBot`	Common Crawl	Open-source crawl used as training data by many models.	Optional — wide influence
`Bingbot`	Microsoft	Regular Bing search + ChatGPT browsing tool retrieval.	Always yes
`DuckAssistBot`	DuckDuckGo	Powers DuckDuckGo's AI Assist.	Yes
`Diffbot` / `BrandBot` / etc.	Various	Niche crawlers used by enterprise AI tools.	Optional — minor traffic

Quick decision: 3 policies that cover 95% of cases

Policy A: maximum AI visibility (recommended for SaaS, content brands, B2B)

# Maximum AI visibility. Allows training + retrieval for all major engines.
User-agent: *
Allow: /

# Block private/auth surfaces from any crawler
User-agent: *
Disallow: /admin/
Disallow: /app/
Disallow: /billing/
Disallow: /accounts/

Sitemap: https://yourdomain.com/sitemap.xml

This is the right policy if your business benefits from being discovered
in AI answers. Almost every SaaS, B2B company, and brand that sells anything
falls into this category. The wildcard User-agent: * applies to
every crawler including the AI ones.

Policy B: allow AI retrieval, block AI training (the Content-Signal compromise)

# Allow real-time retrieval (so AI can cite you when users ask)
# but signal that content should not be used for model training.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=no
Allow: /

Disallow: /admin/
Disallow: /app/
Disallow: /billing/

Sitemap: https://yourdomain.com/sitemap.xml

Use this if you want to be discoverable in ChatGPT/Perplexity/Gemini answers
but you don't want your content baked into next year's model training data.
The Content-Signal header is honored by OpenAI, Anthropic,
Google, and Perplexity as of mid-2025. It's the right middle ground.

Policy C: block everything (only for sites that genuinely don't want AI visibility)

# Block all AI crawlers explicitly. Allow Googlebot/Bingbot for traditional search.
User-agent: GPTBot
Disallow: /
User-agent: OAI-SearchBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Claude-Web
Disallow: /
User-agent: anthropic-ai
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: PerplexityBot
Disallow: /
User-agent: Perplexity-User
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: CCBot
Disallow: /

# Allow traditional search
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

Sitemap: https://yourdomain.com/sitemap.xml

Use this only if you have a strong reason — premium paid content, news org
with monetization concerns, sensitive material. Be aware: blocking AI
retrieval means your brand will not be cited when users ask AI assistants
about your category. For most businesses, this is a strategic mistake.

The Cloudflare gotcha

If your site is behind Cloudflare, there's a feature called "AI Crawl
Control → Managed robots.txt" that injects a Policy-C-style block
into your robots.txt on the wire, regardless of what your origin
serves. The toggle is on by default for many zones. Many brands are blocking
every AI crawler without knowing.

To check: curl https://yourdomain.com/robots.txt. If you see a
block titled "# BEGIN Cloudflare Managed content", you're
affected. Turn the toggle off in Cloudflare → AI Crawl Control → Managed
robots.txt. We wrote up the full story
here.

Validating your `robots.txt`

Three quick checks:

Use Google's robots.txt tester in Search Console — paste a URL and a user-agent, it tells you if the page is fetchable. Their tester is now under the URL Inspection tool.
Curl with each crawler's user-agent and inspect the response:

   curl -A "GPTBot" https://yourdomain.com/robots.txt
   curl -A "ClaudeBot" https://yourdomain.com/robots.txt

Watch Bing Webmaster Tools' Crawl Errors — Bing reports robots.txt-blocked URLs there. Other engines don't surface this as cleanly.

Three rules

Specific user-agents override the wildcard. If you have User-agent: * Allow: / and below it User-agent: GPTBot Disallow: /, GPTBot is blocked. The wildcard isn't a fallback; it's a default that specific rules override.
One User-agent block per crawler. Some sites repeat User-agent: GPTBot with different rules in different blocks; only the first block is honored. Consolidate.
Don't block Googlebot when you mean Google-Extended. These are different crawlers. Googlebot powers Search + AI Overviews. Google-Extended powers Gemini training. Blocking Googlebot tanks your traditional Google traffic.

FAQ

I want to be in ChatGPT but not in Claude. Can I?

Yes. Allow GPTBot, OAI-SearchBot, and
ChatGPT-User; disallow ClaudeBot,
Claude-Web, and anthropic-ai. Practical impact is
modest because most brands want presence everywhere AI assistants exist,
but the option is there.

What about `noai` and `noimageai` meta tags?

These are the page-level equivalent of robots.txt rules. They tell crawlers
not to use the page's content for AI training. Less widely honored than
Content-Signal headers; useful as defense-in-depth on
pages where you really care.

What about `llms.txt`?

A proposed standard for "here's a curated text version of my content for
LLMs to ingest cleanly." Adoption is uneven; OpenAI and Anthropic both
said publicly in 2025 that they prefer to crawl normally. Worth shipping
if it's easy to generate, but don't rely on it as your primary AI-visibility
strategy.

Do I need to also add an `X-Robots-Tag` HTTP header?

Only if you want per-page granularity that robots.txt can't express (e.g.,
"noindex this specific PDF without listing it"). For broad AI-visibility
policy, robots.txt is sufficient.

Bottom line

Most brands win by shipping Policy A. Some by shipping Policy B. Very few
should ship Policy C. Whichever you choose, do it deliberately — and
re-check after every CDN configuration change. The most common reason
brands lose AI visibility isn't a strategy decision; it's a CDN feature
that flipped a switch they didn't notice.