TL;DR: Blocking GPTBot doesn't remove you from ChatGPT search results. OpenAI runs two separate bots: one that trains models (GPTBot) and one that fetches live citations (OAI-SearchBot). Every major AI company uses this dual-bot system. Block the wrong one and you vanish from AI search. Block the right one and you protect your content while staying visible.
Most people who block GPTBot expect their content to disappear from ChatGPT entirely. Then they search for their own article title in ChatGPT and find it sitting right there in the results with a working link and an accurate pull quote. The block didn't fail. You just blocked the wrong bot.
OpenAI doesn't run one bot. They run two. So does Anthropic. So does Google. Every major AI company operates a two-bot system because training and citation are fundamentally different jobs that need fundamentally different infrastructure.
The first bot harvests your content to train the model. It reads your articles, extracts patterns from your prose, and bakes those patterns into billions of model parameters. Your words become part of the AI's permanent knowledge. The model never cites you. It never links back to you. It never sends you traffic. Training bots give you nothing in return for what they take.
The second bot fetches your content in real time when someone asks a question your site might answer. The bot finds your page, reads the relevant section, and surfaces it in the AI's response with your name, a link, and a snippet. This is how you show up in ChatGPT search results. Citation bots drive referral traffic and keep you visible in AI-powered search.
Blocking the training bot does absolutely nothing to the citation bot. They don't share a blocklist. They don't check each other's permissions. They operate as if they work for separate companies.
The Two-Bot System Every AI Company Runs
Here's the full breakdown of how each major AI company splits training from citation:
OpenAI runs GPTBot for training. GPTBot crawls your site, reads your content, and uses it to improve future models. That's where your IP goes in and never comes back out. For live citations, OpenAI runs OAI-SearchBot and ChatGPT-User. These bots fetch content when a user asks a question. Block GPTBot and you stop training. Block OAI-SearchBot and you vanish from ChatGPT search results.
Anthropic operates ClaudeBot for training. ClaudeBot works exactly like GPTBot. It harvests content to build the model's knowledge base. For citations, Anthropic runs Claude-SearchBot and Claude-User. Block ClaudeBot to stop training. Leave Claude-SearchBot allowed if you want to appear in Claude's search answers.
Google uses Google-Extended for training. This is separate from Googlebot, which has been crawling the web for decades. Googlebot handles citations because Google already had a citation infrastructure before AI search existed. Block Google-Extended and you stop contributing to Gemini training. Block Googlebot and you disappear from Google Search entirely, which most sites can't afford.
Perplexity runs PerplexityBot and Perplexity-User. Both fetch content for live answers. Perplexity doesn't train foundation models, so they don't operate a separate training crawler. Every request from Perplexity is a citation request.
Meta operates meta-externalagent for training. They don't currently run a public citation bot for consumer search products, so blocking meta-externalagent has no downside for visibility.
Training crawlers need bulk access to millions of pages, running batch jobs that might revisit your content weeks or months later. Citation crawlers need low-latency access to fresh content, often fetching a single page in response to a specific user query.
This architecture exists because training and citation have fundamentally different requirements. Training bots run in batches. They might crawl your site once and never come back. They don't care about freshness. They care about volume. Citation bots run on demand. They fetch a single page in milliseconds when a user asks a question. They care about speed and freshness, not volume.
Building one bot that does both jobs well is harder than running two specialized systems. So every company runs two. As AI crawling vs traditional crawling explains, the infrastructure behind these systems diverges at nearly every layer, from request patterns to caching strategies to how they handle rate limits.
What You're Blocking Right Now (And What You're Not)
Most robots.txt files I see block training bots and stay silent on citation bots. That's not necessarily wrong. The robots.txt standard works by explicit permission. If you don't mention a bot, it's allowed by default. But I don't like relying on default behavior. Defaults change when companies push updates or rewrite their crawler logic.
Here's what a safe, explicit robots.txt configuration looks like if you want to block training but allow citations:
``txt
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: Bytespider
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ChatGPT-User
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Perplexity-User
Allow: /
``
The first block stops training. The second block guarantees you stay citable. Both blocks matter. Leaving citation bots unstated means trusting default behavior, and I've seen too many cases where defaults shifted after a company updated their crawler documentation.
You'll notice Googlebot isn't in either list. That's intentional. Blocking Googlebot means disappearing from Google Search entirely, which most sites can't survive. Google-Extended is the training-specific bot. Block that and leave Googlebot alone. You get the split you want: no training, yes citations.
One more detail. If you're running a Cloudflare-managed site, your robots.txt likely includes a comment that says # BEGIN Cloudflare Managed content. That marker means Cloudflare auto-generates part of your robots.txt based on dashboard settings. Don't edit robots.txt directly when that marker is present. Cloudflare will overwrite your changes on the next sync. Make changes through Cloudflare's AI Crawl Control panel instead.
How to Edit Robots.txt When Cloudflare Manages It
If you see the Cloudflare marker in your robots.txt, here's the correct path to making changes stick:
- Log into your Cloudflare dashboard and select the domain you're managing.
- Click Domains in the left sidebar.
- Select your specific domain from the list.
- Scroll down to AI Crawl Control or search for "AI" in the page search.
- Toggle the bots you want to block or allow using the switches provided.
If you've been blocking GPTBot and wondering why your articles still show up in ChatGPT, now you know. The bot you blocked wasn't the bot you thought it was. And if you want to stay visible in AI search while protecting your content from training, you need to configure both. For a deeper look at how AI systems decide what to crawl and when, see this comparison of Claude vs GPT for coding tasks, which shows how different models prioritize different types of content even when using similar underlying architectures.
Cloudflare's AI Crawl Control panel handles the syntax for you. Changes take effect within a few minutes and won't be overwritten by automated syncs. If you need more granular control than Cloudflare's toggles provide, you'll need to disable Cloudflare management of robots.txt entirely and manage the file manually as we advance.
Checking Whether Your Block Actually Works
After updating your robots.txt, wait 24 hours and check your server logs or analytics dashboard. Look for requests from the bots you tried to block.
Training bots like GPTBot, ClaudeBot, and Google-Extended should show zero requests and zero bytes transferred after blocking. If you still see requests from GPTBot after a week, either your robots.txt syntax is wrong or the bot isn't respecting the directive. Cloudflare's WAF lets you write a firewall rule that blocks by user agent string if robots.txt alone doesn't work.
Citation bots like OAI-SearchBot and ChatGPT-User should show periodic requests when users ask questions your content might answer. These requests are low-volume and sporadic. You might see a handful per week for a niche topic or dozens per day if you rank well for common queries. This is normal and desirable. Each request represents a potential citation in an AI answer.
If you're not seeing any citation bot traffic after allowing them, check two things. First, make sure your Allow: / directive is present and correctly formatted. Second, verify that your content actually ranks for questions people ask AI tools. Not every page earns citations. The citation bots only fetch pages that match a user query.
Why This Matters More Than You Think
Training without attribution is economically asymmetric. Your content becomes part of a billion-dollar model's knowledge base. You get no traffic, no links, and no royalties. The model answers questions using patterns it learned from your prose, and you never see a visitor.
Citation with attribution reverses that asymmetry. Your content appears as a source. The AI links to your page. Users click through. You get referral traffic from people who were already looking for what you wrote about. This is the only version of AI-powered search that benefits content creators.
Blocking training bots while allowing citation bots is the rational default for most sites. You protect your IP from being absorbed into the model's weights. You stay visible in the AI-powered search results that are increasingly replacing traditional Google queries. You get the upside without the downside.
The mistake most people make is thinking GPTBot controls all of OpenAI's access to your site. It doesn't. GPTBot controls training. OAI-SearchBot controls citations. Block one, allow the other, and you get the outcome you actually want.
The two-bot system isn't going away. Every AI company has adopted it because the engineering requirements for training and citation are incompatible. One bot can't do both jobs well. So they split the work. And you need to split your robots.txt configuration to match.
📦 Publishing Kit — Dev.to
Title Options (5)
Selected: Blocking GPTBot Won't Remove You From ChatGPT Search — Here's Why
Alternates:
- You're Blocking the Wrong AI Bot: The Training vs Citation Crawler Problem
- Why Your Site Still Appears in ChatGPT After Blocking GPTBot
- The Two-Bot System: How AI Companies Split Training From Citations
- Block GPTBot, Still Show Up in ChatGPT: Understanding AI Crawler Architecture
Slug
blocking-gptbot-wont-remove-you-from-chatgpt-search
Tags
webdev, ai, opensource, seo
Top comments (0)