DEV Community: Nikhil Goyal

The robots.txt Mistake That's Killing Your AI Search Visibility

Nikhil Goyal — Tue, 31 Mar 2026 11:45:34 +0000

There's a good chance your website is invisible to ChatGPT, Perplexity, and every other AI search engine — and the fix takes about 2 minutes.

I've been auditing sites for AI readability for the past year, and the single most common issue I find isn't bad content or missing schema. It's robots.txt blocking AI crawlers entirely. The site owner has no idea. They're optimizing content, writing FAQ pages, adding structured data — and none of it matters because the front door is locked.

Here's how to check yours and fix it.

The 30-second check

Run this right now:

curl -s https://yoursite.com/robots.txt

Now look for any of these bot names in Disallow rules:

GPTBot — OpenAI's crawler (powers ChatGPT citations)
OAI-SearchBot — OpenAI's search indexer (powers ChatGPT search)
ChatGPT-User — fetches pages when a ChatGPT user asks for live info
ClaudeBot — Anthropic's training crawler
Claude-SearchBot — Anthropic's search indexer (powers Claude's web search)
PerplexityBot — Perplexity's search crawler
Google-Extended — Google's AI training crawler (feeds Gemini and AI Overviews)
Applebot-Extended — Apple's AI training crawler (feeds Apple Intelligence)

Or just grep for it:

curl -s https://yoursite.com/robots.txt | grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|perplexitybot|google-extended|applebot-extended"

If you see Disallow: / next to any of those, that crawler can't see your site.

Why this happens more than you'd think

In about 4 out of 10 sites I audit, at least one major AI crawler is blocked. Here's how it happens:

1. The wildcard block

The most common culprit. Someone added this years ago and forgot about it:

User-agent: *
Disallow: /

This blocks everything — Googlebot, AI crawlers, all of it. Sometimes it was intentional for a staging site and got copied to production. Sometimes it's a CMS default that nobody changed.

2. WordPress security plugins

Plugins like Wordfence, Sucuri, and All In One Security sometimes add bot-blocking rules automatically. I've seen configs that specifically block GPTBot and ClaudeBot because they were categorized as "scrapers" in early 2024 when AI crawling was more controversial.

Check your security plugin settings — some have an "AI bot blocking" toggle that's enabled by default.

3. The copy-paste robots.txt

A lot of robots.txt files in the wild were copied from blog posts written in 2023-2024, when the default recommendation was to block AI crawlers to "protect your content." The landscape has shifted. If your goal is visibility, those rules are now working against you.

4. CDN or hosting-level blocks

Cloudflare, Vercel, and other platforms offer bot management settings. Some templates or one-click security configs block AI user agents at the infrastructure level, before robots.txt even gets read. If your robots.txt looks clean but AI crawlers still aren't hitting your server logs, check your CDN or hosting settings.

The distinction I wish someone had explained to me earlier

When I first started looking into this, I treated all AI crawlers the same. That was a mistake. They fall into two very different categories:

Training bots scrape your content to train AI models:

GPTBot (OpenAI)
ClaudeBot (Anthropic)
Google-Extended (Google)
Applebot-Extended (Apple)
Bytespider (ByteDance)
CCBot (Common Crawl)

Search bots fetch your pages in real time to answer user queries and cite you:

OAI-SearchBot (OpenAI — powers ChatGPT search results)
ChatGPT-User (OpenAI — fetches pages during live conversations)
Claude-SearchBot (Anthropic — powers Claude's web search)
PerplexityBot (Perplexity — indexes for AI search)

The distinction matters. If you block the search bots, you won't get cited when someone asks ChatGPT or Perplexity for a recommendation in your space. That's live traffic you're turning away.

Training bots are a different calculation. Some site owners are comfortable contributing to model training; others aren't. That's a legitimate choice. But blocking training bots doesn't necessarily remove you from AI answers — models are already trained on historical data, and search bots work independently.

A robots.txt that works for AI visibility

Here's what I recommend as a starting point. It allows all search-related AI bots while giving you explicit control over training bots:

# Search engines
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

# AI search bots — allow these for AI citation visibility
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

# AI training bots — your call on these
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Applebot-Extended
Allow: /

# Block training-only bots you're less comfortable with
User-agent: Bytespider
Disallow: /

User-agent: CCBot
Disallow: /

# Default: allow everything else
User-agent: *
Allow: /

# Sitemap
Sitemap: https://yoursite.com/sitemap.xml

If you want AI visibility but don't want to contribute training data, you can Disallow the training bots while keeping the search bots open. Just know that the line between training and search is blurry and getting blurrier — OpenAI's GPTBot description says it's for "improving AI models," but model improvements directly affect how well ChatGPT cites you in the future.

My take: unless you have a specific reason to block training bots, allow them all. In my experience, sites that allow both training and search bots tend to get cited more consistently than sites that only allow search bots — though I'll admit the sample size is small and I'm still tracking this.

Verifying it's actually working

This bit tripped me up at first — I updated a client's robots.txt and assumed we were done. Took me a week to realize the CDN was still blocking at the edge. Always verify with server logs:

# Check for AI crawler activity in the last 7 days
grep -iE "gptbot|oai-searchbot|chatgpt-user|claudebot|claude-searchbot|perplexitybot" /var/log/nginx/access.log | tail -20

If you're on a managed hosting platform without raw log access, check:

Cloudflare: Security → Bots → look for verified bot traffic
Vercel: Analytics → check for known bot user agents
GA4: Won't show bot traffic directly, but watch for referrals from chatgpt.com, perplexity.ai, gemini.google.com

A few things I've noticed in the logs:

AI crawlers hit fewer pages than Googlebot, but spend more time per page
They tend to favor pages with structured data and clean HTML
ChatGPT-User shows up in bursts — someone is asking ChatGPT about your topic and it's fetching your page live
If you see OAI-SearchBot hitting your site regularly, that's a good sign — you're being indexed for ChatGPT search

Don't panic if you don't see activity immediately. AI crawlers don't re-index on a fixed schedule. Give it 2-4 weeks after opening up your robots.txt before expecting consistent crawler traffic.

What I've seen happen after unblocking

One thing I didn't expect: the effects aren't instant, but they compound. After unblocking AI crawlers on a few client sites, we noticed OAI-SearchBot started hitting pages within 1-2 weeks. Actual citations in ChatGPT responses took another 2-4 weeks after that.

But the interesting part was what happened to sites that stayed blocked. We ran the same queries monthly, and sites that were blocked for 6+ months essentially didn't exist in AI answers — even when their content was objectively better than what was getting cited. The crawlers had built indexing patterns around the sites that were consistently accessible, and the blocked sites had no history to draw on.

It's similar to how Googlebot works — if your site has been returning 403s for months, you don't just flip a switch and rank tomorrow. There's a trust ramp.

Quick note: robots.txt is a request, not a wall

Well-behaved crawlers (GPTBot, ClaudeBot, PerplexityBot) respect robots.txt. But it's not a security mechanism. If you need granular control over AI training specifically, look into X-Robots-Tag: noai, noimageai headers or <meta name="robots" content="noai"> for page-level opt-out.

I've been digging into this stuff while building PageX, and robots.txt misconfiguration is genuinely the most common issue we see — more than bad schema, more than thin content, more than any of the fancy optimization stuff. The boring infrastructure problem is usually the one that matters most.

Has anyone here found surprising blocks in their robots.txt? Or noticed AI crawler activity change after opening things up? Curious what patterns others are seeing in their logs.

How to Make Your Website Easier for ChatGPT and Perplexity to Cite

Nikhil Goyal — Mon, 30 Mar 2026 15:07:07 +0000

I've spent the last year building an AI visibility tool, and in the process I've had to figure out what actually makes AI search engines cite one website over another.

Most of what I assumed turned out to be wrong. Here's what I've learned.

The basics: AI search is a different game

If you're already ranking well on Google, you might assume AI engines will find you too. The data says otherwise.

Ahrefs found that around 80% of URLs cited by ChatGPT, Perplexity, Copilot, and Google AI Mode don't rank in Google's top 100 for the original query. These engines aren't just repackaging Google results — they're making independent decisions about what to cite.

What seems to matter most:

Content structure and extractability — can the AI pull a clean answer from your page?
Freshness — BrightEdge research shows pages updated within 60 days are 1.9x more likely to appear in AI answers
Structured data — sites implementing schema markup and FAQ blocks saw a 44% increase in AI citations
Depth over keyword density — comprehensive coverage of a topic beats keyword stuffing

Traditional SEO signals like backlink count and domain rating still matter, but they're not sufficient on their own.

What I found in my server logs

When I started checking server logs for AI crawler activity, a few things stood out.

First, check whether you're even allowing AI crawlers in:

curl https://yoursite.com/robots.txt | grep -i "gptbot\|claudebot\|perplexitybot\|anthropic\|chatgpt"

A surprising number of sites — including ones that want AI visibility — have blanket blocks on these bots. Sometimes it's an overzealous security plugin, sometimes it's a robots.txt that hasn't been revisited since 2023.

Second, AI crawlers behave differently from Googlebot. They tend to hit fewer pages but spend more time parsing each one. They care a lot about whether the content is directly accessible in the HTML versus buried behind client-side JavaScript rendering.

If your site is a heavy SPA with most content rendered client-side, AI crawlers may be seeing an empty shell.

The content structure that gets cited

After analyzing which pages on client sites get cited versus which get ignored, a clear pattern emerged. It comes down to answer-first content architecture.

Here's what I mean:

<!-- This gets ignored by AI engines -->
<p>When it comes to understanding the complexities of emergency 
plumbing services, there are many factors that homeowners should 
consider before making a decision about which provider to call...</p>

<!-- This gets cited -->
<h2>How much does emergency plumbing cost?</h2>
<p>Emergency plumbing typically costs $150–$500 for common issues 
like burst pipes or severe leaks. After-hours calls usually add 
a $75–$150 surcharge.</p>

This isn't surprising when you look at the data: a Wix study found that 44.2% of all LLM citations come from the first 30% of a page's text. If your answer is in paragraph 7, the AI has already moved on.

The pattern that works best is what I've been calling "answer packs" — structured content blocks with a specific format:

## [Question in natural language]

[Direct answer in 2-4 sentences]

**Key details:**
- Specific fact or data point
- Another relevant detail
- Context that helps the reader decide

*Last updated: March 2026*

The "last updated" line matters. AI engines have a measurable recency bias — one study found that artificially refreshing publication dates alone can shift AI ranking positions by up to 95 places.

Schema markup: the low-hanging fruit most devs skip

If you're a developer reading this, structured data is probably the highest-ROI thing you can implement today. Websites with author schema are 3x more likely to appear in AI answers.

Here's a minimal FAQPage implementation:

{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [{
    "@type": "Question",
    "name": "What should I do if a pipe bursts?",
    "acceptedAnswer": {
      "@type": "Answer",
      "text": "Immediately shut off the main water valve, then call an emergency plumber. While waiting, open faucets to drain remaining water and move valuables away from the affected area."
    }
  }]
}

And if you're running a service business or local operation, LocalBusiness schema is equally important:

{
  "@context": "https://schema.org",
  "@type": "LocalBusiness",
  "name": "Your Business Name",
  "address": {
    "@type": "PostalAddress",
    "addressLocality": "Austin",
    "addressRegion": "TX"
  },
  "telephone": "+1-512-555-0100",
  "priceRange": "$$",
  "openingHoursSpecification": {
    "@type": "OpeningHoursSpecification",
    "dayOfWeek": ["Monday","Tuesday","Wednesday","Thursday","Friday"],
    "opens": "08:00",
    "closes": "18:00"
  }
}

The combination of semantic HTML (<article>, <section>, proper heading hierarchy) plus JSON-LD schema gives AI crawlers a machine-readable map of your content. Without it, they're guessing.

The HTML quality checklist

Here's the quick checklist I run on every page I want AI engines to cite:

Structure:

[ ] Uses semantic HTML5 elements (<article>, <section>, <main>)
[ ] Proper heading hierarchy (h1 → h2 → h3, no skipped levels)
[ ] Key content is in the HTML source, not only rendered via JS

Content:

[ ] Page leads with a direct answer to the primary query
[ ] Specific data points (prices, timelines, specs) are in plain text, not images
[ ] Content is updated within the last 60 days

Machine readability:

[ ] JSON-LD schema on the page (FAQPage, LocalBusiness, Product, HowTo — whatever fits)
[ ] Author information present (name, credentials, schema)
[ ] robots.txt allows GPTBot, ClaudeBot, PerplexityBot

Meta:

[ ] Clean, descriptive URLs (not /page?id=4827)
[ ] OpenGraph and meta description present
[ ] Sitemap includes the page and is submitted

None of this is exotic. It's mostly just good web development hygiene. But it's surprising how many production sites fail 3-4 of these checks.

Some counterintuitive findings

A few things that went against my assumptions:

Question-style headings underperform. I expected "How much does X cost?" headings to get cited more, but research from multiple sources shows straightforward headings actually get more citations than question-format ones (4.3 avg citations vs 3.4).

FAQ sections don't always help. Pages with dedicated FAQ sections showed slightly fewer citations than those without in one study — but this likely reflects that FAQs tend to appear on simpler support pages with less depth overall. The format works; it's the content quality that matters.

Different AI engines prefer different content types. ChatGPT disproportionately cites product and service pages directly. Perplexity leans toward listicles and comparison articles. Google AI Overviews pull from whatever it has already indexed highly. There's no single format that wins everywhere.

Brand mentions correlate more strongly with AI visibility than backlinks. Ahrefs data shows brand mention correlation at r = 0.664, higher than traditional link signals. Being talked about on Reddit, forums, and review sites seems to matter more for AI citation than having a strong backlink profile.

Measuring whether any of this works

The tricky part: an estimated 25-35% of AI-influenced traffic is misattributed in standard analytics setups.

What I do:

Manual prompting — every week, I run 20-30 relevant queries through ChatGPT, Perplexity, and Google AI. I note whether my pages are cited, who else is cited, and what format the cited content uses. Low-tech, high-signal.
GA4 referral sources — filter for chatgpt.com, perplexity.ai, gemini.google.com. The numbers will be small but growing fast.
Server logs — grep for GPTBot, ClaudeBot, PerplexityBot user agents. Track which pages they're hitting and how often.
Correlation tracking — watch for direct traffic spikes that line up with when your site starts appearing in AI answers. This catches the unattributed portion.

The manual prompting step sounds tedious, but it's by far the most useful. You'll learn more in 30 minutes of querying AI engines than from any dashboard.

The opportunity most people are missing

One data point that's been stuck in my head: according to an upGrowth report, technology and SaaS companies already see 18-25% of their traffic from AI referrals, but local service businesses sit at just 3-7%.

That gap is enormous. And it exists mostly because local businesses haven't structured their content for AI readability yet. The first mover advantage in local AI search is wide open.

What I'm still figuring out

I don't want to pretend this is all solved. Here's what's still genuinely hard:

Volatility is real. AI Overview content changes roughly 70% of the time for the same query. Only about 30% of brands remain visible in back-to-back AI responses. Consistency is hard to achieve.
Attribution is messy. Even with careful tracking, connecting AI citations to actual conversions requires a lot of inference.
Platform fragmentation. Optimizing for ChatGPT, Perplexity, and Google AI simultaneously sometimes requires conflicting approaches. There's no universal playbook yet.
The goalposts move. AI engines update their models and citation patterns regularly. What works today might not work in 3 months.

Wrapping up

We've been testing all of this while building PageX, and the biggest lesson so far is that AI systems reward clarity and structure far more than traditional SEO signals. Clean HTML, direct answers at the top of the page, fresh content, and proper schema — it's not glamorous, but it works.

The other lesson: this stuff compounds. Sites that start optimizing for AI readability now are building citation history that'll be hard for competitors to catch up on later. AI engines learn which sources consistently give reliable, extractable answers, and they keep going back.

Curious whether others here are seeing the same patterns in their logs, referrals, or citation data. If you've run the robots.txt check and found something surprising, or if you've noticed AI referral traffic showing up in your analytics, I'd be interested to hear what you're seeing.