DEV Community: Osama Mumtaz

I published an open dataset of every AI crawler user agent (GPTBot, ClaudeBot + 26 more)

Osama Mumtaz — Sat, 08 Aug 2026 00:12:21 +0000

While building AI-crawler tooling I kept needing the same thing: a current,
machine-readable list of AI crawler user agents. Every blog post has a partial
list, half of them outdated, none of them structured. So I published mine as an
open dataset.

GitHub: https://github.com/osamamumtaz01/ai-crawler-user-agents
Live endpoint (always current, CORS enabled): https://geoprompttracker.com/data/ai-crawlers.json

28 crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot,
Google-Extended, Bytespider, Applebot-Extended, Meta-ExternalAgent, and the
rest. JSON + CSV, CC BY 4.0. Also on Kaggle and Hugging Face.

The field that actually matters: `purpose`

The mistake I see constantly: treating "AI bots" as one thing. AI companies run
separate crawlers for different jobs:

Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content to train future models.
Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index pages so assistants can cite them in live answers.

Blocking a training crawler is an opt-out of model training. Blocking a search
crawler removes you from AI answers entirely. Very different decisions — and
because each company uses different bot names for each job, people block the
wrong one all the time.

So each entry carries a purpose field (training / search / both), which
makes policy decisions programmable:

const { crawlers } = await (
  await fetch("https://geoprompttracker.com/data/ai-crawlers.json")
).json();

// "Don't train on my content, but keep citing me"
const rules = crawlers
  .filter((b) => b.purpose === "training")
  .map((b) => `User-agent: ${b.userAgent}\nDisallow: /`)
  .join("\n\n");

The judgment call: `robotsCompliance`

The other field worth explaining. It rates whether each bot honors robots.txt
in practice — not just what the vendor claims. yes, partial, no, or
unknown, and every rating comes with a sourced robotsNote explaining the
reasoning (e.g. Perplexity is rated partial because independent
investigations observed undeclared fetching despite documented compliance).

That's also why robots.txt alone isn't enforcement — for bots rated no or
unknown, pair it with a WAF/CDN rule if blocking actually matters to you.

Other things you can do with it

Detect AI crawler traffic in middleware or log analysis
Audit which bots your current robots.txt actually allows
Research crawler behavior over time (the dataset is reviewed monthly)

If you spot a crawler I'm missing or a compliance rating that's gone stale,
issues and sourced corrections are very welcome — especially for the unknown
entries.

The 30-Second Raw-HTML Smoke Test for AI Visibility

Osama Mumtaz — Wed, 29 Jul 2026 04:11:55 +0000

The 30-Second Raw-HTML Smoke Test for AI Visibility
Your page looks perfect in the browser. Your Lighthouse score is green. Analytics look normal.

None of that tells you what an AI assistant can actually read on your page.

This came out of a genuinely good back-and-forth with @alexshev in the comments of my last post, and they framed it better than I had: treat the raw HTML check as a smoke test. It doesn't prove your site is great for AI readers. It just catches the embarrassing failures early — before you spend a month wondering why you never show up in AI answers.

Why the browser lies to you
When you load a page, your browser executes JavaScript, waits for fetches, hydrates the framework, and renders the final result. What you see is the end state.

Most AI crawlers don't do that. GPTBot, ClaudeBot, and PerplexityBot generally fetch the raw HTML document and read what's there. No rendering queue, no waiting for your useEffect to resolve. Google is the notable exception — Googlebot renders JavaScript, and AI Overviews build on that index — but assume every other AI system reads the initial response only.

So there's a gap between what humans see and what machines can reliably extract. And the gap is invisible precisely because everything looks fine.

The one thing most devs get wrong
DevTools Inspector is not raw HTML.

The Elements panel shows the live DOM — after JavaScript has run. It will happily show you content that no AI crawler will ever see.

What you want is View Source (Cmd/Ctrl + U). That's the actual document the server sent.

If your content is in the Inspector but not in View Source, it's client-rendered. That's the whole failure mode in one sentence.

The test itself
Pick your poison - all three take under a minute.

Option 1 — View Source. Cmd/Ctrl + U, then Cmd/Ctrl + F and search for your headline, your price, your address. Not there? That's your answer.

Option 2 — Disable JavaScript. In Chrome DevTools: Cmd/Ctrl + Shift + P → type "Disable JavaScript" → reload. You're now looking at roughly what a non-rendering crawler gets.

Option 3 — curl. Fastest for a quick gut check:

curl -sL https://yoursite.com | grep -i "your headline here"
Want to actually read the text a crawler would extract? This strips tags and gives you the first 2000 characters of real content:

curl -sL https://yoursite.com | python3 -c "
import sys, re
h = sys.stdin.read()
h = re.sub(r'<(script|style)[^>]>.?</\1>', ' ', h, flags=re.S)
print(re.sub(r'\s+', ' ', re.sub(r'<[^>]+>', ' ', h))[:2000])
"
If that output is mostly navigation and boilerplate — or nearly empty — you've found your problem.

The four failure modes to look for
Alex's list is the useful part here, because it's specific enough to actually run against a page:

Empty HTML. The classic CSR single-page app. Your is
and everything else arrives via JS. Crawlers get a shell.
Missing business facts. Title, offer, location, product details. These are the things an AI needs to describe or recommend you. If they're not in the initial response, you're asking the model to guess.
Invisible offers. Pricing loaded from an API on mount. Very common, very costly — "how much does X cost" is exactly the kind of question people ask assistants.
Client-only content. Tabs, accordions, "load more," lazy-mounted sections. Content that exists but only after an interaction the crawler will never perform.

The fix for all four is the same shape: get the critical content into the initial HTML. SSR, SSG, ISR, or prerendering — whichever your stack supports. You don't have to abandon React. Next.js, Nuxt, Remix, SvelteKit all solve this by default; the problem is almost always plain CSR or content deliberately deferred to the client.

Layer two: structured data as insurance
Here's the nuance I'd add to the smoke test, because passing it isn't quite the finish line.

Even when your content is in the raw HTML, an agent can still flatten it if the structure is ambiguous. It gets the words but loses the relationships — which number is the price, which line is the address, what's the product versus a related item.

That's where JSON-LD earns its keep:

{
"@context": "https://schema.org",
"@type": "Product",
"name": "Acme Widget",
"offers": {
"@type": "Offer",
"price": "49.00",
"priceCurrency": "USD",
"availability": "https://schema.org/InStock"
}
}

That's an unambiguous, machine-readable copy of your hard facts, sitting right there in the initial HTML. It survives even a shallow skim.

So the rule of thumb ends up being two layers:

Critical content in the initial HTML — so it can be read at all.
Hard facts mirrored in structured data — so they can't be misread.
What this test does not prove
Being honest about the limits, since that's what makes a smoke test useful rather than a false comfort:

It doesn't mean your content is good, or that anyone will cite it.
It doesn't tell you whether AI crawlers are actually allowed in — that's robots.txt, a separate check entirely.
It doesn't replicate any specific product's retrieval and ranking pipeline. Nobody outside those companies can.
It catches one specific, common, expensive failure: important content hiding behind client-side timing. That's it. That's enough to be worth 30 seconds.

Run it on something today
Try it on your own site. Then try it on a competitor's — that part is genuinely educational, and you'll find broken ones faster than you'd expect.

If you'd rather not do it by hand, I maintain a set of free tools for this — no signup — that fetch server-rendered HTML the way a crawler would and score what's actually readable. But the curl one-liner above costs you nothing and catches most of it.

The cheapness is the point.

Thanks to @alexshev for the smoke-test framing and the four failure modes — this post is basically our comment thread, cleaned up.

Your React site might be invisible to ChatGPT — here's how to check in 30 seconds

Osama Mumtaz — Tue, 21 Jul 2026 16:33:20 +0000

Quick test before you read on. Run this against one of your own content pages:


bash
curl -s https://yoursite.com/some-page | grep -i "a sentence from your main content"
If that sentence doesn't come back, then as far as most AI crawlers are concerned, it isn't on your page. And that's a bigger deal than it used to be.

Why this happens
Google spent a decade building a rendering pipeline: Googlebot fetches your HTML, queues the page, runs a headless Chrome render, executes your JavaScript, then indexes the result. It's expensive and slow, and Google eats that cost because search is its business.

The AI crawlers — GPTBot, ClaudeBot, PerplexityBot, and the rest — mostly don't do this. They fetch your raw HTML response and move on. No render queue, no JS execution. Rendering the whole web at scale isn't worth it to them when their goal is harvesting text.

The practical consequence is blunt: the AI web is the no-JavaScript web. A page that looks perfect in a browser can be an empty <div id="root"></div> to the crawler deciding whether you're worth citing.

Who this hits
Client-side-rendered SPAs (plain CRA-style React, Vue, Angular without SSR): the worst case — crawlers see a shell.
Content behind "load more" / infinite scroll: only the initial HTML is visible.
JS-injected sections on otherwise-fine pages: the sneaky one. Your article renders server-side, but the pricing table you hydrate from an API — the part that answers the question — is invisible.
Frameworks with SSR/SSG (Next.js, Nuxt, SvelteKit) and classic server-rendered stacks (WordPress, Rails, etc.) are generally fine. WordPress, for all the jokes, is more AI-legible than a slick client-rendered SPA.

The three ways to check
curl + grep (above) — is your text in the raw HTML?
DevTools with JS disabled — reload and see what survives.
Fetch it like a crawler does — I built a free AI Snippet Simulator (no signup) that pulls your server HTML the way GPTBot would and reports what a model can actually read and cite. If it says "too little readable text," you've found the problem — that verdict is the crawler's experience of your page.
The fix, cheapest first
Already server-rendered? Nothing to do — just verify the JS-injected widgets.
On a framework that supports SSR/SSG? Turn it on. SSG for content pages is ideal: complete HTML at build time, fast for humans, fully legible to every crawler.
Can't adopt SSR? Server-render at least the critical content — the H1, the opening answer, key facts — and let JS enhance from there.
Don't forget metadata: JS-injected <title>, meta descriptions, and JSON-LD have the same problem. If a tag manager adds your structured data client-side, crawlers never see it. It belongs in the server-rendered <head>.

The one exception
Google AI Overviews inherit Googlebot's rendering, so JS content Google indexes can surface there. That's why some SPAs appear in AI Overviews while being totally absent from ChatGPT and Perplexity — different pipelines, different capabilities. Treat Google as the exception and raw HTML as the baseline.

Curious how many people here have actually checked — if you curl your own site, is your main content in the response? Genuinely interested in how common the SPA blind spot is in practice.

*(Leave `canonical_url` blank so dev.to is canonical; you still get the in-body link + referral + dev.to's own ranking. Tags: `webdev, ai, javascript, seo` — the `javascript` tag reaches a big, different audience than your first post.)*
---
## Reddit post (fresh angle: training vs search — for r/Agentic_SEO)
Post this in **r/Agentic_SEO** (the sub where you found that build post — it's smaller, on-topic, and friendlier to this than r/TechSEO's shill filter). Keep it link-free; share the tool only if asked.
**Title:** PSA: blocking GPTBot doesn't remove you from ChatGPT — they're completely different crawlers
**Body:**
keep seeing people add `User-agent: GPTBot / Disallow: /` and think they've "blocked ChatGPT." they haven't, and it's worth understanding why.
OpenAI runs (at least) three separate crawlers:
- **GPTBot** — training. this is the one everyone blocks.
- **OAI-SearchBot** — builds the index behind ChatGPT's *search* answers.
- **ChatGPT-User** — fetches a page live when a user asks ChatGPT about it.
so blocking GPTBot opts you out of model *training*, but you're still fully visible in ChatGPT's search answers and live browsing. two different pipelines, two different user agents. i keep finding sites that think they've disappeared from ChatGPT when they're still getting cited in it daily.
the flip side is the more common mistake imo: people do a blanket "block all AI" and accidentally kill OAI-SearchBot + PerplexityBot — which are the ones that actually *send traffic*. so they've de-indexed themselves from the AI answers while keeping the training crawlers that give nothing back.
and it's not just OpenAI — Anthropic splits ClaudeBot (training) from Claude-SearchBot (search), Google splits Google-Extended (Gemini training, a robots token not even a crawler) from Googlebot, etc. almost every company separates "train on me" from "cite me," which means you can allow one and block the other on purpose.
how's everyone here thinking about this split? blocking training but keeping search open? all-or-nothing? i keep going back and forth for content-heavy sites and would love to hear how others land on it.
*(If someone asks "is there a list of all these?" → then reply with the directory link + "full disclosure it's mine.")*
---
**One process note:** the training-vs-search insight now appears in your dev.to piece, your Reddit post, *and* your site's guides — that's fine and deliberate (it's the single most useful counterintuitive fact in this niche, and repetition across channels is how a point sticks), but don't reuse the *exact same wording* on-site vs off-site, which these don't. Go reply to those PH comments first — that's the time-sensitive one.

I checked which AI crawlers actually respect robots.txt — here's what I found

Osama Mumtaz — Fri, 17 Jul 2026 18:26:32 +0000

Most site owners have a robots.txt that mentions Googlebot and maybe Bingbot, and nothing else. But there are now at least 15 AI crawlers hitting your site — some collecting training data, some powering live answers in ChatGPT and Perplexity — and they do not all behave the way their documentation claims.

I spent a while digging into how each one actually behaves and building free tools to check it, and a few things genuinely surprised me. Here's the rundown.

The two jobs AI crawlers do (and why blocking one doesn't block the other)

Before the compliance stuff, the single most common misconception: "blocking ChatGPT."

AI companies run separate crawlers for separate jobs, under different user-agent strings:

Training crawlers collect content to train future models. GPTBot (OpenAI), ClaudeBot (Anthropic), CCBot (Common Crawl).
Search / live crawlers fetch pages so an assistant can cite them in a live answer. OAI-SearchBot, PerplexityBot, ChatGPT-User.

So when someone adds this to block "ChatGPT":
User-agent: GPTBot
Disallow: /

...they've opted out of training, but they're still fully visible in ChatGPT's search answers, which use OAI-SearchBot. Different pipeline, different user agent. Blocking one has zero effect on the other.

The compliance findings

Here's where it gets interesting. Not every crawler honors robots.txt, and the docs don't always match reality:
| Crawler | Operator | Respects robots.txt? |
|---|---|---|
| GPTBot | OpenAI | ✅ Yes (documented) |
| OAI-SearchBot | OpenAI | ✅ Yes |
| ClaudeBot | Anthropic | ✅ Yes |
| CCBot | Common Crawl | ✅ Yes |
| Google-Extended | Google | ✅ Yes (it's a token, not a crawler — more below) |
| Applebot-Extended | Apple | ✅ Yes (also a token) |
| PerplexityBot | Perplexity | ⚠️ Partial — Cloudflare reported undeclared fetching that bypassed blocks |
| Perplexity-User | Perplexity | ❌ No — skips robots.txt by design |
| Bytespider | ByteDance | ❌ No — widely reported to ignore it entirely |
| cohere-ai | Cohere | ❓ Undocumented — can't verify |
Three things worth calling out:
1. Bytespider (ByteDance) just doesn't care. No published documentation, crawls aggressively, and multiple independent reports show it fetching pages on sites that disallowed it. If you actually want it gone, robots.txt won't do it — you need a firewall/CDN rule matching the user agent.
2. Perplexity-User ignores robots.txt on purpose. Perplexity's own docs say so: because the fetch is triggered directly by a user asking a question, they treat it like a browser, not a crawler. Reasonable framing, but it means "block Perplexity via robots.txt" is only half true.
3. Google-Extended and Applebot-Extended aren't crawlers at all. They're product tokens. You'll never see them in your logs. Regular Googlebot/Applebot fetches your pages as always; the token just governs whether that already-crawled content may train Gemini / Apple Intelligence. Which leads to the mistake I see most often...

The mistake: thinking Google-Extended controls AI Overviews

It doesn't. Blocking Google-Extended:

✅ Opts you out of Gemini training
❌ Does nothing to AI Overviews (those are a Search feature, built from the Search index)
❌ Has zero effect on your Google rankings There's no clean "opt out of AI Overviews" switch — the only levers are nosnippet / max-snippet, which also shrink your normal search snippets. So blocking Google-Extended is a rare free opt-out: you lose nothing in Search, you just leave Gemini's training set. ## The other thing that quietly breaks AI visibility: JavaScript Here's one that catches SPAs hard. Most AI crawlers don't execute JavaScript. Unlike Googlebot — which has a full headless-Chrome rendering pipeline — GPTBot, ClaudeBot, and PerplexityBot mostly fetch your raw HTML and move on. So if your content is client-side rendered:


html
<body>
  <div id="root"></div>
  <!-- content injected by JS after load -->
</body>
...that's roughly what the AI web sees of your site: an empty shell. The quick test:

curl -s https://yoursite.com/your-page | grep -i "a phrase from your main content"
If the phrase isn't in the response, it isn't in your server-rendered HTML, and AI crawlers can't read it. The fix is SSR/SSG (Next.js, Nuxt, etc.) or pre-rendering for bot user agents. Google is basically the only exception here — assume everything else reads raw HTML only.

What I built out of all this
I got tired of not being able to answer basic questions — can ChatGPT even see this site? which bots am I accidentally blocking? — and the existing "AI SEO" tools are mostly $30–500/mo SaaS. So I built free browser-based versions of the pieces I needed and put them at geoprompttracker.com:

A directory of every AI crawler with the compliance notes above and copy-paste allow/block rules per bot
An AI Crawler Access Checker that reads your live robots.txt and shows which of the 15 bots you allow or block
An llms.txt generator/validator for the emerging llms.txt convention
A robots.txt generator with per-bot toggles (block training, allow search — or whatever split you want)
Stack, for the curious: Next.js App Router on Vercel, everything static or client-side, no accounts, no signup. The few tools that fetch a URL go through a rate-limited proxy with an SSRF guard; nothing is stored.

The 5-minute audit, if you do nothing else
Decide your policy: training vs. search, per operator. They're separate choices.
Check what your robots.txt currently says with a crawler checker — a lot of "block all AI" configs accidentally kill Perplexity/OAI-SearchBot, which are the ones that send you traffic.
curl a key page and confirm your content is in the raw HTML, not JS-injected.
For the bots that ignore robots.txt (Bytespider, Perplexity-User), use firewall rules if you actually care.
I'd genuinely like feedback on the bot-compliance data — if you've got server logs showing different behavior (especially anything that's stopped respecting robots.txt recently), I'd love to hear it. This space changes month to month and I'm refreshing the list as it does.

---
### Two small notes before you hit publish
1. **Leave `canonical_url` unset.** This article is original to dev.to, so let dev.to be canonical — you still get the backlinks in the body + referral traffic + dev.to's own Google ranking pointing readers to you. (Only set a canonical if you were reposting something already on your site.)
2. **Tags:** I used `webdev, ai, seo, programming` — all have real audiences on dev.to. You can swap `programming` for `nextjs` if you'd rather reach that crowd.
After it's live, drop the link in the r/TechSEO post too — the article and the `/bots` directory reinforce each other. Want me to draft the **r/TechSEO post** next so you can fire both today while the HN account warms up?