I published an open dataset of every AI crawler user agent (GPTBot, ClaudeBot + 26 more)

While building AI-crawler tooling I kept needing the same thing: a current,
machine-readable list of AI crawler user agents. Every blog post has a partial
list, half of them outdated, none of them structured. So I published mine as an
open dataset.

GitHub: https://github.com/osamamumtaz01/ai-crawler-user-agents
Live endpoint (always current, CORS enabled): https://geoprompttracker.com/data/ai-crawlers.json

28 crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot,
Google-Extended, Bytespider, Applebot-Extended, Meta-ExternalAgent, and the
rest. JSON + CSV, CC BY 4.0. Also on Kaggle and Hugging Face.

The field that actually matters: purpose

The mistake I see constantly: treating "AI bots" as one thing. AI companies run
separate crawlers for different jobs:

• Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content to train future models.
• Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index pages so assistants can cite them in live answers.

Blocking a training crawler is an opt-out of model training. Blocking a search
crawler removes you from AI answers entirely. Very different decisions — and
because each company uses different bot names for each job, people block the
wrong one all the time.

So each entry carries a purpose field (training / search / both), which
makes policy decisions programmable:

const { crawlers } = await (
  await fetch("https://geoprompttracker.com/data/ai-crawlers.json")
).json();

// "Don't train on my content, but keep citing me"
const rules = crawlers
  .filter((b) => b.purpose === "training")
  .map((b) => `User-agent: ${b.userAgent}\nDisallow: /`)
  .join("\n\n");

The judgment call: robotsCompliance

The other field worth explaining. It rates whether each bot honors robots.txt
in practice — not just what the vendor claims. yes, partial, no, or
unknown, and every rating comes with a sourced robotsNote explaining the
reasoning (e.g. Perplexity is rated partial because independent
investigations observed undeclared fetching despite documented compliance).

That's also why robots.txt alone isn't enforcement — for bots rated no or
unknown, pair it with a WAF/CDN rule if blocking actually matters to you.

Other things you can do with it

• Detect AI crawler traffic in middleware or log analysis
• Audit which bots your current robots.txt actually allows
• Research crawler behavior over time (the dataset is reviewed monthly)

If you spot a crawler I'm missing or a compliance rating that's gone stale,
issues and sourced corrections are very welcome — especially for the unknown
entries.

Comments

[Mike Viewfy]

Matching on the header string alone trusts a claim the sender writes about itself, which is exactly where a no or unknown rating gets expensive. Most of these operators publish their crawler IP ranges as JSON, so one more field pointing at that endpoint would make the WAF pairing you mention verifiable instead of name-based. It would also give you a path on some unknown entries: logged hits from outside the published range are their own answer.

[Osama Mumtaz]

This is the best suggestion I've had on this dataset - you're right that matching the header alone means trusting a claim the sender writes about itself, and that's precisely where a no or unknown rating stops being actionable.

I went and checked which operators actually publish ranges, and enough do to make the field worth adding: OpenAI ships separate files per bot (gptbot.json, searchbot.json, chatgpt-user.json), Google publishes common-crawlers/special-crawlers/user-triggered-fetchers, Perplexity has perplexitybot.json, and Apple has applebot.json. Anthropic is the notable gap - I couldn't find a published JSON for ClaudeBot, so that one stays reverse-DNS-only for now.

So an ipRangeUrl field is going in. Two things I like about it beyond the WAF pairing you mentioned:

• It makes robotsCompliance falsifiable rather than just documented. Right now the ratings cite investigations; with a range endpoint anyone can re-derive them from their own logs.
• Your point about the unknown entries is the part I hadn't thought through. Hits claiming a user agent from outside the published range aren't ambiguous - they're either spoofing or undeclared infrastructure, and either way that's a finding rather than a shrug.

Will add it to the schema and note explicitly where no endpoint exists, since "operator publishes nothing" is itself useful signal.

[Osama Mumtaz]

@mike_viewfy
Shipped - it's in the dataset now, and the gaps (Anthropic included) are marked explicitly."

[Dmitry Sergeev]

ngl it's wild how many of these bots just ignore robots.txt, thanks for the dataset

[Julian Neagu]

The robotsCompliance field is a smart addition. Knowing what a bot claims and what it actually does are two very different things.

[Osama Mumtaz]

Thanks Julian - that gap is exactly why I added it. A vendor saying "we respect robots.txt" and a crawler actually doing it are separate claims, and only one of them shows up in your logs. Rating observed behavior felt more honest than just restating documentation.