DEV Community

Osama Mumtaz
Osama Mumtaz

Posted on

I published an open dataset of every AI crawler user agent (GPTBot, ClaudeBot + 26 more)

#webdev #opensource #ai #seo

While building AI-crawler tooling I kept needing the same thing: a current,
machine-readable list of AI crawler user agents. Every blog post has a partial
list, half of them outdated, none of them structured. So I published mine as an
open dataset.

GitHub: https://github.com/osamamumtaz01/ai-crawler-user-agents
Live endpoint (always current, CORS enabled): https://geoprompttracker.com/data/ai-crawlers.json

28 crawlers — GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot,
Google-Extended, Bytespider, Applebot-Extended, Meta-ExternalAgent, and the
rest. JSON + CSV, CC BY 4.0. Also on Kaggle and Hugging Face.

The field that actually matters: purpose

The mistake I see constantly: treating "AI bots" as one thing. AI companies run
separate crawlers for different jobs:

  • Training crawlers (GPTBot, ClaudeBot, Google-Extended) collect content to train future models.
  • Search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) index pages so assistants can cite them in live answers.

Blocking a training crawler is an opt-out of model training. Blocking a search
crawler removes you from AI answers entirely. Very different decisions — and
because each company uses different bot names for each job, people block the
wrong one all the time.

So each entry carries a purpose field (training / search / both), which
makes policy decisions programmable:

const { crawlers } = await (
  await fetch("https://geoprompttracker.com/data/ai-crawlers.json")
).json();

// "Don't train on my content, but keep citing me"
const rules = crawlers
  .filter((b) => b.purpose === "training")
  .map((b) => `User-agent: ${b.userAgent}\nDisallow: /`)
  .join("\n\n");
Enter fullscreen mode Exit fullscreen mode

The judgment call: robotsCompliance

The other field worth explaining. It rates whether each bot honors robots.txt
in practice — not just what the vendor claims. yes, partial, no, or
unknown, and every rating comes with a sourced robotsNote explaining the
reasoning (e.g. Perplexity is rated partial because independent
investigations observed undeclared fetching despite documented compliance).

That's also why robots.txt alone isn't enforcement — for bots rated no or
unknown, pair it with a WAF/CDN rule if blocking actually matters to you.

Other things you can do with it

  • Detect AI crawler traffic in middleware or log analysis
  • Audit which bots your current robots.txt actually allows
  • Research crawler behavior over time (the dataset is reviewed monthly)

If you spot a crawler I'm missing or a compliance rating that's gone stale,
issues and sourced corrections are very welcome — especially for the unknown
entries.

Top comments (0)