What 4,700 bot visits taught me about AI crawlers (and a tool to check any site's robots.txt

Bhushan Arun Ladde — Thu, 05 Nov 2026 03:03:19 +0000

A few weeks ago I wanted to answer a simple question: how many of the "visitors" on my small side projects are actually people?

The answer turned out to be: not many. Most of the traffic was crawlers, a lot of them from AI companies, and I had no easy way to see which ones, how often, or where they came from. Server logs tell you a User-Agent string and an IP, and the User-Agent can say anything it likes.

So I built BotSpotter. It's a live map of AI crawlers hitting participating sites, open source under MIT. This post is about what the first week of data showed me, because some of it surprised me.

1. About 1 in 18 "big-name" bots is lying

OpenAI, Google, Microsoft, Apple, Perplexity, DuckDuckGo and Common Crawl all publish the IP ranges their crawlers use. BotSpotter checks every hit that claims to be one of those bots against the published ranges, in-process, before the IP is discarded.

After 1,500 such hits: 1,425 verified, 75 impostors. The most popular disguise is not GPTBot, it's Googlebot (40 of the 75), followed by GPTBot (20), Bingbot (12) and PerplexityBot (3). If you rate-limit or whitelist by User-Agent, this is the traffic you're letting through.

2. Most AI crawling is anonymous

Of 4,770 bot visits so far, 2,494 (56%) came from bots nobody has identified — no registry entry, no documented User-Agent, no published ranges. They're fingerprinted by UA family, header order and network, so the same bot keeps the same name ("Onyx Shark #5358"), but we don't know who runs them. The named AI companies are the visible tip; the bulk of machine traffic doesn't announce itself.

3. A new domain gets a crawl burst, then it decays

The first days after launch showed 500+ bot hits a day. Two weeks later it's ~320. Nothing broke: every search and SEO crawler does a discovery pass on a new domain (Googlebot, Bingbot, Ahrefs, Semrush, Meta all fetched every URL in the sitemap), then drops to maintenance re-checks. If you're measuring bot traffic on a fresh site, the first week is not your baseline.

4. Where the bots live

By hosting network: Google Cloud (2,117 hits), AWS (293), Hetzner (287), OVH (270), Azure (170), Apple (165), Meta (85), DigitalOcean (77). Hetzner and OVH being that high is a tell — a lot of scrapers and unidentified crawlers rent cheap European VMs.

Check any site's robots.txt against AI crawlers

The feature people use most isn't the map. It's botspotter.app/check: type any domain and it fetches the robots.txt and evaluates it against 16 AI crawler tokens using RFC 9309 rules (a named group beats *, longest path match wins, Allow wins a tie). You get a grade per crawler — allowed, kept out of some paths, or blocked — grouped by what the operator says the traffic is for: training, AI search, or agents fetching for a live user.

Try it on a site you know: https://botspotter.app/check?site=nytimes.com

If you own the site, it gives you a paste-ready block for the training crawlers. And there's a second tool on the same page: paste an IP from your logs, pick the bot it claims to be, and it tells you whether the IP is in the operator's published ranges.

One honest caveat the page itself makes: robots.txt is a request, not a wall. The impostors above don't read it.

How it's built

Sensors: a one-line middleware (FastAPI/Django/Flask) or a Caddy/Nginx log tail. They drop browsers locally and send only bot hits — header names, User-Agent, IP, country. No page content, no cookies.
Ingest (FastAPI) → Redpanda → an aggregator → Redis counters + TimescaleDB history. The IP is used once for GeoIP, ASN and the verification check, then dropped. No IP is stored anywhere.
Bots get a stable name from a hash of their fingerprint; humans on the portal are counted with a daily salted hash, no cookies.
Live map: server-sent events into MapLibre. Deploys go through CI rehearsal of the whole stack, an automated change gate, approval, and auto-rollback — it's a hobby project, but it went down twice on launch day and I didn't enjoy it.

Code: github.com/bhushanladde02/botspotter

What I'd like from you

If you run a site, adding it is a GitHub sign-in and one line, and you get a live badge for your footer. More sensors means better numbers for everyone, and right now there are two.

And I'm curious: what's the weirdest bot in your logs?

DEV Community: Bhushan Arun Ladde