If you count AI crawler traffic from your access logs, you probably count by User-Agent. A User-Agent header is written by the client. So that count shows what clients said about themselves.
I reanalysed fifteen days of complete nginx access logs from one origin server behind the Cloudflare CDN. The log holds 345,808 requests. Of these, 46,155 carried one of 22 crawler tokens published by search and AI operators, such as GPTBot, ClaudeBot and Googlebot. The same logs were the base of my earlier llms.txt report (DOI 10.5281/zenodo.23018429). There I attributed requests to crawlers the simple way: every request whose User-Agent contained a crawler token. In Section 5 I noted that about a third of those requests bypassed the CDN. This article is the reanalysis.
What the split showed
29,691 token requests (64.3%) came through the CDN. In the origin log they carry the address of a CDN edge node, so the origin cannot check them by address. The other 16,464 (35.7%) came directly to the origin.
The two routes behave differently.
| Response | Through CDN | Direct |
|---|---|---|
| 2xx | 72.5% | 0.2% |
| 3xx | 6.4% | 57.8% |
| 4xx | 21.0% | 42.1% |
robots.txt was fetched 2,135 times through the CDN and 0 times directly. sitemap.xml was fetched 1,266 times through the CDN and 0 times directly.
A tidy string proves little
11,276 direct requests were under tokens whose operators publish address ranges (Google, OpenAI, Microsoft, Apple, Perplexity, DuckDuckGo). 0 of them came from those ranges. I downloaded the range files 17 days after the log window.
The 11,276 include 5,106 direct requests under the Googlebot token. 99.9% of those used one and the same User-Agent string, and none came from Google's ranges. So a tidy, consistent string proves little.
The CDN route is not verification either. 863 requests presented Google-Extended. Google documents it as a robots.txt product token that is not sent as a User-Agent. 293 of those 863 came through the CDN. The CDN route only means the route does not contradict the name.
The direct share differs by token
Zero direct requests for YandexBot, DuckDuckBot, cohere-ai and MistralAI-User. For bingbot it was 1 request of 2,503. The majority were direct for Google-Extended (66.0%), Perplexity-User (64.4%), Bytespider (62.8%), Claude-User (59.5%), Googlebot (58.3%), Amazonbot (52.8%) and OAI-SearchBot (50.2%).
Ranking changes too. Rank by declared counts versus CDN-routed counts, and second place goes from GPTBot to Claude-SearchBot.
One earlier finding still holds under the stricter rule. 0 of 46,155 token requests asked for /llms.txt or /llms-full.txt. A zero over all declared requests is a zero over every subset.
Four counts per token
I propose reporting four counts for each token.
| Count | Meaning |
|---|---|
| Declared | Requests presenting the token |
| Route-consistent | Behind a CDN, requests that came through it |
| Verified | Source confirmed by a method the operator supports |
| Unverifiable | No check possible |
Verified means one of two methods. The first is published address ranges. The second is a reverse DNS lookup followed by a forward lookup of the returned name. Google and Microsoft describe the second method for their crawlers.
Count Google-Extended-style tokens on their own line. Also state whether the log holds the client address at all.
What to do with your nginx log this week
- Count requests per crawler token by User-Agent. This is your declared column.
- Split each token's count by route. If you use a CDN, separate requests that came through it from direct ones.
- Check direct requests against the address ranges the operator publishes, or with reverse and forward DNS lookups.
- Cloudflare passes the original client address in the CF-Connecting-IP request header. If your origin logs that header, you can check CDN-routed requests against the same ranges. I think the next measurement at my origin should log it.
- Write down what stays unverifiable.
Wording
A sentence about what crawlers do needs the verified count. If the verified count is zero or unavailable, write "clients presenting the token".
Limits
This is one origin, one operator and fifteen days. Three hostnames were pooled in one log without the hostname. The proportions will differ elsewhere. I do not claim every direct request was fake. I claim only that the declared count cannot tell which were.
The paper with all the numbers is open on Qeios: Report AI Crawler Traffic in Two Columns, Declared and Verified (DOI 10.32388/SRD4TB). The log study it builds on: DOI 10.5281/zenodo.23018429.
Evgenii Arsentev, MD, PhD - creator of Arsminal; founder of ArsentevAI and Dad4Kids. ORCID 0000-0002-9120-7298 · Wikidata Q141398931
AI assistance: drafted and edited with an AI tool; the data, measurements and conclusions are my own.
Top comments (0)