If you go looking for AI traffic in your own server logs, you probably grep for GPTBot and stop there. That is the wrong number, and it is wrong in a way that flatters you.
OpenAI runs three separate agents. They mean three different things.
- GPTBot is the training crawler. It feeds model weights. It will never send you a visitor.
- OAI-SearchBot is the retrieval crawl. This is the fetch that runs before an answer cites a page. If you care about being cited, this is the one to watch.
- ChatGPT-User is not a crawler at all. It fires when a real person clicks a citation or asks ChatGPT to open a URL. That is a human who already arrived.
Collapse them into a single "OpenAI" row and you are adding a trainer, a retriever and a reader together, then calling the total your AI traffic.
What it looks like across 14 sites
Seven days to 31 August 2026, one Cloudflare account, 14 properties:
| Agent | Purpose | Fetches |
|---|---|---|
| GPTBot | training | 11,391 |
| ChatGPT-User | a human who clicked | 9,656 |
| OAI-SearchBot | retrieval, precedes a citation | 2,930 |
Retrieval is 12.2% of OpenAI's footprint on the network. Training outweighs retrieval 3.89 to 1.
The bar that matters is the short one. Everything upstream of a citation is happening in that 12.2%, and most dashboards never separate it out.
This is not an OpenAI quirk
The same split shows up at every vendor that runs a separate user agent for human-initiated fetches. Same network, same window:
| Vendor | Crawler | Human |
|---|---|---|
| OpenAI | GPTBot 11,391 | ChatGPT-User 9,656 |
| Anthropic | ClaudeBot 6,881 | Claude-User 1,191 |
| Perplexity | PerplexityBot 1,749 | Perplexity-User 204 |
Three vendors, three crawler-versus-human pairs, and almost nobody separates those either.
How to run it on your own logs
If you are on Cloudflare, the data is already there. Query httpRequestsAdaptiveGroups in the GraphQL analytics API, group by clientRequestHTTPHost and the user agent, and split the buckets by exact agent string rather than by vendor. The whole finding lives in that one grouping decision.
If you are on nginx or Apache, the same split is a grep away. The user agent strings are distinct and stable.
The caveat that matters
These are fetches, not citations. A fetch is two rungs below a citation on the ladder and most fetches never become one. They come from a different instrument than citation counts do, they measure a different event, and the two must never be added together.
Bot identity here is self-reported in the user agent. We are not verifying by IP range, so treat these as what the requester claimed to be.
One more honest bound: this is one network of 14 sites over seven days. The ratio will differ on yours. The point is not our number, it is that "OpenAI" was never one number to begin with.
Top comments (0)