One finding, and it is about an instrument rather than a threat.
Cloudflare's AI Crawl Control has an Optimization tab. It shows two tables. The first is labelled "Understand what content is used in AI answers." The second is labelled, word for word:
Check what AI Assistants are trying to access, but can't.
Read that second label the way it is written. It says these are the things assistants want from you and are not getting. It reads like a content backlog.
Here is the top of that table on a site I work on:
/config.toml
/local.settings.json
/@fs/app/.env
/wp-config.php~
/config.env
Further down: id_ed25519, gcp-key.json, service_account.json, terraform.tfstate, aws-exports.js, backend/settings.py.
Those are not unmet demand. That is a credential scanner working through a wordlist.
Why they are in there
The answer is in Cloudflare's own documentation for the same feature, and it is not hidden:
On the free plan, AI Crawl Control identifies AI crawlers based on their user agent strings.
Verification of who a crawler actually is, through bot-management detection, is a paid-plan feature. On the free plan, "AI assistant" means a request that wrote User-Agent: ChatGPT-User about itself.
A scanner sets that header and is counted as an AI assistant. Every probe it makes becomes a data point about what AI assistants want.
How much of the panel this is
On the sample I checked, over a single day: 79 percent of the "most visited pages" table, and 94 percent of the "demand signals" table, are requests for paths that do not exist and never have.
The provider breakdown says the same thing from another angle. One provider accounts for 92.5 percent of the volume, and it is the one whose user-agent string is the cheapest to imitate. The three providers that are harder to impersonate come to a couple of hundred requests between them, which is a far more believable number for genuine assistant traffic.
The part that is actually dangerous
A metrics panel that mislabels traffic is an annoyance. A metrics panel that mislabels traffic and then tells you what to do about it is something else.
"Check what AI Assistants are trying to access, but can't" is an instruction. Followed literally, on the data it is presenting, it says: create /config.toml. Create /wp-config.php. Publish /.env.
Nothing in the interface marks those rows as probes. There is no flag, no separate bucket, no note. They sit in the same table as your documentation pages, sorted by the same column, styled the same way.
What I would do with it
Do not act on that table. Not one row, not ever.
Filter with an allowlist, not a blocklist. I tried a blocklist first, pattern-matching the obvious probe shapes, and it let id_ed25519 and terraform.tfstate through as legitimate. Listing the paths that actually exist on the site is the only filter that holds, because the probe wordlist is longer than your imagination.
Then keep what survives. Underneath the noise there is a genuinely useful signal that no other instrument gives you: which of your pages assistants actually fetch, as distinct from which pages rank. Those are different lists, and the difference is worth knowing.
The panel is not useless. It is just reporting the wrong thing loudly and the right thing quietly, and the label on top points at the loud half.
Disclosure: I work on Mnemoverse, a memory engine for AI agents. The percentages above come from one day of one site's own export. The Cloudflare documentation quoted is public and says the same thing to anyone who opens it.
Top comments (0)