DEV Community: roadleon

Thirty-Four Out of Thirty-Four. Then Zero Out of Ten.

roadleon — Fri, 25 Sep 2026 11:05:22 +0000

My last article ended by admitting something my own dashboard could not do. Four days before it went out, I had already fixed it — and the fix had an answer waiting.

The article was about twenty requests for an .env file that does not exist on my site. Every one of them said it was ClaudeBot. I could not tell you whether they really were, because my dashboard verified crawlers, not paths: it would report that ClaudeBot as a whole was 68% verified that week, and stop there. Which of its requests were in the verified 68% and which were not, it had no way to say.

That is now a column. Here is what it says.

Two words that had been collapsed into one

The fix is smaller than it sounds, and it is really two fixes. The first one I owe to a reader. When I published an article in early September about crawler verification, a commenter pointed out that my "unverified" number was doing two jobs at once: it counted requests that had been checked against the vendor's published IP ranges and failed, and it counted requests that had never been checked at all. A crawler with no published ranges and a crawler caught wearing a false name both came out as 0% verified. That is not a measurement. That is two different facts wearing the same number.

So the plugin now records whether a check actually ran, separately from whether it passed. The second fix follows from the first: once you know which individual requests were checked, you can total them by path instead of by crawler. The page list grew a column reading verified / checked, and paths where nothing has been checked yet say so in words rather than showing a zero.

The list, with the new column

This is the top ten pages for the seven days ending September 22, in the order the dashboard shows them.

Path (requests)	Verified / checked
`/` (40)	22 / 22
`/tag/news/` (21)	1 / 1
`/.env` (14)	0 / 6
`/topics/ダバオの運転事情/` (9)	6 / 6
`/read-document` (7)	0 / 3
`/tag/知育教室/` (7)	2 / 2
`/api/designer/v1/file-content` (6)	0 / 1
`/attachment_tag/favicon/` (6)	not checked
`/tag/lego/` (6)	2 / 2
`/tag/イスラム/` (6)	1 / 1

Read down the right-hand column and the table sorts itself into two piles. Most of it is a small Japanese site about living in Davao being read the way you would expect: the front page, some tag listings, an article about driving there. Three rows are not like the others, because three of these paths do not exist and never have. /.env is where a web application keeps its database passwords and API keys. /read-document and /api/designer/v1/file-content are endpoints belonging to software this site has never run. All three return 404.

Now add up each pile.

34 / 34

Checks on the real pages. Every one of them matched the vendor it claimed to be.

Six paths had been checked at least once. The seventh, /attachment_tag/favicon/, had not been checked at all, so it is in neither pile.

And the other pile.

0 / 10

Checks on the three paths that do not exist. Not one matched.

All three name ClaudeBot as their most frequent visitor.

Same site. Same week. Same instrument, running the same comparison against the same published list of addresses. The only thing that changes between the two piles is whether the thing being asked for is really there.

This is not an argument about ClaudeBot

It would be easy to read that as an accusation, so let me put the rest of the week's numbers next to it. Over those same seven days ClaudeBot made 599 requests to this site. 94% of them verified as Anthropic. 3% had not been checked yet. Which means that of everything I was able to check, almost all of it was exactly what it said it was.

Anthropic began publishing its IP ranges on August 18, and on the evidence of this site that list works. The overwhelming majority of traffic calling itself ClaudeBot here is Anthropic. That is the point. A name that is usually honest is a much better disguise than one that is usually suspect, and the ten failures are not distributed evenly across the site's traffic — they are sitting on the three paths where somebody would go looking for credentials.

One more detail, small but hard to unsee. Last week /read-document was in this same table, and the name on it was PerplexityBot. This week the same non-existent path is being requested under ClaudeBot. The path did not change. The costume did.

What the new column cannot reach

Splitting "checked and failed" from "never checked" had a second effect I did not plan for: it made visible how much of this traffic never reaches the gate at all. Here is the whole week, sorted by what the instrument was able to do with each request.

Requests	Share	What the instrument could do
1,809	65%	No verification method published by the vendor
~800	29%	Checked, matched the vendor it claimed
~85	3%	Checked, did not match
~70	3%	Not checked yet

AI crawler requests to davaonline.net, seven days ending 2026-09-22. The first row is exact: Meta's two crawlers, Bytespider and Amazonbot publish neither IP ranges nor reverse DNS, so no check is possible. The other three are derived from the dashboard's rounded percentages and are accurate to within a few requests.

Two thirds of it cannot be checked by anyone, because the vendors have published nothing to check against. Meta alone accounts for most of that. An absent failure is not a pass: for those 1,809 requests I have no more idea who sent them than I did before I built any of this.

What this does not prove

Ten checks is ten checks. The three paths received 27 requests between them; only 10 had been checked by the time I looked. The other 17 arrived before the column existed and will never be checked, because the plugin does not re-examine a request it has already processed. I am not counting them, in either direction.
A failed check is not an identification. It means the address did not appear in the vendor's published list. It does not tell me who did send the request, and nothing here should be read as a claim about Anthropic, whose name was used without its involvement.
My own list was three weeks old. A verification result is only as fresh as the snapshot it compares against, and mine dated from September 3. I refreshed it on September 22 before writing this. The published ranges for Anthropic, Perplexity, Mistral, Common Crawl and DuckDuckGo had not changed by a single entry — so none of the failures above are explained by a stale list. OpenAI's had: 22 ranges added for ChatGPT-User. Which means ChatGPT-User's 35% that week is an undercount, and since checked requests are never re-checked, it is an undercount I cannot repair. It will simply be right from now on.
Verified means the company, not the crawler. Anthropic publishes one combined list covering all three of its crawlers, so a match confirms the request came from Anthropic — not which of its crawlers it was.
Cached pages are invisible. When a page cache answers, PHP never runs and nothing is recorded. Every count here is a lower bound.

How to look at your own

If you install it, the thing worth doing is the one I did here: read the page list first and the crawler list second. A crawler name tells you what something claims to be. A path tells you what it wanted. When a path you have never heard of appears on that list, the new column will tell you whether the name attached to it survived contact with the vendor's own records — and when it says not checked, that is an honest blank rather than a zero pretending to be a verdict.

I spent two articles calling this gap a defect in my own tool. It was more useful to close it than to keep describing it.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

The diagnostic service alongside it, AILYS Doctor, is a different thing. Lens tells you what happened; Doctor tells you why, and what to change. It is free to use during its data-collection period, and any future pricing will be announced on the site.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.

Something Asked for My .env File Twenty Times. It Said It Was ClaudeBot.

roadleon — Tue, 22 Sep 2026 00:36:42 +0000

In the last seven days, something asked this site for its .env file twenty times. It said it was ClaudeBot.

An .env file is where a web application keeps its secrets: database passwords, API keys, cloud credentials. It is not meant to be reachable over HTTP at all. On this site it is not: the path returns a 404, as it should.

No training crawler wants that file. There is nothing in it to train on. A crawler that asks for it is not reading your site; it is checking whether you left your keys in the door.

What the list actually looked like

This is the panel that shows which pages AI crawlers read, for the seven days ending September 18. I have left it in the order the dashboard shows it.

Path (requests)	Top crawler
`/` (27)	Bytespider
`/.env` (20)	ClaudeBot
`/tag/news/` (12)	Bytespider
`/pickup-info/mypal-wifi/` (9)	ChatGPT-User
`/tag/ココナッツ/` (7)	meta-externalagent
`/tag/フィリピンイーグル/` (7)	meta-externalagent
`/read-document` (6)	PerplexityBot

Most of that is a small Japanese site being read the way you would expect: the front page, a few tag pages, an article about pocket wifi. Two rows are not like the others. Neither /.env nor /read-document exists here. Both return 404.

My own instrument could not tell me

The obvious question is whether those twenty requests really came from Anthropic. My dashboard has a Verified column for exactly this, and over those seven days ClaudeBot came out at 68% verified — meaning about a third of its requests could not be confirmed as Anthropic.

Which does not answer the question. The column reports a percentage per crawler, not per path. I cannot ask it whether the /.env requests were among the verified ones or the unverified ones. That is the same defect I described two weeks ago and have still not fixed.

68%

Share of ClaudeBot requests confirmed as coming from Anthropic that week.
Which tells you nothing about the twenty that asked for the .env file. Per-path verification is the thing my dashboard is missing.

I did not have to guess

On August 31, the security firm GreyNoise published what it had been watching. Between July 28 and August 23 it counted 824 addresses running scans that forged AI crawler names — six crawler names belonging to four AI companies, with two Amazon crawler names appearing in greater volume still. What they asked for: /.env, /.env.production, /.env.bak, /.aws/credentials.

The detail that stopped me is not the volume. It is the tell they found.

0

Requests for robots.txt in that entire scanning campaign.
The real ClaudeBot asks for robots.txt more than anything else — it is 12% of its traffic. The impostors never asked once.

That is a beautifully simple signature. A crawler that respects your site checks the rules first; it is the most boring thing it does, and it does it constantly. Something that wants your credentials has no reason to look at a file telling it where it may not go.

Why the crawler name was the target

These names were not chosen at random. Site owners have spent the past two years being told to add AI crawlers to their allowlists — to let GPTBot and ClaudeBot and PerplexityBot through so their content gets read, or to block them deliberately if they would rather not be. Either way, the name became a thing that firewalls and plugins make decisions about.

A user agent is a header the client writes about itself. It is a claim, not an identity. I demonstrated this on my own site two weeks ago by asking it for a page while calling myself ClaudeBot, and watching my dashboard record the visit as ClaudeBot. That took one line of curl. The scanners are doing the same thing, at scale, against the specific names that people have been trained to wave through.

An allowlist keyed on a name that anybody can type is not a control. It is a convention being mistaken for one.

What to actually do

Do not serve secrets over HTTP, ever. This is the one that matters. If .env is unreachable, the scan is a wasted 404 and nothing else. Everything below is secondary to this.
Check addresses, not names. OpenAI, Anthropic, Perplexity, Common Crawl and Google all publish IP ranges or reverse-DNS schemes now. If a request claims a vendor that publishes one and does not match it, the name is worth nothing.
Watch for what is missing. A client that never fetches robots.txt but does fetch config paths is not a crawler, whatever it calls itself. That signature costs nothing to look for.
Treat crawler traffic as traffic. The habit of filing "AI crawler" away as a separate, benign category is precisely what this campaign is built to exploit.

What this instrument cannot see

Verification is per crawler, not per path. The central question of this article — were those twenty requests Anthropic or not — is one my own dashboard cannot answer. I am reporting the limit rather than writing around it.
One site, seven days. Twenty requests is a tiny sample. It is consistent with a campaign that GreyNoise measured at 824 addresses; it does not independently prove anything.
I have not attributed anything. I do not know who sent these, and nothing here should be read as a claim about Anthropic, whose name was used without its involvement.
Cached pages are invisible. When a page cache answers, PHP never runs and nothing is recorded. Every count here is a lower bound.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which you should factor in. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. The page list described here is on the main dashboard. If a path you have never heard of appears on it, that is worth ten seconds of your attention.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.

Meta Reads More Than Everyone Else Combined. None of It Was a Question.

roadleon — Fri, 18 Sep 2026 04:42:46 +0000

One company reads this site more than every other AI company put together. In thirty days, not one of those requests came from a person asking a question.

I have been measuring a small WordPress site for a few months now, recording only AI crawlers and sorting each request by what the crawler is for. Over the thirty days ending September 11, it was visited 11,818 times, and 8,979 of those landed on pages with writing on them.

Here is who came.

Vendor	Share	Of which, live questions
Meta	52.3%	0%
Apple	12.0%	0%
Anthropic	9.6%	6%
Perplexity	8.3%	93%
OpenAI	7.9%	21%
ByteDance	5.8%	0%
Common Crawl	2.1%	0%
Amazon	1.5%	0%
Google	0.5%	0%

52.3%

Share of all AI crawler traffic on this site that was Meta.
Everyone else, added together, accounts for 47.7%.

The first column is the one people quote. The second column is the one that decides whether any of it can ever reach you.

Three crawlers, three different jobs

Meta does not run one crawler. It runs several, and it says plainly what each is for. Three of them are AI crawlers, and this instrument watches all three.

meta-externalagent — Meta's documentation says it "crawls the web for use cases such as training foundation AI models or improving products by indexing content directly."
meta-webindexer — it "navigates the web to improve Meta AI search result quality for users," which is the crawler whose job includes getting citations right.
meta-externalfetcher — it fetches individual links at a user's request. This is the one that means somebody asked something a few seconds ago.

Over thirty days, the split between them was not close.

Meta crawler	Requests
meta-externalagent (training)	6,027
meta-webindexer (answer quality, citations)	155
meta-externalfetcher (a person asked)	0

Thirty-nine requests collecting material for a model, for every one request improving the thing that could cite you. And the third row is not small. It is empty.

0

Requests from Meta's user-initiated fetcher, in thirty days.
It did not appear once. Five other vendors run one — OpenAI, Perplexity, Anthropic, DuckDuckGo, Mistral — and all five of them appeared.

The path back exists. It is barely used.

This is what makes the ratio worth looking at rather than just complaining about. Meta is not a company without a route from its answers to your site. It documents one. meta-webindexer exists precisely so that Meta AI can search well and attribute accurately.

On this site, that crawler did 155 requests while the training collector did 6,027. The machinery that might send a reader back is running at about two percent of the machinery that takes material away.

Compare the shape of that with the two vendors at the other end. Perplexity's traffic here is 93% user-initiated: almost everything it does on this site is a person, right now, waiting for an answer. OpenAI is 21%. Meta is zero.

That is not a moral ranking. A training crawler is not doing anything wrong by collecting training data; that is its job, and Meta says so in the open. But the mix tells you what a given company currently wants from your writing, and the mixes are not remotely alike.

An awkward footnote: I cannot prove any of this was Meta

Everything above rests on 6,182 requests that said they were Meta. Meta publishes no IP ranges and no reverse-DNS scheme for these crawlers, so there is no way to confirm that any single one of them came from Meta. My dashboard marks the entire vendor with a dash where other vendors have a percentage.

So the largest reader of this site is also the one I can least verify. I wrote about that problem a week ago and it has not improved: most requests on this site still arrive with no proof of who sent them, and none of Meta's ever can.

I do not think these requests are forged. The volume is steady, the behaviour is consistent, and there is no obvious reason to impersonate a crawler that most site owners have never heard of. But I am reporting a number I am not able to check, and that should be said out loud rather than buried.

Which brings us back to the two

In the first piece I wrote, this site had been read about ten thousand times in a month and had received two human arrivals, one from ChatGPT and one from DuckDuckGo.

I did not know it at the time, but the single largest contributor to that reading number was the one vendor structurally least likely to appear in the arrival number. The gap between "read" and "visited" was not evenly caused. It had a majority shareholder.

What this does not mean

It does not mean Meta never fetches on request. It means that on this site, in these thirty days, its fetcher did not appear. One site is not a census.
It does not mean Meta never cites anyone. Meta runs a crawler whose documented job is citation quality. It ran here 155 times. What I can say is the ratio, not the policy behind it.
It does not mean training collection is theft. That argument exists and it is not this article. This article is about what the numbers say a company is doing, which is a smaller and more checkable claim.
52.3% is of the crawlers this instrument tracks. Meta also runs crawlers for ads and for link previews. Those are not AI crawlers and are not counted here.

What this instrument cannot see

Cached pages are invisible. When a page cache answers, PHP never runs and nothing is recorded. Every count here is a lower bound.
One site, one month, one language. This is a Japanese site about a city in the Philippines. A different subject, a different language, a different publishing rhythm would draw a different table.
Arrivals are undercounted by design. AI clients running as phone or desktop apps send no referrer, so a reader who does follow a link can land in your analytics as direct traffic. The arrival number is a floor.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which you should factor in. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. The vendor table described here is the main dashboard, and the split by purpose — training, answer indexing, live fetch — is the column worth looking at second.

You may find the same shape. You may find the opposite. The point is that this is a question with a factual answer about your own site, and it takes a week to get.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.

2,894 Requests Hit My Top Page. Nobody Ever Requested It.

roadleon — Fri, 11 Sep 2026 06:28:41 +0000

The most-read page last month, on a site I look after, was a URL that nobody ever requested.

I did not set out to find that. I was going to write about Meta.

Meta's crawlers account for 43% of all AI crawler traffic on the site I have been measuring — more than OpenAI, Anthropic and Perplexity combined. That is a strange enough fact to want an explanation, so I opened the panel that lists which pages the crawlers actually read, expecting to see what Meta was so interested in.

The top row was /tag//. Two slashes. 2,894 requests in thirty days, most of them from Meta's crawler, which is more than half of everything that crawler did on the site.

A better story than the one I had

So I loaded it. /tag// returns a 301 to /tag/, and /tag/ returns a 404. A redirect to a dead end.

For about ten minutes this was a much better article than the one I had planned. The largest AI crawler on this site was spending half of its visits hammering a URL that goes nowhere, ninety-six times a day, and the site owner had no way to know. Write that down, publish it, done.

Before writing it I wanted to know where the crawler had found the URL. That is where it fell apart.

The site's home page contains 29 links to tag pages. Every one of them is a well-formed absolute URL. None has a double slash.
The sitemap lists 276 tag URLs. None has a double slash, and /tag/ itself is not in it.
The 301 carries x-redirect-by: WordPress, so PHP is running on every one of those requests. Nothing is being cached or shortcut.

A crawler does not invent URLs out of nothing, and there was nothing here to find it in. Which left one place I had not looked.

The call was coming from inside the house

My own recording code, one line of it:

$path = sanitize_text_field( $path );

sanitize_text_field() is the reflexive way to clean a string in WordPress. It appears in every tutorial and most plugins. Its documentation lists five things it does. Four of them are what you expect: checks for invalid UTF-8, converts a bare < to an entity, strips tags, collapses whitespace.

The fifth is "Strips percent-encoded characters."

Five words, no explanation, and no warning anywhere on the page about what that means for a URL. Here is what it means for a URL:

What the crawler requested	What I recorded
`/tag/%e5%9f%b4%e8%bc%aa/`	`/tag//`
`/topics/%e3%83%8b%e3%83%a5%e3%83%bc%e3%82%b9/`	`/topics//`
`/tag/dabawenyo/`	`/tag/dabawenyo/`

The site is Japanese. Its tag slugs are Japanese, so in a URL they are percent-encoded, so my sanitiser deleted them. Every Japanese tag page on the site was being written to the database as the same six characters, and then grouped together by a GROUP BY url_path that was doing exactly what it was told.

Meta was not stuck. Meta was reading hundreds of different tag pages, perfectly normally. I was the one turning them into one row.

0

Times anyone requested the URL at the top of my most-read-pages list.
It was not a page. It was 2,894 requests to many different pages, with the distinguishing part deleted.

Who else this happens to

Everyone whose URLs are not ASCII.

A slug in Japanese, Chinese, Korean, Thai, Greek, Hebrew, Arabic, or any Cyrillic language is percent-encoded in a URL. So is a French or Portuguese or Turkish slug with an accent in it. On all of those sites, any tool that runs a path through sanitize_text_field() is storing a path with the identifying part removed — and storing it silently, in a form that still looks like a path.

The function is not broken. Stripping percent sequences is a reasonable defence for a free-text field, where an encoded payload has no business being. It is only catastrophic when the field is percent-encoded on purpose, which is to say whenever it is a URL. The documentation states the behaviour and leaves you to work out the consequence.

Why this kind of bug survives

I had been looking at that dashboard for weeks. So had the site's owner — I manage it, they own it. Neither of us noticed, and I want to be precise about why, because the reason generalises past this one function.

It never threw anything. No error, no warning, no empty result. A crash on day one would have cost me an afternoon.
The output was plausible. /tag// looks like a slightly malformed URL, which is a thing that genuinely happens. It reads as a finding, not as a defect. I nearly published it as one.
It failed by merging, not by losing. No request was dropped. The totals were all correct. Damage that shows up as a smaller number gets noticed; damage that shows up as one confident large number does not.
It only affects other people's alphabets. Every test I had written used ASCII paths, because I wrote them. On an English site this code is flawless.

That last one is the part I keep turning over. The bug was invisible to me specifically, in a way it would not have been to someone who tested in their own language first.

What to do instead

If you are storing a whole URL, WordPress has esc_url_raw() for it. If you are storing a path and you want it to survive intact, do the work explicitly rather than reaching for the general-purpose cleaner: reject invalid UTF-8, strip control characters, cap the length, and leave the percent sequences alone. That is roughly ten lines, and it is the ten lines I should have written the first time.

Then write a test with a non-ASCII path in it. Mine now has four, and they exist for exactly one reason, which is that I will not remember this in a year.

What this did and did not break

Being specific matters here, because I have published numbers from this instrument twice and I would rather say plainly which ones this touches.

Broken: which pages were read. On any non-English site, that panel has been unusable. Not subtly wrong — unusable.
Not broken: how much was read. The path is used in one other place, a check that decides whether a request counts as a content page. That check is a list of exclusions — robots.txt, sitemaps, static file extensions, login and admin endpoints — and every one of them is ASCII. A Japanese path is classified the same before and after the damage.
Not broken: anything else. Request counts, the split by crawler purpose, the verified percentages: none of them touch the path.

The fix went in on September 6. Old records cannot be recovered — the deleted bytes are gone — so the page list will be correct from that date and wrong before it, which is its own small ongoing embarrassment.

And Meta?

Still 43% of everything, still unexplained, still crawling more of this site than OpenAI, Anthropic and Perplexity put together. I will get to it. It turns out I did not have the instrument to write that article yet, and I would not have known that if the top row of my own dashboard had been slightly less strange.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which by now you have enough evidence to factor in. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. If you run it on a site whose URLs are not in English, update it — the page list before this fix was telling you nothing.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

Updated September 15, 2026: this note originally said Doctor would become a paid subscription on that date. The change has been postponed, and Doctor remains free for now.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.

984 Requests Said They Were Perplexity. None Could Prove It.

roadleon — Sun, 06 Sep 2026 01:32:13 +0000

This morning I told my own website that I was ClaudeBot.

It took one line of curl and a user agent string copied out of Anthropic's own documentation. Three requests to one article page. Then I opened my dashboard: Anthropic, ClaudeBot, training — 1,698 requests had become 1,701, last seen at 09:22.

None of them was ClaudeBot. They came from a laptop in Japan, over ordinary home broadband, from an address that appears on no crawler list anywhere.

The instrument was not broken. It was doing exactly what it is built to do, which is to write down what arrived and what it called itself. A user agent is a claim. Anyone can type it. The whole question is how much of what you are measuring is a claim you can check, and the honest answer is worse than I expected — including, as it turned out, in ways that were my own fault.

The column nobody looks at

There is a way to check, for some of them. A vendor can publish the IP addresses its crawlers run from. If a request claiming to be GPTBot arrives from an address on OpenAI's published list, it is GPTBot. If it arrives from my laptop, it is not. Some vendors publish such a list. Some publish nothing at all. So my dashboard has a column called Verified, and it is the least glamorous number on the page.

Here is one site, davaonline.net, over the thirty days ending September 4, 2026: 13,491 crawler requests, of which 10,390 landed on pages with actual writing on them. Sorted by volume, with the share of each that could be confirmed:

Crawler (requests)	Verified
meta-externalagent (5,728)	no method published
ClaudeBot (1,698)	no method published
Applebot (1,479)	98%
Perplexity-User (984)	0%
ChatGPT-User (887)	6%
Bytespider (767)	no method published
GPTBot (598)	81%
CCBot (440)	no method published
Amazonbot (342)	no method published
OAI-SearchBot (242)	60%
meta-webindexer (110)	no method published
PerplexityBot (103)	0%
Claude-User (69)	no method published
GoogleOther (25)	100%
Claude-SearchBot (19)	no method published

1 in 6

Requests that arrived with proof of who sent them.
Roughly 2,200 confirmed out of 13,491. Everything else is a string I wrote down because it was handed to me.

That single number hides three completely different failures, and separating them is the only useful thing I can do here.

All 13,491 requests, by why verification did or did not happen
No verification method published by the vendor	6,947	51%
Published, but my bundled copy was out of date	2,226	16%
Published, I had it, and the address did not match	4,318	32%

Failure one: nothing to check against

Meta, ByteDance and Amazon publish no IP ranges and no reverse-DNS scheme for the crawlers in that table. There is no list to compare an address to. Not a hard case, not an unlucky one — there is simply no mechanism, so verification is not slow or partial, it is impossible.

That accounts for 6,947 of the 13,491 requests. More than half the traffic in this table comes from vendors who have decided that site owners do not need a way to tell whether it is really them. The largest single crawler on the site, meta-externalagent at 5,728 requests, is in this group.

Amazon is a near miss worth naming: it does publish a page of addresses for Amazonbot, but as a web page whose contents only appear after JavaScript runs. A list a machine cannot read is a list that does not exist for this purpose.

And a small trap for anyone else doing this: ByteDance's obvious candidate URL returns HTTP 200. The body is a 404 page. If you check for a published list by looking at status codes, you will conclude they publish one.

Failure two: the list existed and I was out of date

Look at the Anthropic rows and the CCBot row in that table. They say no method published. That was true when the plugin's bundled snapshot of vendor ranges was taken, on July 20. It is not true now.

Anthropic began publishing a range list on August 18, replacing an explicit statement that it did not. Common Crawl published one on August 11. Both landed inside the thirty days measured above, and my instrument went on reporting the old world, because a design decision I am otherwise happy with — the plugin never talks to the network, so vendor ranges ship as a snapshot — means the snapshot goes stale silently.

That is 2,226 requests, sixteen percent of the total, reported as unverifiable when they were merely unverified by me. I fixed it on September 4; the count of crawlers this instrument can verify went from eleven to fifteen. The fix also caught a Google crawler that had been listed as verifiable for months while its ranges were never actually bundled.

I would rather not have found this while writing an article about verification. But it is the more useful half of the lesson: whether a crawler is verifiable is not a property of the crawler. It is a property of how fresh your copy of somebody else's list is, and it decays without telling you.

Failure three: the list existed, I had it, and it did not match

This is the one I cannot explain away.

Perplexity publishes IP ranges. I had them. Over thirty days, 984 requests arrived calling themselves Perplexity-User and 103 calling themselves PerplexityBot. The share of them coming from an address on Perplexity's own published list was not low.

0

Requests claiming to be Perplexity that came from Perplexity's published addresses.
Out of 1,087, over thirty days, checked against a list I had.

OpenAI publishes too, and keeps it current — its ChatGPT-User list was refreshed on August 14. Six percent of the 887 requests claiming to be ChatGPT-User came from it.

There are at least four explanations and I can rule out none of them:

The requests are not from those vendors. Forging a user agent costs one line of curl, as established at the top of this article. Somebody scraping the web has an obvious incentive to wear a name that site owners have been told to allow.
The published list is incomplete. Perplexity's user-fetch list contains four address ranges and was last updated in October 2025. Its crawler list contains eight, last touched in February 2025. A company serving live questions at scale from four ranges, unchanged for eleven months, is at least worth a raised eyebrow.
Real fetches run outside the published infrastructure. User-initiated fetching often runs from different machines than bulk crawling, and nothing obliges a vendor to put those in the same file.
My own verification did not run. Addresses are checked by an hourly background job that processes a bounded batch and discards anything older than a day. A request that was never checked and a request that was checked and failed both appear in that column as unverified. My dashboard does not currently tell them apart, which on reflection is a defect and not a detail.

The last one deserves the discomfort it causes. I can report the share of traffic I confirmed. I cannot report the share I disproved. Those are different quantities, and an instrument that reports only the first should say so out loud rather than let a low percentage imply an accusation.

What you can still conclude

Less than you would like, but not nothing, and the shape of it is stable.

The verified count is a floor on real crawler activity and never a ceiling. Every failure mode above pushes the same way: it turns real visits into unconfirmed ones, never the reverse.
The unverified count is not an accusation. Most of it is vendors who published nothing, which is a policy choice on their side, not evidence about the request.
The gap is worth watching per vendor rather than in aggregate. A vendor that publishes ranges and still matches nothing is telling you something. A vendor that publishes nothing is telling you only that it published nothing.
None of this changes the finding from last week's piece: this site was read roughly ten thousand times and visited twice. Whether every reader was who it claimed to be does not make the reading less real.

What this instrument cannot see

One site, one month. A different site, in a different language, on different hosting, would draw a different table. This is an instrument reading, not a study.
Cached pages are invisible. When a page cache answers, PHP never runs and nothing is recorded. Every count here is a lower bound.
The percentages are rounded. The dashboard reports whole percentages, so the "roughly 2,200" above is derived from rounded figures and should be read as approximate.
Verification confirms a vendor, not a crawler. Anthropic publishes one combined list for all three of its crawlers, so a confirmed hit means the request came from Anthropic, not which of its crawlers it was.

The uncomfortable part is not that some crawlers cannot be verified. It is that the line between the ones that can and the ones that cannot moved twice last month, in my favour both times, and my instrument did not notice until I went looking for something to write about. If you are running any measurement of this kind, the question is not whether your numbers are right today. It is when you last checked whether the world they describe still exists.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which you should factor into everything above — including the parts where I am the one who got it wrong. Every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded. The Verified column described here is on the main dashboard, and vendors with no published method are marked as such rather than folded into a single number.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

Updated September 15, 2026: this note originally said Doctor would become a paid subscription on that date. The change has been postponed, and Doctor remains free for now.

See a sample diagnosis · AILYS Doctor

Illustration generated with AI and selected by the author.

Read 10,202 times. Visited twice.

roadleon — Thu, 03 Sep 2026 01:02:41 +0000

The site looked dead. Thirty days of analytics, and the traffic arriving from AI came to two visits — one from ChatGPT, one from DuckDuckGo. Two. From a dashboard like that you would conclude that no AI has ever noticed the site exists.

That conclusion is wrong by about ten thousand.

Over the same thirty days, AI crawlers requested pages from that site 13,306 times, and 10,202 of those requests were for pages with actual writing on them. The two instruments are not contradicting each other. They are answering different questions, and almost everybody is only asking the first one.

The site is davaonline.net, a WordPress site running a plugin that does the opposite of normal analytics. It ignores human visitors completely and records only AI crawlers: which ones arrived, what they fetched, and whether the thing they fetched was a page or just a file.

Here is the whole month.

The numbers

Thirty days	Count
AI crawler requests	13,306
Requests that hit a content page	10,202
Requests made to answer a live question	1,902
People who arrived by following an AI answer	2

Read it downward. The story is in the descent.

Visiting is not reading

The gap between the first two numbers — 3,104 requests — is not rounding error. It is robots.txt, sitemap.xml, and a tail of files that are not your writing. A crawler that fetches robots.txt and leaves has visited your site and read nothing. Counting that as AI is reading me is counting a knock at the door as a conversation.

Which is why the instrument sorts every request by purpose instead of reporting one number:

Purpose	Requests (vs previous 30 days)
Training — collected to train a model	9,514 (+1,175%)
AI search — indexed for an answer engine	1,890 (+432%)
User fetch — retrieved to answer a live question	1,902 (+3,558%)

These are not interchangeable. Training collection is a bet on the future: your words may end up inside a model that ships next year, or they may not. Search indexing is a bet on retrieval: you become findable inside somebody else's answer engine. User fetch is neither a bet nor a future. It means a person asked a question a few seconds ago, and a machine went to your page to answer them.

Reading is not traffic

1,902 times in thirty days, this site was contacted by a crawler that retrieves on behalf of somebody asking a question right now. Two people arrived.

One came from ChatGPT. One came from DuckDuckGo. That is the complete AI referral traffic for the month.

A note on the arithmetic: the 1,902 and the 10,202 slice the same pile of requests in different directions — one by what the crawler is for, the other by what it asked for. Neither is contained in the other, and the 1,902 includes some share of robots.txt fetches rather than pages. Read at its most conservative, the ratio does not soften.

This gap is not a funnel to optimize. It is the shape of the thing. The page was retrieved, read, and used as the source of an answer that appeared somewhere else entirely. The answer was delivered. The visit never happened. Meanwhile those fetches grew 3,558% over the previous thirty days — two arrivals is what that growth looks like from the traffic side.

A conventional analytics dashboard reports this month as though almost nothing occurred. It is not wrong about visits. It is answering a question that has quietly stopped being the important one.

What this instrument cannot see

Before any of that means anything, here is what these numbers are not.

Two is a floor, not a count. Arrivals are identified by referrer. AI clients running as desktop or phone apps frequently send none at all, and those visits land in your analytics as direct traffic, indistinguishable from somebody typing your URL. The true figure is higher than two. Nobody can tell you how much higher.
Cached pages are invisible. When a page cache answers a request, PHP never runs, so nothing is recorded. Every number here is a lower bound.
Not every crawler can be verified. OpenAI and Perplexity publish IP ranges, so their requests can be confirmed. Meta and ByteDance publish nothing, so theirs are matched on the user-agent string alone — and user-agent strings can be forged. Anthropic sits in between: it began publishing a range list on August 18, partway through the month measured here, and the plugin was still carrying an older snapshot that did not contain it — so its requests are counted here as unverified as well. The dashboard marks which is which. This article does not separate them.
One site, one month. Different subject matter, a different language, a different publishing rhythm would all produce different numbers. This is an instrument reading, not a study.

None of those caveats move the conclusion, because they all push the same direction. Correcting for every one of them would make the reading number larger and leave the arrival number roughly where it is.

How to measure your own site

The instrument is a WordPress plugin called AILYS Lens, and it is free. I build it, which you should factor into everything above. It is also built so that measuring costs you nothing else: every calculation runs on your own server, nothing is transmitted anywhere, and human visitors are never recorded — only AI crawlers are. Requests that are not crawlers pass through in under a millisecond, and no CSS or JavaScript is loaded on your front end.

Install it, leave it alone for a week, and you will know whether anything is reading you. It is a question with a factual answer, and until recently there was no instrument that would give it to you.

The uncomfortable part of this month is not the two. It is that a site can be read ten thousand times and still produce a dashboard that looks like failure — and that its owner has no ordinary way to tell the difference between being ignored and being read in silence. That is a strange position to publish from. It seems worth knowing which one you are in.

A note on AILYS Doctor

AILYS Lens is free permanently. That is a design commitment rather than a pricing stage — every feature runs locally on your server, so there is nothing for a paid tier to unlock.

Updated September 15, 2026: this note originally said Doctor would become a paid subscription on that date. The change has been postponed, and Doctor remains free for now.

See a sample diagnosis · AILYS Doctor

Correction, September 4, 2026: this article originally said that Anthropic publishes no IP ranges. Anthropic began publishing them on August 18, 2026, before this article went up. The passage above has been rewritten. The counts themselves are unchanged — the requests really were recorded unverified — but the reason was a stale snapshot on my side, not an absent list on theirs.