My 410 was not failing. Bing was never asking twice.

Two weeks ago I switched a batch of dead URLs from 404 to 410 Gone and watched the daily count of them go up. I wrote a correction saying the 410 had not worked.

That correction was also wrong, and the number that shows why took one command.

The thing I had not counted

My crawler log has one line per crawler request: time, crawler, status served, path, IP. I had been counting requests per day. What I had never counted was how many of those paths the log had seen before:

day         distinct   new     repeated
29 Sep        972      972        0
30 Sep      3,912    3,909        3
 1 Oct      2,484    2,482        2
 5 Oct      4,055    4,019       36
 8 Oct      4,662    4,609       53
10 Oct      5,034    4,932      102
12 Oct      2,318    2,285       33

Over 97% of what Bing asks for every day is a URL it has never asked for before. In fifteen days: 50,844 distinct dead URLs, and almost no repeats.

So the 410 was doing exactly what a 410 does. It tells a crawler this one is gone, do not come back, and Bing does not come back — the repeat column is 0.7% to 2%, and those few are probably retries in flight. What a 410 cannot do is shrink a queue of URLs the crawler has not reached yet. I had been reading a backlog being drained as a fix that was not working.

The backlog exists because, during a compromise in September, a cloaker generated roughly 291 invented URLs per page load and handed them to any crawler that asked for a page, for six days. Nobody knows how many it minted. What I know is that Bing found at least 50,844 of them, and is still working through the list at about 4,000 a day.

The same log, read as a whole

Fifteen days, every crawler, by status served:

crawler	200	410	other
googlebot	513	0	3× 404
bingbot	2,347	51,326	51× 404, 2× 302
bytespider	4,141	0	—
yandexbot	228	38	3× 404
claudebot	209	0	—

Three things in that table I would not have guessed.

Googlebot has never once asked for a spam URL. Not one 410 in fifteen days. Whatever Google discovered during the compromise, it has already dropped. Its crawl of this site is now 100% real pages — and it is small: 513 requests, 291 distinct URLs, against a sitemap of 6,215.

Bing's crawl is 95.5% dead URLs. It is spending twenty-two requests on the attacker's imagination for every one it spends on a real page of mine.

ByteDance's crawler fetches eight times more real pages than Google does. 4,141 against 513. It is not going to send me anything, but it is the single heaviest consumer of real pages on the site.

The fix I did not ship

The obvious next move is to stop Bing asking at all. The spam URLs all end in digits, so ten lines of robots.txt would do it:

Disallow: /*0$
Disallow: /*1$
…

I checked that against my own sitemap before writing it, and it would have blocked 105 real pages — /tools/rot13, /tools/image-to-base64, and their four language copies each. A rule that costs real pages to save a crawler's time is a bad trade.

It is also the wrong goal. A URL that is disallowed in robots.txt is a URL the crawler cannot fetch, so it never sees the 410, so it never learns the URL is gone. Blocking would freeze 50,000 dead URLs in Bing's index instead of removing them. The crawl is not costing me anything — 4,000 requests a day is nothing, and the pages they hit do not exist, so they cost a 410 and a database-free code path.

So: no action. The right answer is the one already deployed, and the rest is patience.

What I would tell past me

• Count distinct, not total. "4,000 requests a day, not falling" and "4,000 new URLs a day" lead to opposite conclusions, and one sort -u separates them. If you are eyeballing a list by hand, sorting it first makes repeats visible immediately — the shape of the data is the diagnosis.
• Write the rule, then test it against your real URLs before shipping it. My ten robots lines looked harmless. Try the pattern against a sample of paths that must keep working; mine failed on a tool called rot13.
• A correction can be wrong too. I corrected a claim with a bigger sample and still had the wrong model of what was happening. The fix was not more data — it was one more column.