A reader asked a question my log could not answer. The fix was one column.

I have been publishing numbers from a crawler log I added to my own site after a compromise: one line per crawler request — time, crawler, status served, path, IP. Last week I used it to show that a flood of requests for dead URLs was not the 410 failing, because almost none of those URLs were repeats.

A reader, jonas_bra, replied with the sentence that makes a measurement honest:

first seen in our logs isn't the same as newly created

A low repeat rate suggests Bing isn't repeatedly revisiting the same dead pages, but it doesn't tell you whether those URLs are still being discovered through links elsewhere.

He is right, and I had not noticed. My log can tell you a path is new to the log. It cannot tell you whether the URL was minted during the attack in September and is only now being reached, or whether something out there is still handing Bing fresh ones today. Those are different problems — one is a backlog that ends, the other is a live source that does not — and I had been reporting on the first as if I had ruled out the second.

What the existing data can settle

Two of the three questions turned out to be answerable with what I already had.

Is my own site still emitting them? I fetched fourteen pages — homepage, every section hub, the sitemap, robots.txt, a few tools — and matched every href against the cloaker's URL shape. Zero. The generator and the files it lived in are gone, so my site is not a source.

Is the backlog plausibly finite? The malware handed out about 291 invented URLs per page load, to anything that asked for a page, for six days. Bing currently fetches roughly 3,500 of them a day. Six days of being served 291 URLs per request, at anything like that rate, mints URLs on the order of a million. Against that, the 80,394 distinct ones my log has seen in 24 days is a fraction of a backlog, not evidence of a live source.

Neither of those is proof. Together they make "finite backlog" the better-supported story — but "better-supported" is not "measured", and the difference matters.

The column I did not have

What would actually settle it is the referer. A crawler that followed a link usually says where it came from. My log did not record it, because when I wrote the thing I was asking "who is crawling and what are they being served", and that question does not need a referer.

So the log has a sixth column now:

$ref = (string) ($_SERVER['HTTP_REFERER'] ?? '');
if (strlen($ref) > 200) { $ref = substr($ref, 0, 200) . '...'; }

And the summary counts how many hits carried one.

The important part is what that column can and cannot prove, and it is worth being explicit because the asymmetry is the whole value:

• A non-empty referer is proof of continuing discovery. If something links to /stimulated/a0d3o2b2o25378994, that link exists and I can go and look at it.
• An empty referer proves nothing. Bing frequently sends none. A column of blanks is consistent with a finite backlog and with a live source that Bing reached without passing a referer along.

One direction is evidence; the other is silence. Writing that down in the code comment matters as much as writing the code, because in three weeks I will be reading a column of blanks and the temptation will be to call it an answer.

The habit underneath

This is the third time in two months that the fault in one of my instruments was not a bug but a missing dimension:

• A sitemap check counted URLs and could not tell "empty" from "unreachable" — it needed a second outcome, not a better number.
• A link check printed how many articles answered as though that were how many exist — it needed a denominator.
• This log could tell new-to-me from seen-before, and could not tell new-to-me from newly-created — it needed the referer.

In each case the data looked complete, every number in it was correct, and the conclusion was wrong anyway. You cannot find that by staring at your own output. It took a stranger reading a post to find this one, which is the strongest argument I know for publishing the numbers rather than just acting on them.

Three small things, if you keep logs like this

• Write down what each column can falsify, not just what it records. "Non-empty proves X, empty proves nothing" belongs next to the field, in the code.
• Keep the raw line, not just the aggregate. Every one of these gaps was recoverable only because the per-request lines were still on disk; a daily counter would have thrown the evidence away. If you are pulling a value out of a URL by hand while debugging, a URL parser saves you from the usual off-by-one in the path split.
• Publish the numbers. The reader who finds your missing column is not reading your code — they are reading your conclusion, and they are the only person in the loop who does not already believe it. For what it is worth, my posts are about a thousand words; the useful corrections have all come from people replying to the specific figure, not to the argument.

Current state, so the next correction has something to bite on: 24 days of log, 80,394 distinct dead URLs, 3,000–4,000 new a day and not declining, repeat rate up from about 1% to about 5%, Googlebot has asked for none of them, and the referer column starts filling tomorrow.