Since 15 May a small Cloudflare Worker of mine asks the Common Crawl index, once a day, whether about a dozen URLs of my site are in the current crawl. It is a cheap, boring check: one request to get the list of collections, then one CDX lookup per URL, and a row in a table with the answer for each one.
The answers turned out to be less interesting than the failures.
The numbers
173 snapshots have per-URL results, 1,889 lookups in total. 605 of them did not get an answer at all:
- 344 x HTTP 503
- 103 x HTTP 504
- 80 x HTTP 521
- 52 x aborted by my own timeout
- 18 x HTTP 502, 7 x HTTP 500, 1 x HTTP 520
That is 32 percent of all lookups. In 11 of the 173 snapshots every single lookup failed.
Split by time of day it gets more lopsided. The scheduled run fires at 04:00 UTC. Lookups made between 03:00 and 06:00 UTC failed 491 times out of 1,339, about 37 percent. Lookups at any other hour failed 114 times out of 550, about 21 percent.
I would not read too much into that split. Most of the off-hour snapshots are manual reruns, and I usually start a rerun after I have seen that the morning failed, which means later, when things may already be better. So there is a selection effect baked in. But the direction has held over five months.
October is the worst month so far: 79 failed lookups out of 132.
The last three mornings
On 10 and 11 October the run did not even get to the URL lookups. The request for the list of collections came back with 521, and that call had no retry, so the whole stage failed and wrote nothing. The URL lookups had retries since August, the collection list did not.
So I added three attempts with 10 and 20 second waits to that call. On 12 October it went through. Then all nine URL lookups came back 521.
Both earlier days I reran the stage in the afternoon. On 10 October 12 of 12 lookups answered. On 11 October 10 of 12 did. Same code, same URLs, a few hours later.
While writing this, at about 06:10 UTC, a plain curl got a 200 for the collection list and a 504 for one URL lookup.
What I take from it
Retries with waits of a few seconds help with a request that drops. They do nothing for an index that is down for hours. If the upstream has a bad window, the fix is a second attempt hours later, not a third attempt 20 seconds later. That is the next change for this stage.
The other thing I am glad I did early: a failed lookup is stored as an error, never as "not included". The inclusion rate is computed only over the URLs that actually got an answer, and when too few answered it stays empty instead of showing a number. On 11 October that is the difference between "7 of 10 answered URLs are in the crawl" and "7 of 12 URLs are in the crawl". Only one of those is true. If errors were folded into "not included", the October numbers would look like my site was falling out of Common Crawl, when really the index was just not picking up the phone.
If you poll the Common Crawl index on a schedule, it is worth looking at how many of your lookups actually came back before you look at what they said.
The run snapshots behind these numbers are public on the research dashboard of my site.
Top comments (0)