Correction added a few hours after publishing, 2 August.
The measurement in this post used a site: query to ask which of my articles a search engine
knows about. That operator under-reports, and I can now prove it on my own pages: one article
had been ranking second on a real content query since 31 July, and the same site: query
still listed it as absent on 1 August. I counted a page that was ranking as not indexed, for
more than twenty-four hours, and this post is built on that count.
I remeasured with something that has a positive control: query each page on its own exact
title. A title query is close to navigational, so an indexed page should come back first, and
four of my pages do come back first, which is what makes a negative result readable at all.
The rule I should have followed from the start is that a measurement with no positive control
does not measure anything.
The conclusion below survives, on better evidence. Four of five are found, the only missing
one is the youngest, and the ordering by age is exact with no exception. The delay is one to
two days rather than the three I wrote. So the argument holds and the specific numbers in the
next section do not. I am leaving the original text in place rather than quietly editing it,
because the mistake is the more useful half of the story.
I run a small blog and I publish on a few platforms. Last night I finally measured whether any of it is findable, and the first number I got was useless in a way worth writing down, because I nearly acted on it.
Here is the number: two of my five articles on one platform come back in search. Forty per cent. My first reading was that something was wrong with three of them, and my first instinct was to go and look at what those three had in common. Thin content, a bad title, some quarantine flag on new accounts. I have seen all three happen.
Then I sorted the five by publication date instead of looking at them as a set.
| Published | Age | Found |
|---|---|---|
| Article 1 | 3 days | yes |
| Article 2 | 3 days | yes |
| Article 3 | 2 days | no |
| Article 4 | 2 days | no |
| Article 5 | 1 day | no |
There is no overlap. The two oldest are found, the three newest are not, and the boundary sits cleanly between three days and two. That is not a forty per cent success rate. That is a delay, and the difference matters because the two readings lead to opposite actions. A rate means go and fix the three failures. A delay means leave them alone and wait.
The right question turned out not to be "how many are found" but "how many of the ones old enough to be found are found". On that measure I am at two out of two.
The mistake I almost published instead
I also have four articles on a self-hosted blog subdomain, and those return nothing at all. Zero, on any query, including quoted phrases lifted straight from the titles. They are well past the three day boundary, so the delay explanation does not cover them.
So I fetched them and looked at the head. All four came back carrying this:
<meta name="robots" content="noindex,nofollow">
Case closed, obviously. Except I looked at the response size before writing it up, and each response was about 5 700 bytes. A real article of mine is around 120 000. What I had fetched was a bot challenge page from the CDN in front of the blog, and the challenge page carries its own noindex, which is entirely correct behaviour on its part.
I would have published a true statement about the wrong object. The meta tag was real. It just did not belong to my article.
Refetched through a real browser that clears the challenge, reading every robots meta rather than the first one, all four pages are clean: no robots meta at all, and a self referencing canonical. Nothing on the page is blocking anything.
Two habits came out of this and I would keep both:
- Check the response size before you read the response. A page that is twenty times too small is not your page. This is cheap, it is one number, and it would have caught the error before I formed an opinion.
-
Query all matching elements, not the first one.
querySelectorreturning a robots tag does not mean it is the robots tag. I have been bitten by a platform that emits two of them, where the first carries nothing interesting and the second carries the decision.
The control query, which is the part people skip
Three queries in a row returned zero results for the blog. Before concluding the blog was invisible I ran a fourth one, in exactly the same form, for an article I already knew was indexed.
It came back first. So quoted phrase queries do work on that engine, the method is sound, and the three zeros are real rather than an artifact of how I was asking.
This costs one extra query and it is the difference between a measurement and a guess. A negative result only means something if you can show the same instrument returns a positive one. Without that fourth query I had three zeros and no way to tell whether I had found a problem with my blog or a problem with my search syntax.
What was actually wrong
Nothing on the pages. The blog subdomain is four days old, it has no inbound links from anywhere, and its sitemap sits behind the same challenge as everything else.
Meanwhile the platform where two of five are indexed is a large, heavily crawled domain, so a new URL there gets visited within days without me doing anything. A brand new subdomain with no links pointing at it has none of that. Crawlers cannot visit what nothing points to, and my sitemap, which is the one discovery path that does not need links, is not reachable either.
I want to be careful about that last part, because it is the piece I cannot verify. Search crawlers are usually allowlisted by these systems, and the allowlist works on network address rather than on the name a client gives. So I cannot test it by claiming to be a crawler, and a failed test proves nothing. I am leaving it as a plausible contributor and not a conclusion.
What is established is enough to decide: the pages are fine, and nobody has a route to them. The fix is not another article. Publishing a fifth piece into a folder nobody visits produces exactly what the first four produced.
What I would tell my earlier self
Judge indexation in age cohorts, never as a lump. A ratio computed across mixed ages hides the only variable that explains it, and it will make you rewrite articles that had nothing wrong with them.
Do not compare two platforms until the articles are the same age. I had this exact comparison scheduled for tomorrow and I have pushed it back by a day, because my newest articles on the second platform are one day old, which is inside the window where the first platform also shows nothing. Running it tomorrow would have produced a confident and completely wrong verdict against the second platform.
And declare a failure only past the delay. Three of my five articles are not failures, they are young. The four on the blog are a real failure, and the reason I can say that with any confidence is that they are old enough for the word to mean something.
Disclosure: I build BlueTicks for Gmail, a browser extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year and there is a free tier. The blog above is its blog, which is why I was looking at any of this in the first place.
Top comments (0)