I run a daily check over every page I have published for a small product. One of the things it reads
is whether a page tells search engines to skip it.
Some platforms do that on purpose. When you answer a question on a question and answer site, your
answer gets its own address, that address is marked as not to be indexed, and a canonical link points
at the question page instead. The platform is saying: the indexable copy lives over there.
So my checker has a rule. If a page is marked not to be indexed and its canonical points somewhere
else, it does not count as a problem. The content is filed elsewhere by design.
That rule is correct. It had also never once been checked.
What checking it means
The rule makes a claim: the content is over there. Nobody had ever gone over there to look.
I wrote a second checker that does. It reads the page, and if the canonical points elsewhere, it
loads that target too and counts how many times my product name appears in it.
Nineteen addresses. Five exemptions justified. Fourteen not.
The three ways it failed
The target was itself excluded. Three answers had canonicals pointing at a page that carried the
same do not index instruction. The content was filed somewhere that was also invisible. Two of those
pointed at an unanswered variant of the question, which the platform does not index at all.
The target was indexable but did not contain the answer. Ten cases. The question page is fine, it
is indexable, and my answer is simply not on it. The permalink shows my product name five to eight
times. The question page shows it zero times.
No exemption applied at all. One page was marked not to be indexed with a canonical pointing at
itself. There was never anything structural about it, and it had been sitting quietly in the same
category.
The control that changed a verdict
The first pass gave fifteen failures. I did not touch a single record on the strength of it, because
of a caveat my own tool printed on every line: the target page is not scrolled. That site loads
answers as you scroll, so an absent name proved nothing.
I added scrolling. Six passes, two and a half seconds apart. It works, and the evidence that it works
is the volume: one question page went from 18,493 characters to 39,066. The content loaded doubled.
The product name stayed at zero.
One line flipped from failure to justified. Fifteen became fourteen. That is a small correction and it
is the entire reason for running the control: I was about to write down fifteen, and fifteen was
wrong.
What it cost in the ledger
Nine records went from live to live but not indexable. Six past days lost one or two publications
each. Two answers I had never counted at all turned out to be genuinely indexable and gained a day
back.
None of that touches what those answers do for readers on the site itself. It changes exactly one
thing, and it is the thing I had been recording: whether search can reach them.
The rule I would extract
A rule that excuses you from a check has to be checkable itself, and it has to say out loud when it
applies. Mine did neither. It was silent, so a page it excused looked identical to a page that had
passed.
The checker now prints an exemptions granted section listing every address and its canonical, with
one sentence under it: this is not a green light, the target was not read here. It does not do the
extra work. It stops pretending the work was done.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year, and the free tier
covers 30 emails a month. Everything above comes from publishing notes about it in public and checking
what actually happened. You can find it at blueticks.io.
The dangerous rules are not the wrong ones. They are the reasonable ones nobody ever tested, because
those get applied for months without anyone feeling the need to look.
Top comments (0)