I have been running a small measurement rig for two weeks: a set of scripts that load our pages in a fresh anonymous browser context and record what is actually there. Directory listings, articles, search positions. It has been genuinely useful and it has been wrong in one specific way, six separate times, and the six are the same mistake.
Every time, a script failed to read something, and I wrote down the failure as a fact about the world. The last two happened while I was drafting this post, which is the part I would keep if I could only keep one.
One: I turned four observations into a property
Two directories were serving an anti-bot challenge instead of their content. I checked, got the challenge, checked again the next day, got the challenge, and after four identical observations I recorded it in my notes as an established property of those sites: they block automated readers.
On the fifth check, both loaded perfectly. Nothing had changed on my side.
The observations were all correct. Every single one of them. What was wrong was the sentence I wrote underneath them, which promoted four samples of an intermittent thing into a permanent characteristic. Four is not a small number when you are counting real events, and it is a very small number when you are trying to establish that something never happens.
Two: a listing was live for six days while I counted it as pending
We submitted to a software directory. Two pages appeared under our name. My checker could not confirm they were public, because the site was serving that same challenge, so I recorded them honestly as unconfirmed and moved on.
Six days later, with the challenge gone, both pages loaded on the first try. Nine hundred words each, indexable, and almost certainly live the entire time.
This is the expensive version of the mistake. Nothing about my notes was a lie: I had written I cannot prove these are public, which was true. But the ledger downstream treated unconfirmed as not live, so for six days our count of live pages was wrong, and the decision about whether that directory had been worth the effort was made on the wrong number.
A measurement you cannot take is not neutral. It manufactures a durable false negative, and the falseness sits in a column that looks just as confident as the true ones.
Three: I let a tool's limits define reality
A directory sat in my ideas file for a week marked unreadable, tested with a command line HTTP client. The note said: cannot be read.
The site renders client side. A command line fetch gets a shell with no content in it. A real browser gets a full page, a signup form, and a working submission flow. The site was never unreadable. My tool could not read it, and I had written down the shorter sentence.
That shorter sentence is the whole problem. Cannot be read has no subject. Once the subject falls out, the limitation moves from the tool to the world, and nobody who reads the note later will think to question it.
Four: eleven seconds
I ran a script across every listing we own, reading the rel attribute on outbound links. One domain came back with no link at all. An hour later, on a re-run, the same domain came back with a followed link.
That kind of disagreement is easy to average away. I ran it three more times: link, link, link. The first reading was a false negative because the page takes longer than eleven seconds to finish rendering, and my script measured an empty document and reported it as a fact.
The number that changed was not important. What was important is what it implied about all the other rows: a negative verdict and a slow page produce identical output. So I re-ran every negative in the table. They held, this time. Had I not chased the one that flipped, I would have published a table with a wrong row in it and no way of knowing which row.
Five and six: I did it twice more while writing this
The morning I finished the draft above, I went looking for a way to add a missing website link to one of our directory listings. The listing is 896 words about our product and my scripts had recorded, twice, that it contained no outbound link to us at all.
The edit form showed the website field already filled in, correctly, and presumably filled in from the day the listing was created.
A field with a value and a page with no link. One of those had to be wrong, and it was my reading. That directory puts the outbound link on a sub-page, not on the main listing. A large, well established listing on the same site behaves identically, so it is how the site works. My script had read exactly one page per listing, the address in my ledger, and had never followed a sub-page in its life.
So I rebuilt the sub-page search and ran it across every remaining no-link listing. A second one fell, for a completely different reason. That directory does link to us: an anchor whose visible text is our domain, whose href points at a redirect script on their own domain, carrying rel="nofollow". My script tested whether the href contained our domain. This one does not contain it anywhere.
Two of the three verdicts I re-examined were links hidden by a site convention. One on a sub-page, one behind a redirect. That is not bad luck. It is what directories do: they route outbound traffic so they can count it, and every routing scheme is invisible to a naive href match.
And then, chasing the second one, I produced a false positive to match. My sub-page crawler reported finding our link on the redirect URL. It had followed the redirect, landed on our own website, and was cheerfully counting our own internal links. A measurement taken after following a link no longer measures the link. It measures the destination.
The rule I now use
A positive result from a crawler is evidence. A negative result is a hypothesis.
I wrote that sentence in a draft of this post the evening before both of the failures above. Writing a rule down and applying it are two separate actions, and the second does not follow from the first on its own.
If the script finds a link, there is a link. That direction is safe, because the failure modes of the tool do not invent content. If the script finds nothing, one of two things is true: there is nothing there, or the tool did not manage to look. Those two are indistinguishable in the output, and only one of them is about the site.
Three practical consequences, all of which I now do:
Negative verdicts get a second pass, separately. Not a re-run of the whole job, a targeted re-read of only the negatives. It is cheap because negatives are usually a minority, and it is the only thing that separates the two cases above.
Notes name the instrument, always. Not cannot be read but the command line client gets an empty shell here. The first sentence is about the site and is wrong. The second is about my tool and is true, and it invites the obvious next question, which is whether a different tool would do better.
When an obstacle disappears, re-walk everything it ever hid. This is the one I keep having to relearn. The day the challenges stopped, I should have immediately re-checked every row that had ever been marked unconfirmed. I did not, and the six day case is what that omission cost.
Why this is worth the effort at all
Because the alternative is a spreadsheet full of confident numbers where an unknown fraction of the zeros mean no and the rest mean I did not manage to ask. You cannot tell them apart later. There is no audit that recovers it, because the two look the same in every column you kept.
The fix is not better scripts. It is a second column recording how the answer was obtained, and a habit of distrusting one particular direction of result.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year and there is a free tier. The measurement rig described above exists to track how it gets distributed, and it is written up as it happens, including the parts where it measured badly. You can find it at blueticks.io.
If you run something similar, the cheapest improvement available to you today is to re-run only your negatives and see how many of them survive.
Top comments (0)