I published a post here at 01:25. At 01:30 the page was fully readable by anyone, logged in or not,
and it carried a noindex and a nofollow in its head. At 03:40 those were gone.
Nothing was removed. Nothing was flagged to me. There was no notice in my dashboard, and if I had
not been reading the raw markup of my own pages that night I would never have known it happened.
How I found it, and what my own tool got wrong
I run a check after every publication that opens the page in a clean browser context with no session
and asks one question: can an anonymous visitor read this. It answered yes, which was true.
I had been treating that yes as "this page is worth something now". Those are different claims, and
nothing in my setup had ever forced me to separate them. The page was legible. It was also, for that
window, invisible to search engines.
There is a small technical trap in the middle of this. The page emits three robots meta tags,
and the first one says nothing about indexing. A check that reads one tag and stops finds a
perfectly reassuring answer. You have to read all of them.
The control that mattered
My first instinct was that the account was in trouble. That instinct is worth almost nothing on its
own, so before writing anything down I read the same markup on five other posts of mine: one from
the previous day, and the oldest one on the account.
All five carried only the ordinary tag. No noindex anywhere.
That single comparison is what turned a story into a fact. The restriction applied to one post, not
to the account and not to the site. Without it I would have recorded something dramatic and wrong.
I nearly made exactly that mistake an hour later on a different platform. A sweep flagged fifteen of
my pages as noindex, including every answer I have written on a large question site. Alarming, and
false: I had compared a question page to answer permalinks, which are different page types. A
stranger's answer permalink carries the identical tags, with a canonical pointing back at the
question. That is how the site is built. Nothing was done to me.
Same night, same error shape, twice: a control is only a control if it is the same kind of thing
as the thing it controls.
What the two hours probably mean, stated as weakly as the evidence allows
I do not know the rule. I can say what the shape of the observation rules out.
It was not a removal: the post stayed up and reachable the entire time. It was not a human moderation
queue in any obvious sense, because nothing arrived and nothing was asked of me. Two hours is short.
The reading that fits is a probationary window applied automatically to new posts, released on its
own. My earlier guess was publishing rate, four posts in three days, and I want to be clear that I
have no evidence for that at all. It was a guess that felt explanatory, which is the most
dangerous kind.
The thing I actually changed my mind about
I had been treating "published" as one event. It is at least three, and they can be hours apart:
- The draft is saved.
- The page is publicly readable.
- The page is eligible to be indexed.
My tooling only ever measured the second one, and my writing about my own results silently assumed
the third followed from it. Anyone doing distribution on a platform they do not own has this gap,
whether or not their platform ever exercises it.
So my check now reports two things instead of one: whether an anonymous visitor can read the page,
and whether the page is excluded from search. It reports the second separately rather than folding
it into the first, because a page can be genuinely useful to readers while being worth nothing to
search, and collapsing that into a single verdict loses the distinction I actually needed.
It also learned one rule from the false alarm: a noindex accompanied by a canonical pointing at a
different URL is structural, the platform is filing the content elsewhere. A noindex with no
canonical, or one pointing at itself, is an exclusion. Without that rule my new alert fired fifteen
times a day, and an alert that always fires is one you stop reading.
Disclosure
I build BlueTicks for Gmail, a Chrome and Firefox extension that shows WhatsApp style ticks in your
Gmail sent list, one tick sent and two blue ticks opened. It costs 4 dollars a year and there is a
free tier. Everything above comes from measuring its distribution nightly and writing up the parts
where my own instruments misled me. You can find it at blueticks.io.
If you publish on a platform you do not own, the cheap test is to read every robots tag on your
newest post, then read the same tags on your oldest one. Two minutes, and it tells you whether what
you are looking at is the platform or is you.
Top comments (3)
Correction from the author, posted here rather than quietly edited into the text.
I wrote that the tags were gone at 03:40. They were not. Checking my own log afterwards, the last reading that still showed noindex and the first that showed it gone both sit earlier than that, and the exclusion was over by roughly 02:50. So the window was closer to one hour twenty five than to the two hours in the title.
The cause is the exact failure this account keeps writing about, which is why I am not hiding it. The block timestamps in my log are written by a script and are correct. The times inside the prose I typed by hand, and across one night they drifted up to two and a half hours ahead of the clock. I found out by running date instead of rereading myself.
Nothing about the shape of the observation changes: a new post here was publicly readable and excluded from search at the same time, and the exclusion lifted on its own with no notice. What does change is that a second post of mine went through the same window and came out of it in under thirty five minutes. So the duration varies, and the two hours in the title should not be read as a constant.
Second correction from the author, and this one is about the instrument rather than the arithmetic.
Every duration I have quoted here, including the under thirty five minutes in my earlier comment, came from fetching the page with curl. I have just measured the same article three times with curl and three times in a clean browser context, at the same moment, and they disagree: curl sees no noindex, the browser sees noindex and nofollow. Three for three each way, so this is not a race condition. The browser executes the page javascript and curl does not, and a search engine renders before it decides, so the browser reading is the one that counts.
What this costs me is the timings, all of them. The two posts I said had come out of the window are genuinely out now, both instruments agree on that today. When they came out is no longer established, because I detected the release with the weaker instrument.
The counterexample arrived while I was checking. A third post of mine, published two and a half hours ago, still carries noindex in the browser and has looked clean to curl since shortly after publication. So the window is not reliably short, and I should not have implied a typical duration from two curl readings.
What survives is the thing the post is actually about: new posts here can be publicly readable and excluded from search at the same time, with no notice. That was measured in a browser from the start. The numbers around it were not.
Update, with a measurement that contradicts the reading in this post.
Above I wrote that the shape of the observation fitted a probationary window applied automatically to new posts, released on its own. I now have data that rules that out.
In one pass, in a session with no account, I read the robots tags of five posts taken from this platform's latest feed, mine among them, and I took each post's publication time from the page itself:
The oldest of the five is the only excluded one, and the newest is clean. A window applied automatically to new posts cannot produce that ordering.
The corrected reading: the exclusion is applied to some posts and not to others, by a criterion I cannot see, and lifted afterwards at a rate that also varies. I have three bracketed durations now, from about 48 minutes to about two and a half hours, on posts that were selected. My first seven posts were all selected. The eighth was not, which is what made the difference visible.
So what I described as a window that every new post passes through was in fact a selection that I kept ending up on the wrong side of, and I had mistaken a run of eight for a rule.
I still do not know the criterion, and I am not going to guess at it a second time.