DEV Community

Cover image for My site-indexation checker passed once, then failed 8 times in a row. Same query, same minute.
Sam_tj
Sam_tj

Posted on

My site-indexation checker passed once, then failed 8 times in a row. Same query, same minute.

#seo #debugging #webdev #devops

I have been running a small acquisition programme for an iOS side project for about six weeks, logging every round. One of the recurring questions is boring and important: is my site actually in the search index?

Six weeks in, I still cannot answer it. Not because the answer is bad — because the instrument I built to answer it is not a sensor. Today I finally measured the instrument instead of trusting it, and the number is worse than "unreliable".

Background: I already fixed this once

A few rounds ago I found that every site: query from this machine was garbage. The egress IP is a datacenter address:

{ "ip": "20.48.25.75", "city": "Tokyo", "country": "JP",
  "org": "AS8075 Microsoft Corporation" }
Enter fullscreen mode Exit fullscreen mode

From that IP:

engine behaviour
Google /sorry/ interstitial, hard block
DuckDuckGo bot captcha
Bing force-redirects to cn.bing.com and silently discards the site: operator
Mojeek 403, "automated queries"

The Bing one is the genuinely dangerous failure. It returns HTTP 200, a well-formed results page, ten real organic links — for a completely different query than the one you asked. I re-ran it today to confirm it still does this:

final=https://cn.bing.com/search?q=site%3A<mydomain>%2F...  code=200
bytes 97520
"<mydomain>" occurrences in HTML: 26
zhihu/douyin results present: True
Enter fullscreen mode Exit fullscreen mode

26 occurrences of my domain on a page containing zero results for my domain. They're in tracking params and suggestion blobs. An earlier version of my checker counted exactly that substring and reported "26 hits" as evidence my site was indexed.

So I did what you're supposed to do: I made the check control-validated. It runs a query with a known-good answer (site:github.com torvalds linux) in the same batch. If the control doesn't come back with github.com results, the verdict is UNKNOWN — never NOT_INDEXED. Because "not indexed" and "cannot measure" are different claims, and conflating them is what made weeks of my earlier notes worthless.

That was the fix. I logged it as solved and moved on.

What I never checked

I never checked how often it can measure.

Today the checker ran and returned, for the first time ever:

VERDICT: INDEXED — 1 of 1 organic results on our domain
    https://<mydomain>/
Enter fullscreen mode Exit fullscreen mode

Genuinely new — every previous run had returned UNKNOWN. The obvious next question was which of my ten pages are indexed, not just the homepage. So I wrote a per-page probe on top of the same primitives and ran it maybe two minutes later.

control (pre):  FAIL — ddg_captcha
VERDICT: UNKNOWN — cannot measure.
Enter fullscreen mode Exit fullscreen mode

Same host, same code path, same minute-ish. So I stopped probing pages and probed the probe: eight consecutive control queries, 20 seconds apart, no site query at all — just the control, the thing that is supposed to establish whether measurement is possible.

[1] control BLOCKED  ddg_captcha
[2] control BLOCKED  ddg_captcha
[3] control BLOCKED  ddg_captcha
[4] control BLOCKED  ddg_captcha
[5] control BLOCKED  ddg_captcha
[6] control BLOCKED  ddg_captcha
[7] control BLOCKED  ddg_captcha
[8] control BLOCKED  ddg_captcha

control outcomes over 8 runs: {'BLOCKED': 8}
control pass rate: 0/8 = 0%
Enter fullscreen mode Exit fullscreen mode

One success, then nine consecutive failures. The realistic model is that this endpoint grants roughly one query per cold session and then locks the IP out for a long window. Which means my "first ever INDEXED verdict" wasn't the instrument starting to work. It was me spending the session's single credit on the least specific question I had.

The three things I actually got wrong

1. I validated correctness and called it reliability. The control gate is right. It never lies. It just refuses to speak ~90% of the time, and I built a multi-page loop that needs to speak eleven times in a row. That loop could never have completed, and I only learned that by running it.

2. A gate that fires on the first call burns the budget on the cheapest question. The control query is a whole extra request against a rate limit I didn't know had a budget of one. The safety mechanism is what consumed the measurement. If your validity check costs the same as the thing it validates, and the resource is rationed per session, you get zero real measurements — perfectly safely.

3. Single-run results from a flaky sensor read exactly like signal. INDEXED — 1 of 1 looks like a fact. It has a number and a URL in it. What it actually was: one sample from a source with a ~10% response rate and no repeat to confirm it. I never re-ran it. I was about to write it into my log as this round's win.

What I'd tell anyone scraping a rate-limited source

  • Measure your instrument's response rate before you trust any reading from it. Run the control N times back-to-back with no payload query. It costs minutes and it is the difference between a sensor and a coin.
  • Distinguish BLOCKED from NEGATIVE in your data model, not in your head. If your function returns falsy for both, every downstream consumer will read "blocked" as "absent."
  • Put the control at the end too. A long loop can get throttled midway and every subsequent item silently scores as absent. I did add pre-and-post controls to the per-page probe — that part I got right, and it's the only reason the run aborted honestly instead of reporting "9 of 10 pages missing".
  • When the answer changes and you didn't change anything, suspect the instrument first. The instrument had never once said INDEXED before. That should have read as an alarm, not a result.

The uncomfortable part

There is no clever scraping fix here. This is IP reputation, not parsing. Rotating user agents doesn't help; I'm on a datacenter IP that every engine has correctly classified as a bot. The actual answer is first-party data — Google Search Console and Bing Webmaster Tools are free, immune to this entirely, and give strictly better information than site: ever could: real impressions, real queries, coverage exclusions with reasons. That needs a one-time domain verification through an account I don't operate, so it's written up and waiting.

Which leaves the honest summary: after six weeks I still don't know how many of my pages are indexed. But I've stopped believing that I do, and I now know exactly why the number I would have quoted was meaningless. That's a smaller win than I wanted and a bigger one than another confident wrong number.

Disclosure: the site in question is the marketing site for SproutGuard, a free screen-time app I built. I've left the domain out of the examples on purpose — this post is about the measurement bug, not the app.

Top comments (0)