DEV Community

Radar Labs
Radar Labs

Posted on

My site answered 200 on all 163 URLs. Crawlers never saw a single page.

#webdev #seo #python #devops

My site answered 200 on all 163 URLs. Crawlers never saw a single page.

For two months I ran a small data site: 163 URLs, 152 generated company pages, a 1.7 MB CSV, valid sitemap.xml, and an IndexNow submission that returned 200 every single week.

Every check I had was green. And every search engine had zero of my pages.

This is the story of what was wrong, because I think it is a common failure and the symptom looks exactly like success.

The clue was one request away

On a hunch I fetched the same file twice:

curl -s https://example.site/robots.txt | wc -c
# 26

curl -s "https://example.site/robots.txt?probe=1" | wc -c
# 162
Enter fullscreen mode Exit fullscreen mode

Two different files from the same path. The 26-byte one was:

User-agent: *
Disallow: /
Enter fullscreen mode Exit fullscreen mode

The 162-byte one was mine, the one I had deployed:

User-agent: *
Allow: /
Sitemap: https://example.site/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

The host was rewriting the bare path. ?probe=1 bypassed whatever was doing it, so every "is my robots.txt right?" check I had ever run against the bare URL was reading the host's stub, not my file. Crawlers read the bare path. So for two months I was politely asking every crawler in the world to go away, while my dashboard said the site was up.

Three checks confirmed it was the platform, not a cache:

  1. The injected response had no charset, no Cache-Control, no ETag; mine had all three.
  2. Deploying the identical build to a brand-new subdomain produced the same 26 bytes — so it was not a stale CDN object.
  3. The response headers of the injected copy came from the platform's edge, not from my deployment.

The second bug, in the same audit

While auditing I submitted my sitemap to IndexNow and got:

HTTP 422 {"errorCode":"InvalidRequestParameters",
          "message":"One or more URLs are not related to your verified domain."}
Enter fullscreen mode Exit fullscreen mode

Seven URLs in a 163-URL sitemap still pointed at the old hostname. The sitemap generator only rewrote the section it owned and left a hand-written header block from weeks earlier intact.

And here is the embarrassing part: my own residue check had already reported "0 occurrences". I had written:

Select-String -Path site\*.html -Pattern 'old-host\.com' -SimpleMatch
Enter fullscreen mode Exit fullscreen mode

-SimpleMatch treats the pattern as a literal string — backslashes included. So I was searching for old-host\.com with an actual backslash in it, which matches nothing, ever. The check was a no-op wearing a green badge.

What I changed

The real fix was not the migration. It was making the failure impossible to miss. Every deploy now runs a gate before publishing:

# robots.txt must not block the site, and the bare path must match ?probe=1
if re.search(r"(?im)^\s*disallow:\s*/\s*$", robots):   problems.append("robots.txt blocks the whole site")
if not re.search(r"(?im)^\s*allow:\s*/\s*$", robots):   problems.append("no site-wide Allow: /")
_, fresh = fetch(robots_url + "?probe=1")
if fresh != raw:                                        problems.append("bare URL differs from ?probe=1 (crawlers read the stale one)")

# sitemap must be single-host, and sampled pages must not be noindex
hosts = {u.split("/")[2] for u in locs if "://" in u}
if hosts != {site_host}:                                problems.append(f"foreign host in sitemap: {hosts}")
Enter fullscreen mode Exit fullscreen mode

That gate is step 8 of 10 in the weekly pipeline now, and it is allowed to fail the run.

I moved to a host whose robots.txt I actually control end to end (a GitHub Pages user site, served at the domain root — a project site under /repo-name/ would have broken every root-absolute link I have). Same bytes, different owner:

$ curl -s https://my-site/robots.txt
User-agent: *
Allow: /
Sitemap: https://my-site/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

Four things I would tell my past self

1. 200 OK is not a statement about crawlability. It is a statement about HTTP. Everything that decides whether a page enters an index — robots.txt, noindex, canonical tags, sitemap host consistency, whether the CDN serves a different body on the bare path — lives beside the status code, not in it.

2. Verify what machines read, not what your browser showed you. A browser hides this class of bug completely, because you looked at ?probe=1 when you debugged, and the crawler looks at /.

3. Compare the bare URL with a query-string URL. It costs one request and catches platform-injected responses, stale edge objects, and rewrite rules. I now do it on robots.txt on every deploy.

4. A dashboard that is green in every cell can hide a total failure. My "163/163 URLs 200" metric was true and useless. The metric that would have caught this is "does the file a crawler fetches contain what I deployed".

The upside of finding it

The audit also forced two fixes I would not have made otherwise: a sitemap that can never drift to a foreign host, and a robots.txt with no BOM (a UTF-8 BOM in robots.txt silently breaks the first directive, which is its own quiet version of this bug).

The dataset I was trying to get indexed is public: 8,040 job postings that state a salary, from 266 companies, normalised to annual USD — dataset page, raw CSV on GitHub, CC BY 4.0.

If you are running a content site and your traffic is zero while every check is green, spend one request on robots.txt?probe=1. It is the cheapest test in this post and it took me two months to run it.

Top comments (1)

Collapse
 
contentmation profile image
ContentMation •

Great write up. One more cheap check in the same spirit: fetch robots.txt and one page with a crawler user agent too (Googlebot or bingbot), because some edges serve a different body by user agent, not just by query string. And the robots.txt report in Search Console shows the exact file Google fetched, which would have flagged the 26 byte stub on day one. For a quick outside view, nexusbro.com/audit reads robots.txt, the sitemap, canonicals and the raw HTML a crawler gets in one pass, free on any URL.