Two days ago we requested indexing on ten pages and one came back rejected — not by Google, but by our own response header: X-Robots-Tag: noindex, follow on /sign-up.
A sitemap sweep (3,162 URLs, zero hits) proved the header wasn't template-wide. Yesterday a commenter suggested the next move: stop hunting for URLs and search for where the header gets set — across CDN rules, server config, and app middleware. This post is that search, done end to end.
Layer 1 — CDN. Our zone runs on Cloudflare. The rules list: zero URL rewrite rules, zero redirect rules, zero configuration rules, zero origin rules, zero response-header transform rules. Two cache rules exist; both only set caching behavior. Clean — nothing at the edge touches headers.
Layer 2 — server config. None exists to search: the site is a Next.js app. No nginx file, no .htaccess. If you're on a similar stack, you can skip this layer.
Layer 3 — app middleware. Found it in one grep: a single middleware sets X-Robots-Tag: noindex, follow under three conditions —
- the route is in a private-page list: auth pages (sign-in, sign-up, password reset, verify-email, redeem-invite), account areas (settings, admin, personal data and history pages), newsletter preference pages;
- the URL carries search or filter params (q, search, tag, sort and friends);
- a solutions page carries a
?run=parameter.
One API route sets the header directly as well.
The confirm step. Reading code says what should happen; curl says what does. One URL per matching route family, following redirects:
| URL | X-Robots-Tag |
|---|---|
| /sign-up | noindex, follow |
| /en/sign-in | noindex, follow |
| /learn?q=seo | noindex, follow |
| /solutions?run=test | noindex, follow |
| / | none |
| /solutions | none |
(Both /solutions curls return 404 for us — the header answer still reads, because middleware runs before routing; those two rows demonstrate the ?run= rule, not a live page.)
Every family behaved exactly as the middleware promised. (One gotcha we hit first: our apex domain 308-redirects to www, and the redirect response itself carries no header — always follow redirects before reading the answer, or you'll conclude the header vanished when it hasn't.)
The ending is mundane. The header that "rejected" our indexing request is by design: /sign-up sits in the private-page list on purpose, alongside settings and admin. Nothing is broken. If we ever want the signup page indexed, that's a product decision — a one-line change — and it's now sitting with the product side while the daily indexing quota keeps going to crawlable pages.
The full runbook this thread built, in order:
- Test the route, not the template (one curl against the exact URL Google inspected).
- Sweep the sitemap for the header (a one-line curl loop; we ran 3,162 URLs in minutes with a parallel wrapper).
- Search the layers where the header gets set — CDN rules, server config, app middleware — instead of hunting URLs nobody has linked yet.
- Curl one URL per matching route family to confirm the code matches reality.
Four steps, maybe an hour, and the mystery goes from "Google rejected us" to "our own middleware is doing exactly what someone told it to." Which is the best kind of mystery: the one that was never a mystery.
Top comments (0)