The host you stopped thinking about
Cloudflare Pages gives every project a pages.dev subdomain. You point a custom domain at it, you tell people the custom domain, and the other one carries on serving. That is the whole bug.
On 26 August 2026 we checked ours from outside. pip-web-9oj.pages.dev returned 200 with the current production build on it, every path, and no X-Robots-Tag of any kind. The site was crawlable on two hostnames, and the copy did not know it was the copy.
The documentation covers the other host
Cloudflare does add the header, and says so plainly: every preview deployment generated by Cloudflare Pages includes the X-Robots-Tag: noindex HTTP response header.
A preview deployment is the per-commit one, at <hash>.pip-web-9oj.pages.dev. The production alias is a different address, it is the one printed in your dashboard, and that page does not mention it either way. So the sentence that sounds like reassurance is about a host you were not worried about. We read it as covering both for five weeks.
The canonical tag was already right, and did not help
Every page on the site carries a canonical pointing at playpip.io, including the pages served on the alias. It was correct the entire time.
It is also not an instruction. Google’s own documentation calls rel="canonical" a strong signal that the specified URL should become canonical, and the page it appears on is upfront that Google decides which version to show. A signal is weighed. A header is obeyed.
On a domain five weeks old with almost no links pointing at it, that difference is the difference between a weak set of ranking signals and two weaker ones. Nobody arrives at your site by ranking twice.
Why it is not a redirect
The obvious fix is a 301 from the alias to the real domain. On Pages a redirect with logic in it means a Function, and the only Function that sees a request for a static file is functions/_middleware.ts at the root. Cloudflare’s docs are specific about what that costs: if you want to run a middleware on your entire application, including in front of static files, create a functions/_middleware.js file.
Entire application includes the host you actually care about. You would be putting a Worker invocation in front of every chunk, image and font on playpip.io in order to tidy up a hostname nobody visits. You can check the hostname and return early inside the middleware, but by then the invocation has happened: that is the part you were trying not to pay for.
public/_headers is read by the CDN, applies per host, and costs nothing at runtime.
The rule
https://pip-web-9oj.pages.dev/*
X-Robots-Tag: noindex
A _headers pattern written with a scheme and a host matches that host only. playpip.io is untouched, and preview deployments are a third host again and keep the noindex Cloudflare gives them, which is worth preserving: a preview’s permanent address is how you audit a build from outside after something has gone wrong on it.
The same file, one edit later
Now delete the host from that pattern.
/*
X-Robots-Tag: noindex
The site is gone from search. Every path, on the real domain. The deploy is green, the suite passes, the site opens perfectly for every human being who visits, and there is nothing in the repository a linter or a type checker could object to, because _headers is a text file the CDN interprets after you have shipped it. It is the one file here that can do that, and nothing was reading it.
So something does now. tests/headers.test.ts parses the file the way Pages does, an unindented line being a pattern and the indented lines under it being its headers, and holds one invariant: no rule that sets noindex may match the site itself. It collects the offending rules and compares the list to empty, rather than asserting rule by rule, so it still runs an assertion when the file contains no noindex at all. A test that passes by finding nothing to check is the other way to be green and wrong.
Then break the file three ways and watch, because a test written against a file that already passes has proved nothing about itself. Dropping the host fails two tests. Deleting the whole rule fails one. Taking the noindex line out and leaving the pattern behind fails two others, and the invariant test goes green, which is the right answer: there is no noindex left anywhere to reach anything. It is still executing a comparison at that point rather than skipping an empty loop, and that is the only reason its green is worth reading.
What we cannot tell you
What the duplicate cost us, because we never measured it. The alias was never verified as a Search Console property of ours, so there is no report to go back and open, and we fixed it the morning we found it rather than leaving it up to find out. It may have cost nothing at all. It was also five weeks of a young domain’s crawling spent twice on the same pages, which is the sort of thing worth fixing without a number attached.
Thirty seconds
If you are on Pages, run curl -I https://<your-project>.pages.dev/ and look for x-robots-tag in the output. If it is not there, you are the second website about your own product, and you are competing with yourself for the handful of people who were going to find either one.
Top comments (0)