Three Cloudflare defaults that silently broke my static site on launch day

#astro #cloudflare #webdev #seo

I shipped a static Astro site to Cloudflare Pages this week. Build was green, every local check passed, the custom domain resolved. Three things were still broken, and all three were platform defaults I never touched.

None of them throw an error. That is the whole problem.

1. Cloudflare rewrites your robots.txt

My repo has a four-line robots.txt. Allow everything, point at the sitemap. Here is what the live site actually served:

# BEGIN Cloudflare Managed content
User-agent: *
Content-Signal: search=yes,ai-train=no,use=reference
Allow: /

User-agent: Amazonbot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: GPTBot
Disallow: /
...
# END Cloudflare Managed Content

User-agent: *
Allow: /

That whole block is injected. It is the "Managed robots.txt" toggle under AI Crawl Control, on by default for a new zone. Googlebot and Bingbot still get through, so your search indexing looks fine and you have no reason to look.

If you actually want AI crawlers blocked, great, it did that for you. I wanted the opposite and had no idea it was happening. The toggle is at AI Crawl Control → Overview.

Check with one command, and read the whole body, not the status code:

curl -s https://yoursite.com/robots.txt

2. Pages adds a trailing slash, and it may not be the direction you assumed

I had configured trailingSlash: 'never'. Every canonical on the site pointed at the no-slash form. On Cloudflare Pages:

GET /bosses/how-many-bosses   →  308  →  /bosses/how-many-bosses/

Pages adds the slash. So every canonical URL on 33 pages pointed at a URL that redirects. Not fatal, but it costs a hop on every indexable page, and the local static server I tested against did the opposite, which is why I never saw it.

Worth knowing: this is a host behavior, not a framework one. You cannot settle it locally. Deploy one page and curl it.

3. Flipping trailing slash then broke my sitemap, silently

This is the one I actually enjoyed.

My sitemap filter excluded noindex pages by exact path match, built from filesystem routes, so /tags/foo. After the switch, url.pathname became /tags/foo/. The lookup stopped matching. Nothing errored. The sitemap went from 33 URLs to 44, quietly re-adding 11 pages that still carried noindex in their HTML.

A sitemap that lists noindexed pages is a contradiction you send Google on purpose. Fix is a one-liner, normalise before comparing:

const p = decodeURIComponent(new URL(url).pathname).replace(/\/+$/, '') || '/';
return !noindexPaths.has(p);

The lesson is not "normalise your paths". It is that a config change in one file silently changed the contract for a lookup in a different file, and both were "working" the entire time.

Bonus: Always Use HTTPS is off

New zone, default off. http:// serves a 200 instead of redirecting, and Search Console will happily index the http version as a separate property splitting your signals. SSL/TLS → Edge Certificates → Always Use HTTPS.

What I would do differently

I had a local production build, a header check, a link checker, and a sitemap test. All green. Every one of these three lived in the gap between "my build output" and "what the host actually serves".

So: deploy first, then run your checks against the live origin. Not the build directory. Not localhost. The origin.

The site was gawrguraquestforbread.com, a small fan wiki, if you want to see what the fixed output looks like.

DEV Community