3 SEO bugs a Next.js app on Cloudflare Pages ships by default (and how we fixed them)

We run SongUp AI, an AI song generator, as a Next.js 14 App Router app on the edge runtime, deployed to Cloudflare Pages. It worked. It also quietly sabotaged our search visibility in three ways that none of the defaults warned us about.

Disclosure: I work on SongUp AI. Code below is simplified from our middleware.ts.

1. Google indexed our preview deployments — and ranked one above the real domain

Every Cloudflare Pages deployment answers on <hash>.<project>.pages.dev as well as on <project>.pages.dev. Those URLs serve a byte-identical copy of the site. Google found them, indexed them, and for a while ranked a pages.dev URL above www.songupai.com for our own brand name.

We already had rel="canonical" pointing at the real domain. It didn't help: canonical is a hint, and Google can overrule it when it sees two identical sites.

What works is a directive: X-Robots-Tag: noindex on every response that isn't served from the real host.

const CANONICAL_HOST = 'www.example.com';

function isIndexableHost(request: NextRequest): boolean {
  const host = (request.headers.get('host') || '').toLowerCase().split(':')[0];
  if (!host) return true;
  if (host === CANONICAL_HOST || host === 'example.com') return true;
  if (host === 'localhost' || host.startsWith('127.')) return true;
  return false;
}

export function middleware(request: NextRequest) {
  const response = NextResponse.next();
  if (!isIndexableHost(request)) {
    response.headers.set('X-Robots-Tag', 'noindex, nofollow');
  }
  return response;
}

The trap: don't also block those hosts in robots.txt. A disallowed URL is never recrawled, so Google never sees the noindex, and the already-indexed copies stay in the index indefinitely. Crawlable + noindex is what actually removes them.

2. Locale from Accept-Language means crawlers only ever see English

Our UI is translated into 30 languages, and the locale is picked from the browser's Accept-Language header on the same URL. Great for users. Invisible to search engines: Googlebot doesn't send Accept-Language, so it only ever sees the English page, and there is nothing to put in hreflang.

The fix was a small set of real, crawlable localized URLs whose language comes from the path, not the header:

const LANDING_PATH_LOCALE = /^\/(de|fr|es|pt)(?:\/|$)/;

const pathLocale = LANDING_PATH_LOCALE.exec(pathname)?.[1];
if (pathLocale) {
  activeLocale = pathLocale;            // crawlers see German on /de
} else if (acceptLang) {
  activeLocale = matchAcceptLanguage(acceptLang) ?? defaultLocale;
}

Then every page in the cluster declares the same hreflang set through the Metadata API:

export const HREFLANG_LANGUAGES = {
  en: 'https://www.example.com',
  de: 'https://www.example.com/de',
  fr: 'https://www.example.com/fr',
  es: 'https://www.example.com/es',
  pt: 'https://www.example.com/pt',
  'x-default': 'https://www.example.com',
};

export const metadata = {
  alternates: { canonical: 'https://www.example.com/de', languages: HREFLANG_LANGUAGES },
};

app/sitemap.ts accepts the same object under alternates.languages, so the sitemap and the <link rel="alternate"> tags can't drift apart.

Two details that mattered:

• Don't write the locale cookie on the localized pages. A visitor who lands on /de from Google shouldn't have the rest of the site switch to German.
• Only localize what you can deliver. Our songs can be sung in eight languages, so we built pages only for languages the product actually supports. A /it page promising Italian songs we can't make would be worse than no page. (Ours are at songupai.com/de, /fr, /es and /pt if you want to see the result.)

3. A Set-Cookie on every response makes every page uncacheable

Our middleware originally mirrored the detected locale into a NEXT_LOCALE cookie on every request. Any response with Set-Cookie is treated as private by caches, so the edge cache was effectively off for every page, including for crawlers.

The fix: English is the default everyone falls back to, so English responses get no cookie. Only set it when the locale differs from the default and from what the cookie already says; clear a stale one instead of rewriting it.

const shouldSetCookie =
  !pathLocale && activeLocale !== defaultLocale && cookieLocale !== activeLocale;
const shouldClearCookie =
  !pathLocale && activeLocale === defaultLocale && cookieLocale !== undefined;

We applied the same idea to a GDPR-zone cookie: only visitors in the EU/EEA/UK/CH get it, so everyone else receives a cookie-free, cacheable response.

Bonus: tell Bing directly with IndexNow

Google ignores IndexNow, but Bing uses it — and Bing's index feeds ChatGPT search and Microsoft Copilot. Setup is a text file and one POST:

// public/<KEY>.txt contains the key itself
await fetch('https://api.indexnow.org/indexnow', {
  method: 'POST',
  headers: { 'Content-Type': 'application/json; charset=utf-8' },
  body: JSON.stringify({
    host: 'www.example.com',
    key: KEY,
    keyLocation: `https://www.example.com/${KEY}.txt`,
    urlList, // up to 10,000 URLs per request
  }),
});

A 202 means accepted with key validation pending. Run it after deploys that add or rewrite pages, not on every deploy.

Checklist

• [ ] X-Robots-Tag: noindex on every non-canonical host (*.pages.dev, preview URLs)
• [ ] Those hosts are not disallowed in robots.txt
• [ ] At least one crawlable URL per language you really support, with a matching hreflang cluster in pages and sitemap
• [ ] No Set-Cookie on default-locale responses
• [ ] IndexNow key file live and submissions after content deploys

If you've hit other Cloudflare Pages + Next.js SEO surprises, I'd like to hear them in the comments.

Comments

[Raknaos]

The Disallow + noindex interaction is the one that keeps catching people: blocking the preview host in robots.txt stops the crawler before it can read the X-Robots-Tag, so the noindex never gets seen and the URL stays eligible. You have to let it crawl and then reject the index, which is the opposite of the intuition, and both halves need to be in the same checklist or one silently undoes the other.

The Set-Cookie point generalises past locale. Anything that writes a cookie on every response — an A/B bucket assignment, a CDN session id, a consent flag applied unconditionally — quietly turns every cacheable page into a private one. And the symptom is never TTFB, it is an indexation stall that looks like an SEO problem rather than a caching problem. Making the cookie conditional is the right shape, and clearing a stale one instead of rewriting it is the detail most implementations miss.

Sharing one alternates object between the Metadata API and sitemap.ts is what I would steal from this. hreflang clusters drift silently: a locale added to one side and not the other raises no error anywhere, and the engines just pick one of the two signals and move on.