DEV Community

ushiro
ushiro

Posted on

Search Console Shows Your Canonical Tag. Googlebot Never Gets It.

#nextjs #webdev #seo #react

grep found my <link rel="canonical">. Search Console's URL Inspection showed it. It was still
landing outside <head> for the crawler that actually indexes the site.

Here is the check that finally showed it, on a dynamic route in Next.js 15.5:

#    HTML bytes   </head> at   canonical at
1       151,428        4,766         103,412
Enter fullscreen mode Exit fullscreen mode

The tag exists. It is 98,000 characters past the end of the head.

Why grep can't see this

Every tool I reached for first answers a different question than the one that matters.

curl -s https://example.com/some-page | grep -c 'rel="canonical"'
# 1
Enter fullscreen mode Exit fullscreen mode

That 1 is true and useless. Google ignores canonical and hreflang found outside <head>
in the raw HTML. React moves the tag into the head on the client, so a browser never shows you
anything wrong either — open DevTools and it is sitting in <head>, exactly where you put it.

The question is not whether the tag is present. It is whether it appears before </head>.

import urllib.request

UA = "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
req = urllib.request.Request("https://example.com/some-page", headers={"User-Agent": UA})
html = urllib.request.urlopen(req).read().decode("utf-8", "ignore")

head, canon = html.find("</head>"), html.find('rel="canonical"')
print(head, canon, "inside" if 0 <= canon < head else "OUTSIDE")
Enter fullscreen mode Exit fullscreen mode

The cause is a design decision, not a bug

Next 15.2+ streams metadata by default. The shell flushes as soon as the page body is ready, and
whatever generateMetadata returns is injected later — after </head> has already gone out on
the wire.

Next knows some clients cannot cope with that, so it keeps an exception list of user agents that
get a blocking render instead. It lives in next/dist/shared/lib/router/utils/html-bots.js:

const HTML_LIMITED_BOT_UA_RE =
  /[\w-]+-Google|Google-[\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot|baiduspider|yandex|sogou|bitlybot|tumblr|vkShare|quora link preview|redditbot|ia_archiver|Bingbot|BingPreview|applebot|facebookexternalhit|facebookcatalog|Twitterbot|LinkedInBot|Slackbot|Discordbot|WhatsApp|SkypeUriPreview|Yeti|googleweblight/i;
Enter fullscreen mode Exit fullscreen mode

Read the two Google patterns closely. [\w-]+-Google matches anything ending in -Google.
Google-[\w-]+ matches anything starting with Google-. Neither one matches the string
Googlebot.

Run it yourself:

const { HTML_LIMITED_BOT_UA_RE: re } =
  require('next/dist/shared/lib/router/utils/html-bots.js');

for (const ua of ['Googlebot', 'AdsBot-Google', 'Mediapartners-Google',
                  'Google-InspectionTool', 'Bingbot', 'Applebot']) {
  console.log(re.test(ua) ? 'blocking ' : 'streaming', ua);
}
Enter fullscreen mode Exit fullscreen mode

Output on next@15.5.21:

streaming Googlebot
blocking  AdsBot-Google
blocking  Mediapartners-Google
blocking  Google-InspectionTool
blocking  Bingbot
blocking  Applebot
Enter fullscreen mode Exit fullscreen mode

This is deliberate. The reasoning is that Googlebot executes JavaScript, so it will see the tag
after React relocates it — but only if the URL reaches the rendering queue, which is a second
pass and not guaranteed. The ad crawlers and the link-preview bots, which genuinely only read raw
HTML, are on the list.

The part that cost me the most time

Google-InspectionTool is the user agent behind URL Inspection in Search Console.

It matches Google-[\w-]+. It gets a blocking render. So the one tool you would naturally reach
for to verify the problem is served a complete, correct document — while plain Googlebot, the
thing that actually crawls you, is served the streamed one.

Bingbot is on the list too, so Bing Webmaster Tools looks healthy as well. Two independent
consoles agreeing that everything is fine is not confirmation here. They are both talking to the
blocking path.

It only happens sometimes, and only on some routes

This is what kept it hidden. Two conditions have to line up.

It is timing-dependent. Fetch a URL once, on its own, and the metadata almost always lands
inside <head> — the response is fast enough that the injection wins the race. It surfaces under
back-to-back load, which is what a crawler actually does to you. Measured across 116
/{provider}/{category} URLs fetched in sequence:

User agent canonical outside <head>
Chrome 21 / 116
Googlebot 16 / 116
Bingbot 0 / 116

Bingbot's zero is the control: same URLs, same moment, blocking render, never broken.

It is dynamic routes only. Anything served from ISR replays a complete, already-assembled
document, so the streaming race never happens. The x-nextjs-cache response header tells you
which kind of route you are looking at:

$ curl -sI https://example.com/some-hub | grep -i x-nextjs-cache
x-nextjs-cache: STALE      <- replayed from cache, safe

$ curl -sI https://example.com/some-dynamic-page | grep -i x-nextjs-cache
                           <- no header, rendered per request, at risk
Enter fullscreen mode Exit fullscreen mode

If everything you serve is statically generated or ISR-cached, none of this applies to you.

The fix

htmlLimitedBots in next.config.mjs lets you supply your own regex. One catch worth knowing:
it replaces Next's list rather than extending it. base-server.js reads
config.htmlLimitedBots || HTML_LIMITED_BOT_UA_RE_STRING, so writing /Googlebot/i there would
silently drop Bingbot, Applebot, Twitterbot and everything else.

Build it from Next's own source, so the list cannot go stale underneath you:

import { createRequire } from 'node:module';
const require_ = createRequire(import.meta.url);

const FALLBACK = '[\\w-]+-Google|Google-[\\w-]+|Chrome-Lighthouse|Slurp|DuckDuckBot';

let nextBots;
try {
  nextBots = require_('next/dist/shared/lib/router/utils/html-bots.js')
    .HTML_LIMITED_BOT_UA_RE.source;
} catch {
  nextBots = FALLBACK; // only if that internal path moves
}

export default {
  htmlLimitedBots: new RegExp(`Googlebot|${nextBots}`, 'i'),
};
Enter fullscreen mode Exit fullscreen mode

After deploying, the same back-to-back measurement:

canonical outside <head>, Googlebot UA:  0 / 116
Enter fullscreen mode Exit fullscreen mode

Re-checked ten days later on a dynamic route, ten parallel requests: </head> at 4,766,
canonical at 2,248, every single time.

Blocking costs nothing at this size. Median response with a Googlebot UA was 144–317ms.
Bingbot — already blocking, on the same URLs — was 129–222ms. Browsers keep streaming, because
Chrome does not match the pattern.

What this is not

It is an indexing signal, not a reachability one. Canonical outside <head> never stopped a
single crawl, and it explains nothing about crawl volume, coverage, or a page missing from the
index entirely. I checked whether it explained my own indexing gap before believing that it did,
and it did not — the pages I most wanted indexed were ISR-cached, so their canonical had been in
<head> the whole time.

What it does mean is narrower, and still worth fixing: on dynamic routes, some fraction of
Googlebot's fetches were told nothing about which URL is canonical or which language alternates
exist. On a site with one URL per thing, that fraction is a rounding error. On a site with ten
locales pointing at the same content, it is the entire signal.

The general lesson is one I keep relearning. When you verify a crawler-facing behaviour, send the
crawler's user agent, and check the thing's position, not its presence.

This came out of a project that crawls the docs of 15 AI vendors and records every change with
the date it happened: aichangewatch.com

Top comments (0)