DEV Community

Ankit Gupta
Ankit Gupta

Posted on Fully Autonomous

I pointed a canonical at the real page. Google indexed the 1 KB stub instead.

I run a static browser-games site. Every game has two URLs: /play/<slug>, a full page with the game, a description and instructions, and /game/<slug>, a bare player that other sites can put in an iframe. I wanted Google to index the first and ignore the second. It took three attempts, and the one I was most sure about is the one that failed.

How it started: the SPA fallback

At first /game/<slug> had no file behind it. Cloudflare Pages answers any unmatched path with index.html when a project has no 404.html, so a crawler fetching /game/battleship received the homepage's title, a canonical pointing at /, and index, follow. My JavaScript rewrote all three after the app loaded.

The raw HTML and the rendered page disagreed, and Google settled it in its own favour. URL Inspection in Search Console said:

/game/battleship   Submitted and indexed
                   Google-selected canonical:  /game/battleship
                   User-declared canonical:    /play/battleship   <- ignored
/play/battleship   URL is unknown to Google
Enter fullscreen mode Exit fullscreen mode

The empty shell was indexed. The 988-word page was invisible.

The same fallback did something worse. A URL under a language prefix that existed in no form at all drew 93 impressions at an average position of 8.8, because the fallback returned 200 for it too.

Attempt 1: a real file with noindex

I made /game/<slug> a real file of about 1.1 KB: the iframe, an attribution link, and noindex, follow. Because every URL that needed the fallback now had a file, I could ship a real 404.html, and unmatched paths finally returned 404.

Then I talked myself out of the noindex.

Attempt 2: canonical instead of noindex

The argument was a good one. noindex throws away whatever ranking a URL has earned, while a canonical moves it to the target. And these URLs had earned some: over eleven days, 48 different /game/ URLs drew impressions. Battleship had 68 impressions at around position 32, with 5 clicks.

So I swapped noindex for a canonical pointing at /play/<slug>. Nothing else about the stub was wrong: the canonical was correct, the URL was in no sitemap, and no internal link pointed at it.

Google recrawled the stub the next day. Search Console, the day after that:

/game/battleship   Submitted and indexed
                   Google-selected canonical:  /game/battleship   <- kept the stub
                   User-declared canonical:    /play/battleship   <- declared, ignored
Enter fullscreen mode Exit fullscreen mode

Both URLs were now in the index, competing with each other. That was the exact situation I had set out to end.

Why it failed

rel=canonical is a hint, and Google follows it between pages it considers duplicates of each other. A 1.1 KB page holding an iframe is nothing like a 988-word page about the same game. They share a subject, not content. So the hint was discarded.

That changes what the choice had been. I thought I was choosing between discarding the signal and forwarding it. The forwarding was never going to happen.

Attempt 3: noindex again, canonical removed

I put noindex, follow back and deleted the canonical instead of leaving the two side by side. A noindex next to a canonical that points somewhere else are two instructions that contradict each other, and the risk usually described is the noindex being applied to the canonical target. Here the target is the page I need indexed.

Sites that embed the player are not affected. An iframe requests the file and never reads a canonical.

What I would tell myself earlier

  1. Read "Google-selected canonical" in URL Inspection. The canonical you declared is only a request.
  2. Do not fix title, canonical and robots in JavaScript on a fallback page. The raw HTML and the rendered page will disagree, and you do not choose which one Google believes.
  3. A canonical between two pages that are not near-duplicates can be ignored. Mine was. Use noindex, or a redirect if nothing needs the URL.
  4. Do not combine noindex with a canonical to a different URL.
  5. Make unmatched URLs return 404. A fallback that answers 200 for everything will get URLs indexed that you never created.

The pages in this story are on 360 PlayZone, and the example is its Battleship page. If you have watched Google accept a canonical between two pages this different, I would like to hear how.

Written by an AI assistant from my own build notes and Search Console data.

Top comments (0)