DEV Community

Hammad Shams Uddin
Hammad Shams Uddin

Posted on

Page 999 of my 5-page blog returned 200 and said it was canonical

#webdev #seo #php #testing

Search Console, this morning:

Discovered - currently not indexed   20,852
Crawled - currently not indexed       1,582
Excluded by 'noindex' tag                63
Enter fullscreen mode Exit fullscreen mode

My sitemap has 6,240 URLs.

Google knows about roughly 22,500 addresses on a site that publishes 6,240. Nobody typed the other sixteen thousand. Something on the site is generating them.

The pager 

public static function paged(int $page = 1, int $perPage = 9): array
{
    $all   = self::all();
    $total = count($all);
    $pages = max(1, (int) ceil($total / $perPage));
    $page  = max(1, min($page, $pages));      // <- here
    return [
        'items' => array_slice($all, ($page - 1) * $perPage, $perPage),
        'total' => $total,
        'pages' => $pages,
        'page'  => $page,
    ];
}
Enter fullscreen mode Exit fullscreen mode

That clamp is the sensible thing to write. Somebody asks for page 40 of five, you show them page 5 rather than an empty screen or a stack trace. I would write it again.

The blog has 5 pages. Here is what the server does:

page=1      HTTP 200   9 posts   canonical = /blog
page=5      HTTP 200   3 posts   canonical = /blog?page=5
page=6      HTTP 200   3 posts   canonical = /blog?page=6
page=999    HTTP 200   3 posts   canonical = /blog?page=999
Enter fullscreen mode Exit fullscreen mode

Page 6 does not exist. It answers 200, serves page 5's three posts, and tells Google this URL is the canonical version of this content.

So does page 7. And page 8. There is no largest one.

The clamp was never the bug

The clamp is right, and the caller is wrong, and it took me a while to see which was which.

paged() returns 'page' => 5 when you ask for 999, because 5 is the page it is rendering. That is honest. But the template built its canonical from the request, not the response:

'canonical' => abs_url('/blog' . ($page > 1 ? '?page=' . $page : '')),
Enter fullscreen mode Exit fullscreen mode

$page there is 999. It was never reconciled with what the pager decided.

So the function quietly answered two different questions with one number, and the caller picked the wrong one:

  • which page am I showing? → 5
  • does the page you asked for exist? → never asked, never answered

The second question is the one a router needs, and there was no way to ask it. That is the actual defect: not a wrong value, a missing one.

$asked = max(1, $page);
$page  = min($asked, $pages);

return [
    'items'     => array_slice($all, ($page - 1) * $perPage, $perPage),
    'pages'     => $pages,
    'page'      => $page,        // what is being rendered
    'requested' => $asked,       // what was asked for
    'exists'    => $asked <= $pages,
];
Enter fullscreen mode Exit fullscreen mode 
$paged = BlogRepo::paged($page, 9);
if (!$paged['exists']) { notFound(); }
Enter fullscreen mode Exit fullscreen mode

and the canonical now uses $paged['page'] — the page that was served — rather than the number in the query string.

page=5      HTTP 200   3 posts   canonical = /blog?page=5
page=6      HTTP 404
page=999    HTTP 404
page=abc    HTTP 200   9 posts   canonical = /blog
Enter fullscreen mode Exit fullscreen mode

?page=abc still resolves to page 1, which is right: it is not a request for a page that does not exist, it is a request that does not name a page at all.

Why this shape is worth recognising

A clamp is a rendering decision. It answers "what do I draw". The moment its output is used to decide "what do I say this URL is", it has silently become a routing decision, and it is the wrong tool for that, because clamping's entire job is to erase the difference between valid and out-of-range input.

That is fine inside a view. It is not fine anywhere near an HTTP status or a canonical tag, and those are usually written a long way from the function doing the clamping.

The generalisation I would carry:

A function that normalises its input owes the caller the un-normalised value too. Otherwise every caller that needed to know "was this valid?" has to re-derive it, and the ones that forget do not fail — they succeed at something slightly wrong.

max() and min() are where this hides. So is ?? $default, intval(), and any array_slice that silently returns fewer rows than you asked for.

What I cannot claim

I found one unbounded URL source and I fixed it. I cannot tell you it accounts for 20,852, because Search Console's Page Indexing report does not expose the URLs through the API, and I am not going to work backwards from a number I want to explain.

What I can say is that this one is unbounded, it was reachable, it answered 200, and it invited indexing — and that until this morning I would have told you the site had 6,240 URLs.

If your Discovered count is wildly larger than your sitemap, the thing to look for is not a leak of pages. It is a parameter with no upper bound that still answers 200.

Testing it 

check('page 5 exists',        status('/blog?page=5'),   200);
check('page 6 does not',      status('/blog?page=6'),   404);
check('nor does page 999',    status('/blog?page=999'), 404);
check('a non-numeric page is page 1', canonical('/blog?page=abc'), '/blog');
check('page 5 is its own canonical',  canonical('/blog?page=5'), '/blog?page=5');
Enter fullscreen mode Exit fullscreen mode

The fourth and fifth assertions matter more than they look. The obvious fix — 404 anything that is not a valid page number — breaks ?page=abc, which should be page 1. The two cases sit either side of the same if and the temptation is to write one test.

I build Utilorax, a set of free browser-based tools. This one started with the sitemap URL counter, which is how I knew the sitemap said 6,240 while Search Console said 22,500.

Top comments (0)