DEV Community

Ali Asghar
Ali Asghar

Posted on Originally published at automateai.blog

A canonical tag bug was hiding 800 pages from Google

#django #seo #webdev #python

My Django blog had ~900 posts. Google had indexed almost none of them.
I assumed it was a content quality problem. It was partly that — but there
was a template bug underneath it that made the whole site nearly uncrawlable.

Bug 1: the canonical block default

In base.html:

<link rel="canonical" href="{% block canonical %}{{ site_config.site_name }}{% endblock %}">
Enter fullscreen mode Exit fullscreen mode

The default value of that block is the site name, not a URL. Only
post_detail.html overrode it. So every category page, tag page and search
page emitted:

<link rel="canonical" href="AutomateAI Blog">
Enter fullscreen mode Exit fullscreen mode

Google resolves that as a relative path — /category/x/AutomateAI%20Blog —
which 404s. The page gets dropped. And those category pages were the only
crawl path to the posts.

The fix:

<link rel="canonical" href="{% block canonical %}https://example.com{{ request.path }}{% endblock %}">
Enter fullscreen mode Exit fullscreen mode

Requires django.template.context_processors.request in your TEMPLATES
context_processors.

Bug 2: paginated pages canonicalising to the homepage

home.html hardcoded it:

{% block canonical %}https://example.com{% endblock %}
Enter fullscreen mode Exit fullscreen mode

With paginate_by = 10 and 900 posts that's ~90 pages, all declaring
themselves duplicates of page 1. Google drops them, and the links on them
carry no weight. So the oldest posts were both 89 clicks deep and on
pages Google had discarded.

Fix:

{% block canonical %}https://example.com/{% if page_obj.number > 1 %}?page={{ page_obj.number }}{% endif %}{% endblock %}
Enter fullscreen mode Exit fullscreen mode

How to check your own site 

curl -s https://yoursite.com/category/whatever/ | grep -i canonical
curl -s "https://yoursite.com/?page=3" | grep -i canonical
Enter fullscreen mode Exit fullscreen mode

The category page should return its own URL. The paginated page should
return its own URL, not the homepage.

What I'd check first next time

If Search Console shows a lot of "Discovered – currently not indexed",
don't assume it's a ranking problem. Check whether Google can actually
reach the pages at all:

  • canonical on category and paginated pages
  • crawl depth (how many clicks from the homepage to your oldest post?)
  • whether robots.txt is blocking a path you actually use

Mine had all three. The canonical was the one I'd never have found without
curling the raw HTML — it looked fine in the browser.

Top comments (0)