DEV Community

seotto.com
seotto.com

Posted on Edited on

Find duplicate and missing title tags with a Python crawl script

#seo #python #automation #webdev

Duplicate title tags are one of the most common technical SEO problems, and one of the easiest to create by accident: a template change gives every product page the same title, a pagination setting repeats the category name, a CMS falls back to the site name when a field is empty. Search engines then have a harder time telling those pages apart. Here's a small script that reads your sitemap, fetches each page and reports duplicate, missing and overlong titles.

What it checks

  • Missing <title> or empty title
  • Titles shared by more than one URL
  • Titles that are very long or very short
  • Duplicate meta descriptions, as a bonus

The script 

#!/usr/bin/env python3
import sys, re, collections, concurrent.futures as cf
import requests
from xml.etree import ElementTree as ET
from html.parser import HTMLParser

UA = {"User-Agent": "Mozilla/5.0 (title audit)"}
NS = {"sm": "http://www.sitemaps.org/schemas/sitemap/0.9"}

def sitemap_urls(url):
    root = ET.fromstring(requests.get(url, headers=UA, timeout=20).content)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("sm:sitemap/sm:loc", NS):
            yield from sitemap_urls(loc.text.strip())
    else:
        for loc in root.findall("sm:url/sm:loc", NS):
            yield loc.text.strip()

class Head(HTMLParser):
    def __init__(self):
        super().__init__()
        self.title, self.desc, self._in_title = None, None, False
    def handle_starttag(self, tag, attrs):
        a = dict(attrs)
        if tag == "title" and self.title is None:
            self._in_title, self.title = True, ""
        if tag == "meta" and (a.get("name") or "").lower() == "description":
            self.desc = (a.get("content") or "").strip()
    def handle_endtag(self, tag):
        if tag == "title":
            self._in_title = False
    def handle_data(self, data):
        if self._in_title:
            self.title += data

def fetch(url):
    try:
        r = requests.get(url, headers=UA, timeout=20)
        p = Head(); p.feed(r.text)
        title = re.sub(r"\s+", " ", p.title or "").strip()
        return url, r.status_code, title, p.desc or ""
    except requests.RequestException as e:
        return url, type(e).__name__, "", ""

def main(sitemap):
    urls = list(dict.fromkeys(sitemap_urls(sitemap)))
    titles, descs = collections.defaultdict(list), collections.defaultdict(list)
    with cf.ThreadPoolExecutor(max_workers=4) as pool:
        for url, status, title, desc in pool.map(fetch, urls):
            if status != 200:
                print(f"[status {status}] {url}")
                continue
            if not title:
                print(f"[missing title] {url}")
            elif len(title) > 65 or len(title) < 15:
                print(f"[length {len(title)}] {url}: {title}")
            titles[title].append(url)
            if desc:
                descs[desc].append(url)
    for t, us in titles.items():
        if t and len(us) > 1:
            print(f"\n[duplicate title x{len(us)}] {t}")
            for u in us: print("   ", u)
    for d, us in descs.items():
        if len(us) > 1:
            print(f"\n[duplicate description x{len(us)}] {d[:80]}")
            for u in us: print("   ", u)

if __name__ == "__main__":
    main(sys.argv[1])
Enter fullscreen mode Exit fullscreen mode

Run it:

pip install requests
python3 titles.py https://example.com/sitemap.xml
Enter fullscreen mode Exit fullscreen mode

It uses only four parallel workers on purpose. Be gentle with your own server, especially in production.

Reading the results

Duplicates on paginated pages (/blog/page/2, /blog/page/3): add the page number to the title, e.g. "Blog, page 2". That's usually a template setting.

Duplicates on near-identical pages (product variants, filters, tracking parameters): decide whether they should exist as separate URLs at all. Often the fix is a canonical tag or removing them from the sitemap, not a new title.

Missing titles: almost always a template or CMS field fallback. Find which template the pages share.

Length warnings: there's no fixed limit, since search engines truncate by pixel width, not characters. Treat 65 characters as a rough guide for when the end of the title may be cut off, not a rule.

Why titles change without anyone noticing

Titles are generated from templates, plugins and CMS fields, so a single change can rewrite hundreds of them at once: a plugin update changes the separator, a theme update drops the brand suffix, a translation file overwrites a field. A one-off audit catches today's problems; it doesn't catch next week's.

Turning it into a regular check

You could run the script on a schedule and diff its output with the previous run. Save the URL-to-title map as JSON and compare. That's the core idea behind SEO monitoring tools. SEOtto does this for titles, meta descriptions, H1s, canonicals and noindex tags on every crawl, and alerts when a value changes.

Related reading

Top comments (0)