DEV Community

Datawright Labs
Datawright Labs

Posted on

Bulk meta tags and Open Graph checker in Python: titles, descriptions, canonicals and og:image (2026)

#seo #python #webdev #tutorial

Meta tags are the part of a page nobody looks at until a link preview shows up blank in Slack, or Search Console says 400 pages have duplicate titles. Browser extensions check one page at a time. This post builds a script that checks a whole list and writes one CSV row per page, with a plain list of what's wrong.

What gets checked

Check Why it matters
Missing or long <title> (over ~60 chars) Search results truncate it, and it's still the strongest on-page signal.
Missing description, or outside ~70-160 chars Google often rewrites it anyway, but a missing one means it picks random page text.
No canonical, or more than one different canonical Duplicate URLs (tracking params, www, trailing slashes) split ranking signals.
noindex in meta robots or the X-Robots-Tag header The classic "why did this page vanish" bug after a staging deploy.
Zero, several or empty <h1> Not a ranking rule, but a reliable sign of a template problem.
Missing og:title / og:image, or an og:image that doesn't load Blank or broken previews on Slack, LinkedIn, Facebook, iMessage and Discord.
No twitter:card X falls back to a tiny summary card.
Invalid JSON-LD, duplicate titles across the list Structured data that doesn't parse is ignored; duplicate titles compete with each other.

The script 

# Usage: python3 meta_tags_check.py urls.txt
# Bulk SEO meta tags + Open Graph checker: fetches each page, extracts title, description, canonical, robots,
# h1, Open Graph / Twitter tags and JSON-LD types, flags common problems and writes meta_report.csv. Standard library only.
import csv, json, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'

class Meta(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.d = {'title': '', 'h1': [], 'canonical': [], 'jsonld': []}
        self.stack, self.buf = [], ''
    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or '') for k, v in attrs}
        if tag == 'svg': self.stack.append('svg')
        if tag in ('title', 'h1') and 'svg' not in self.stack or tag == 'script' and a.get('type', '').lower() == 'application/ld+json':
            self.stack.append(tag); self.buf = ''
        elif tag == 'meta':
            key = (a.get('property') or a.get('name') or '').lower()
            if key and key not in self.d: self.d[key] = a.get('content', '').strip()
        elif tag == 'link' and 'canonical' in a.get('rel', '').lower().split():
            self.d['canonical'].append(a.get('href', ''))
    def handle_data(self, data):
        if self.stack and self.stack[-1] in ('title', 'h1', 'script'): self.buf += data
    def handle_endtag(self, tag):
        if not self.stack or self.stack[-1] != tag: return
        self.stack.pop()
        text = ' '.join(self.buf.split())
        if tag == 'title' and not self.d['title']: self.d['title'] = text
        elif tag == 'h1': self.d['h1'].append(text)
        elif tag == 'script':
            try:
                items = json.loads(self.buf)
                for it in items if isinstance(items, list) else items.get('@graph', [items]):
                    t = it.get('@type') if isinstance(it, dict) else None
                    self.d['jsonld'] += t if isinstance(t, list) else [t] if t else []
            except ValueError:
                self.d['jsonld'].append('INVALID')

def image_ok(url):
    try:
        req = urllib.request.Request(url, method='HEAD', headers={'User-Agent': UA})
        with urllib.request.urlopen(req, timeout=10) as r:
            return r.status < 400 and r.headers.get('Content-Type', '').startswith('image/')
    except Exception:
        return False

def check(url):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers={'User-Agent': UA}), timeout=20) as r:
            final, html = r.url, r.read(2_000_000).decode(r.headers.get_content_charset() or 'utf-8', 'replace')
            xrobots = r.headers.get('X-Robots-Tag', '')
    except Exception as e:
        return {'url': url, 'error': f'{type(e).__name__}: {e}'[:120]}
    p = Meta(); p.feed(html); d = p.d
    title, desc, robots = d['title'], d.get('description', ''), (d.get('robots', '') + ' ' + xrobots).lower()
    og_image = urljoin(final, d.get('og:image', '')) if d.get('og:image') else ''
    problems = []
    if not title: problems.append('missing title')
    elif len(title) > 60: problems.append(f'title {len(title)} chars')
    if not desc: problems.append('missing description')
    elif not 70 <= len(desc) <= 160: problems.append(f'description {len(desc)} chars')
    if not d['canonical']: problems.append('no canonical')
    elif len(set(d['canonical'])) > 1: problems.append('multiple canonicals')
    if 'noindex' in robots: problems.append('noindex')
    if len(d['h1']) != 1: problems.append(f"{len(d['h1'])} h1")
    elif not d['h1'][0]: problems.append('empty h1')
    for k in ('og:title', 'og:image'):
        if not d.get(k): problems.append(f'missing {k}')
    if og_image and not image_ok(og_image): problems.append('og:image broken')
    if not d.get('twitter:card'): problems.append('no twitter:card')
    if 'INVALID' in d['jsonld']: problems.append('invalid JSON-LD')
    return {'url': url, 'final_url': final, 'title': title, 'description': desc, 'canonical': ' '.join(d['canonical']),
            'robots': robots.strip(), 'h1': ' | '.join(d['h1'])[:200], 'og_title': d.get('og:title', ''), 'og_image': og_image,
            'twitter_card': d.get('twitter:card', ''), 'jsonld_types': ' '.join(sorted(set(map(str, d['jsonld'])))),
            'problems': '; '.join(problems), 'error': ''}

if __name__ == '__main__':
    urls = [l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')]
    t = time.time()
    with ThreadPoolExecutor(10) as pool:
        rows = list(pool.map(check, urls))
    titles = {}
    for r in rows:
        if r.get('title'): titles.setdefault(r['title'], []).append(r['url'])
    for r in rows:
        if len(titles.get(r.get('title'), [])) > 1: r['problems'] = (r['problems'] + '; duplicate title').lstrip('; ')
    fields = ['url', 'final_url', 'title', 'description', 'canonical', 'robots', 'h1', 'og_title', 'og_image',
              'twitter_card', 'jsonld_types', 'problems', 'error']
    with open('meta_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, fields); w.writeheader(); w.writerows(rows)
    for r in rows:
        flag = 'ERR' if r['error'] else '!!' if r['problems'] else 'ok'
        print(f"{flag:3} {r['url'][:34]:34} {r['error'] or r['problems'] or r['jsonld_types']}"[:150])
    print(f'{len(rows)} URLs in {time.time() - t:.1f} s -> meta_report.csv')
Enter fullscreen mode Exit fullscreen mode

Three choices matter. HTMLParser from the standard library is enough here: meta tags are flat, so there's no need for BeautifulSoup. <title> elements inside inline <svg> icons are ignored, because many sites have several of them and a naive parser picks up "Close" or "Search" as the page title. And Open Graph tags are read from either property= or name=, since real sites use both (MDN uses name="og:title").

Results 

$ python3 meta_tags_check.py urls.txt
!!  https://www.python.org             description 52 chars; no canonical; 5 h1; no twitter:card
!!  https://github.com                 title 70 chars; description 186 chars
!!  https://example.com                missing description; no canonical; 0 h1; missing og:title; missing og:image; no twitter:card
!!  https://www.bbc.com/news           title 93 chars; description 249 chars; empty h1; missing og:image; no twitter:card
ok  https://developer.mozilla.org/en-U
!!  https://kubernetes.io              description 559 chars; no canonical
!!  https://www.theguardian.com/intern description 62 chars; 0 h1; missing og:title; missing og:image; no twitter:card
!!  https://stripe.com                 2 h1
ok  https://www.allrecipes.com/recipe/ NewsArticle Recipe
!!  https://docs.python.org/3/         description 34 chars; no twitter:card
ERR https://expired.badssl.com         URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_
11 URLs in 0.5 s -> meta_report.csv
Enter fullscreen mode Exit fullscreen mode

What this sample shows:

  • Big sites ship "problems" on purpose. BBC News has a 93-character title and a 249-character description, and the Guardian's international front has no Open Graph tags at all. Treat the length rules as a review list, not a verdict; on your own site they're usually worth fixing.
  • Multiple <h1>s are common. python.org uses five (one per carousel slide), Stripe two. Fine for users, but on a template you control it's often accidental.
  • Missing canonicals show up on well-run sites. python.org and kubernetes.io have none on their home pages. Not fatal, but on pages that are reachable through several URLs it's the cheapest duplicate-content fix there is.
  • Structured data comes along for free. The Allrecipes recipe page declares Recipe and NewsArticle JSON-LD, which is what powers its rich result.

Gotchas I hit

  • Bots get different HTML. Some sites serve a challenge page or 403 to unknown user agents, and the Facebook or Googlebot user agents get blocked from cloud IPs more often than a browser one. Send a browser User-Agent and treat challenge pages as "unknown", not "missing tags".
  • Check that og:image actually loads. A tag pointing at a deleted file is worse than no tag, because the preview breaks silently. The script does a HEAD request and expects an image/* content type. Some CDNs reject HEAD; fall back to a ranged GET if you see false alarms.
  • Relative og:image URLs exist. They're not valid per the Open Graph spec, but they're common; resolve them against the final URL before testing.
  • Cap the download. Some pages are several MB of inline JS. The script reads 2 MB, which always includes the <head>.
  • JavaScript-rendered tags won't appear. Single-page apps that inject meta tags client-side look empty to a plain HTTP fetch, which is also roughly what most link-preview bots see. That's a real finding, not a script bug.
  • Don't point it at internal addresses. If you expose this as a service that takes user input, block private and reserved IPs, or it becomes an SSRF tool.

Running it at scale

For a few hundred URLs the script is enough. If you'd rather not handle redirect tracking, charset detection, hreflang, Google rich-result required properties per schema.org type, microdata and RDFa, bot-challenge detection, Googlebot/Facebook/mobile user agents and a duplicate-title/description summary yourself, I also published it as a cloud tool: Meta Tags & Open Graph Checker on Apify (disclosure: I built it). It can return only the pages with problems, and costs $1 per 1,000 URLs checked; unreachable pages, bot-challenge pages, 4xx/5xx pages and invalid inputs are free.

Either way, run it before a launch and after every template change. A broken link preview is the first thing people see when they share your page.

Top comments (0)