A Free Python Script to Audit Your Site's On-Page SEO (Titles, Meta, H1s, Broken Links)

GroupAITools — Sun, 25 Oct 2026 03:43:39 +0000

Paid SEO crawlers are great, but sometimes you just want a quick answer to a few simple questions. Does every page have a title of sensible length? Is any page missing a meta description? Are there pages with zero or several H1 tags? Which links are broken?

You can answer all of these with about 100 lines of Python. In this post we'll build a small crawler that walks your site, checks the basics on every page, and writes the results to two CSV files you can open in Excel or Google Sheets.

What the Script Checks

For each HTML page on your domain:

Title tag: whether it exists and is roughly 30–60 characters
Meta description: whether it exists and is roughly 70–160 characters
H1 tags: exactly one per page
Canonical tag: whether one is present
Image alt text: a count of images with no alt
Links: every outgoing link (internal and external) is checked, and anything returning 4xx, 5xx or a connection error is flagged as broken

The length ranges are common rules of thumb, not hard rules from Google. Adjust them to suit your site.

Setup

You need Python 3.8+ and two libraries:

pip install requests beautifulsoup4

The Code

Save this as seo_audit.py:

#!/usr/bin/env python3
"""Tiny on-page SEO auditor: crawls a site and reports title, meta, H1 and broken-link issues."""
import sys, csv
from collections import deque
from urllib.parse import urljoin, urlparse, urldefrag
import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; mini-seo-audit/1.0)"}
MAX_PAGES = int(sys.argv[2]) if len(sys.argv) > 2 else 50

def fetch(url):
    try:
        return requests.get(url, headers=HEADERS, timeout=15, allow_redirects=True)
    except requests.RequestException:
        return None

def check_link(url, cache={}):
    if url not in cache:
        try:
            r = requests.head(url, headers=HEADERS, timeout=10, allow_redirects=True)
            if r.status_code in (405, 403):  # some servers dislike HEAD
                r = requests.get(url, headers=HEADERS, timeout=10, stream=True)
            cache[url] = r.status_code
        except requests.RequestException:
            cache[url] = "ERR"
    return cache[url]

def audit_page(url, html):
    soup = BeautifulSoup(html, "html.parser")
    issues = []
    title = (soup.title.string or "").strip() if soup.title else ""
    if not title:
        issues.append("missing <title>")
    elif not 30 <= len(title) <= 60:
        issues.append(f"title length {len(title)} (aim 30-60)")
    meta = soup.find("meta", attrs={"name": "description"})
    desc = (meta.get("content") or "").strip() if meta else ""
    if not desc:
        issues.append("missing meta description")
    elif not 70 <= len(desc) <= 160:
        issues.append(f"meta description length {len(desc)} (aim 70-160)")
    h1s = soup.find_all("h1")
    if len(h1s) != 1:
        issues.append(f"{len(h1s)} H1 tags (aim for exactly 1)")
    if not soup.find("link", rel="canonical"):
        issues.append("no canonical tag")
    missing_alt = [i for i in soup.find_all("img") if not i.get("alt")]
    if missing_alt:
        issues.append(f"{len(missing_alt)} images without alt text")
    links = set()
    for a in soup.find_all("a", href=True):
        href, _ = urldefrag(urljoin(url, a["href"]))
        if href.startswith("http"):
            links.add(href)
    return title, desc, issues, links

def main():
    start = sys.argv[1].rstrip("/") + "/"
    domain = urlparse(start).netloc
    queue, seen, rows, broken = deque([start]), {start}, [], []
    while queue and len(rows) < MAX_PAGES:
        url = queue.popleft()
        r = fetch(url)
        if r is None or "text/html" not in r.headers.get("Content-Type", ""):
            continue
        title, desc, issues, links = audit_page(url, r.text)
        rows.append([url, r.status_code, title, len(desc), "; ".join(issues)])
        print(f"[{len(rows)}] {url} -> {len(issues)} issue(s)")
        for link in links:
            status = check_link(link)
            if status == "ERR" or (isinstance(status, int) and status >= 400):
                broken.append([url, link, status])
            if urlparse(link).netloc == domain and link not in seen:
                seen.add(link)
                queue.append(link)
    with open("seo_report.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["url", "status", "title", "meta_desc_len", "issues"])
        w.writerows(rows)
    with open("broken_links.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["found_on", "broken_url", "status"])
        w.writerows(broken)
    print(f"\nAudited {len(rows)} pages, found {len(broken)} broken links.")
    print("Wrote seo_report.csv and broken_links.csv")

if __name__ == "__main__":
    if len(sys.argv) < 2:
        sys.exit("usage: python seo_audit.py https://example.com [max_pages]")
    main()

How It Works

Crawling. The script does a breadth-first crawl, starting from the URL you give it. It uses a deque as a queue and a seen set so each page is visited only once. Only links on the same domain (netloc) are added to the queue, so it won't wander off and crawl the whole internet. MAX_PAGES caps the crawl, so you can run a quick 20-page check or a fuller 500-page pass.

Parsing. BeautifulSoup reads each page, and audit_page() collects issues as plain strings. That makes it trivial to add your own checks. For example, you could flag noindex robots meta tags or pages missing Open Graph tags.

Link checking. Every absolute link is checked once. The mutable default argument cache={} is usually a Python gotcha, but here it's used on purpose as a simple memo, so a link that appears in your footer on 300 pages is requested only once. HEAD requests are cheap. Some servers reject them with 405 or 403, so the script falls back to a streamed GET.

Output. You get two files:

seo_report.csv: one row per page, with status, title, meta description length and a semicolon-separated list of issues
broken_links.csv: which page contains the broken link, the link itself, and its status

Running It

python seo_audit.py https://example.com 50

The second argument is the page limit (default 50). You'll see progress in the terminal:

[1] https://example.com/ -> 2 issue(s)
[2] https://example.com/about -> 0 issue(s)
...
Audited 50 pages, found 3 broken links.
Wrote seo_report.csv and broken_links.csv

Open the CSV, sort by the issues column, and you have a to-do list.

Things to Keep in Mind

Be polite. This script doesn't throttle requests. On a small shared host, add time.sleep(0.5) in the main loop. Only crawl sites you own or have permission to audit.
JavaScript sites. requests sees only the raw HTML. If your site renders content client-side (some React or Vue apps), titles and H1s may appear missing. You'd need a headless browser such as Playwright for those.
External link false positives. Some sites (LinkedIn is a classic) return 999 or 403 to bots even though the page works in a browser. Double-check external "broken" links by hand before you remove them.
robots.txt. The script doesn't read robots.txt. If you crawl beyond your own site, use urllib.robotparser to respect it.
Speed. Link checking is sequential, so large sites take a while. A concurrent.futures.ThreadPoolExecutor around check_link is an easy upgrade.

Ideas for Extending It

Detect duplicate titles and descriptions across pages, using a Counter on the title column
Flag thin pages by counting words in soup.get_text()
Check that og:title and og:image exist for nicer social previews
Pull URLs from sitemap.xml instead of crawling, to find orphan pages
Run it weekly in GitHub Actions and send yourself the CSV

Wrapping Up

A full-featured crawler is worth paying for on big sites. For small blogs, portfolios and side projects, though, a script like this catches most on-page problems in minutes, and you can see exactly what it checks. I built this while working on the SEO for GroupAITools, a small Indian site, because I wanted a fast sanity check between bigger audits. Feel free to copy it, change it and make it your own.

DEV Community: GroupAITools