DEV Community

Tùng Nguyễn
Tùng Nguyễn

Posted on Edited on

How to Build an Automated Technical SEO, Sitemap & Lead Audit Pipeline with Python & REST API

#python #seo #api #webdev

Technical SEO audits and B2B lead discovery often live in separate tools: one crawler checks <title>, <meta name="description">, H1 tags, canonical URLs, OpenGraph tags, and sitemap.xml / robots.txt health, while another script scrapes decision-maker emails and social profiles.

In this tutorial, we will combine both into a single high-speed Python Technical SEO, Sitemap & Lead Audit Pipeline — and show how you can call it directly via our RapidAPI Hub REST API or Apify Cloud Actor without managing servers or proxies.

⚡ Instant Cloud APIs, Interactive Demo & Pro Suite

🔍 What a Complete Technical SEO + Contact Audit Checks

When auditing prospect websites for an SEO agency, SaaS outreach campaign, or web development pitch, extracting the exact technical gaps alongside verified contact details lets you personalize cold outreach automatically (e.g., "Hi team, I noticed your homepage is missing a meta description and has 3 images without alt text..."):

  1. On-Page SEO Signals:
    • Title tag presence and character length (30–60 chars ideal)
    • Meta description presence and character length (70–160 chars ideal)
    • Canonical URL (<link rel="canonical">), Robots meta tag, and OpenGraph (og:title, og:image)
    • Heading hierarchy (H1, H2, H3 counts) and missing alt attributes on <img> tags
  2. Infrastructure & Discovery:
    • robots.txt status (200 OK vs missing)
    • sitemap.xml discovery and total indexed URL count
  3. Decision-Maker Contact Enrichment:
    • De-obfuscated emails (mailto: and body text) + phone numbers + LinkedIn / X / Facebook / Instagram links.

🐍 Python Implementation: Single-Pass SEO & Contact Auditor 

import re
from urllib.parse import urljoin
import requests
from bs4 import BeautifulSoup

EMAIL_RE = re.compile(r'[a-zA-Z0-9._%+\-]+@[a-zA-Z0-9.\-]+\.[a-zA-Z]{2,}')

def audit_website_seo_and_contacts(url: str) -> dict:
    base = url.rstrip('/')
    headers = {'User-Agent': 'Mozilla/5.0 (compatible; SEOAuditBot/1.0)'}
    resp = requests.get(base, timeout=15, headers=headers)
    soup = BeautifulSoup(resp.text, 'html.parser')

    title = soup.title.string.strip() if soup.title and soup.title.string else ''
    meta_desc_tag = soup.find('meta', attrs={'name': re.compile(r'^description$', re.I)})
    meta_desc = (meta_desc_tag.get('content') or '').strip() if meta_desc_tag else ''
    canonical_tag = soup.find('link', rel=lambda v: v and 'canonical' in v.lower())
    canonical = canonical_tag.get('href') if canonical_tag else None

    h1_tags = [h.get_text(" ", strip=True) for h in soup.find_all('h1')]
    imgs = soup.find_all('img')
    imgs_missing_alt = sum(1 for img in imgs if not (img.get('alt') or '').strip())

    # Check robots.txt and sitemap.xml
    has_robots = False
    has_sitemap = False
    try:
        r_rob = requests.get(urljoin(base + '/', 'robots.txt'), timeout=6, headers=headers)
        has_robots = (r_rob.status_code == 200 and len(r_rob.text) > 10)
    except Exception:
        pass
    try:
        r_site = requests.get(urljoin(base + '/', 'sitemap.xml'), timeout=6, headers=headers)
        has_sitemap = (r_site.status_code == 200 and '<urlset' in r_site.text[:500])
    except Exception:
        pass

    emails = sorted({
        m.lower() for m in EMAIL_RE.findall(resp.text)
        if not m.lower().endswith(('.png', '.jpg', '.svg', '.webp', '.js', '.css'))
    })

    issues = []
    if not title:
        issues.append('Missing <title> tag')
    if not meta_desc:
        issues.append('Missing <meta name="description">')
    if len(h1_tags) == 0:
        issues.append('Missing H1 heading')
    elif len(h1_tags) > 1:
        issues.append(f'Multiple H1 headings ({len(h1_tags)})')
    if imgs_missing_alt > 0:
        issues.append(f'{imgs_missing_alt} images missing alt text')

    return {
        'url': base,
        'statusCode': resp.status_code,
        'title': title,
        'titleLength': len(title),
        'metaDescription': meta_desc,
        'metaDescriptionLength': len(meta_desc),
        'canonical': canonical,
        'h1Count': len(h1_tags),
        'imagesTotal': len(imgs),
        'imagesMissingAlt': imgs_missing_alt,
        'hasRobotsTxt': has_robots,
        'hasSitemapXml': has_sitemap,
        'seoIssues': issues,
        'emails': emails,
    }
Enter fullscreen mode Exit fullscreen mode

🌐 Calling the Live REST API on RapidAPI Hub

Prefer a managed REST endpoint with a free monthly tier? You can call our Website Sitemap SEO Audit Contact Scraper API on RapidAPI Hub:

import requests

url = "https://website-sitemap-seo-audit-contact-scraper.p.rapidapi.com/v2/acts/tungnguyen84~website-sitemap-seo-audit-email-scraper/run-sync-get-dataset-items"
headers = {
    "x-rapidapi-host": "website-sitemap-seo-audit-contact-scraper.p.rapidapi.com",
    "x-rapidapi-key": "YOUR_RAPIDAPI_KEY",
    "Content-Type": "application/json"
}
payload = {
    "startUrls": [{"url": "https://example.com"}],
    "maxPagesPerSite": 3
}

response = requests.post(url, json=payload, headers=headers)
print(response.json())
Enter fullscreen mode Exit fullscreen mode

🚀 Explore the Full Developer Toolkit

Top comments (0)