Audit 1,000 websites for speed, SEO, AI-crawler access and domain expiry (with plain Python, no scraping)

If you run an agency, manage a portfolio of sites or just inherited 300 client domains, you eventually need answers to four boring but expensive questions:

1. Which domains expire soon? (A lapsed domain takes email and the website down with it.)
2. Can someone send email in our name? (Missing SPF or DMARC = easy spoofing.)
3. Which pages fail Core Web Vitals? (Google uses real-user data for ranking.)
4. Can AI assistants actually read our content? (Many sites block GPTBot or ClaudeBot by accident.)

The good news: all four can be answered from official, documented sources – no scraping, no captchas, no gray areas. Here is how, in a few lines of Python each.

1. Domain expiry with RDAP (the official successor of WHOIS)

WHOIS output is unstructured text that differs per registry. RDAP returns JSON and is mandated by ICANN for all generic TLDs. IANA publishes which server is responsible for which TLD:

import httpx

boot = httpx.get("https://data.iana.org/rdap/dns.json").json()
servers = {tld: urls[-1] for tlds, urls in boot["services"] for tld in tlds}

def expiry(domain: str) -> str | None:
    base = servers[domain.rsplit(".", 1)[1]].rstrip("/")
    r = httpx.get(f"{base}/domain/{domain}", follow_redirects=True)
    if r.status_code == 404:
        return None  # not registered (likely available)
    events = {e["eventAction"]: e["eventDate"] for e in r.json().get("events", [])}
    return events.get("expiration")

print(expiry("example.com"))  # e.g. 2027-08-13T04:00:00Z

Gotchas I hit in production:

• Rate limits differ per registry. Throttle per RDAP server, not globally, and honour Retry-After on 429.
• Many country TLDs (.de, .at, .ch, .io …) have no RDAP. You can still tell registered vs. likely available via DNS: a delegated domain has NS records, an unregistered one returns NXDOMAIN.
• Do not store registrant data. Most of it is redacted anyway; leave the rest out (GDPR).

2. Email spoofing protection: SPF and DMARC via DNS-over-HTTPS

No resolver setup needed – Google and Cloudflare answer DNS queries as JSON:

def txt(name: str) -> list[str]:
    r = httpx.get("https://dns.google/resolve", params={"name": name, "type": "TXT"}).json()
    return ["".join(p for p in a["data"].split('"') if p.strip())
            for a in r.get("Answer", []) if a["type"] == 16]

spf = [t for t in txt("example.com") if t.lower().startswith("v=spf1")]
dmarc = [t for t in txt("_dmarc.example.com") if t.lower().startswith("v=dmarc1")]
print(spf, dmarc)

What to flag:

• No SPF, more than one SPF record (invalid per RFC 7208), or +all / ?all.
• No DMARC, or p=none (monitoring only – nothing is blocked).
• Domains that send no email should still publish v=spf1 -all and p=reject. Parked domains are a favourite for phishing.
• A single MX record pointing to . is a null MX (RFC 7505): the domain deliberately accepts no mail.

3. Core Web Vitals with the PageSpeed Insights API

The PSI API returns both Lighthouse lab data and real-user CrUX data in one call. A free API key from Google Cloud raises the quota to 25,000 requests per day:

API = "https://www.googleapis.com/pagespeedonline/v5/runPagespeed"
r = httpx.get(API, params={"url": "https://example.com", "strategy": "mobile",
                           "category": "performance", "key": KEY}, timeout=150).json()

score = r["lighthouseResult"]["categories"]["performance"]["score"] * 100
field = r.get("loadingExperience", {}).get("metrics", {})
lcp = field.get("LARGEST_CONTENTFUL_PAINT_MS", {}).get("percentile")     # ms, p75
inp = field.get("INTERACTION_TO_NEXT_PAINT", {}).get("percentile")       # ms, p75
cls = field.get("CUMULATIVE_LAYOUT_SHIFT_SCORE", {}).get("percentile")   # x100!
print(score, lcp, inp, cls and cls / 100)

Things that are easy to get wrong:

• "Good" thresholds at p75: LCP ≤ 2.5 s, INP ≤ 200 ms, CLS ≤ 0.1. A page can score 90 in Lighthouse and still fail on real users.
• CLS comes multiplied by 100 in the field data.
• Low-traffic pages have no field data; PSI may fall back to the whole origin (origin_fallback). Report which one you got.
• Lighthouse scores fluctuate by a few points between runs. If you monitor, alert on changes of 10+ points, not on every wobble.
• Pages that Lighthouse cannot load return HTTP 400 with codes like FAILED_DOCUMENT_REQUEST – don't retry those.

4. Can AI assistants read your site?

Generative search (ChatGPT, Perplexity, Claude, Gemini) is becoming a traffic source. Two quick checks:

robots = httpx.get("https://example.com/robots.txt").text.lower()
for bot in ["gptbot", "oai-searchbot", "claudebot", "perplexitybot", "google-extended"]:
    print(bot, "mentioned" if f"user-agent: {bot}" in robots else "not mentioned")

has_llms_txt = httpx.get("https://example.com/llms.txt").status_code == 200

A proper check parses robots.txt rules per user agent (a blanket Disallow: / under User-agent: * blocks them too), and adds structured data (JSON-LD), titles, meta descriptions, canonicals and headings.

Doing it for hundreds of sites

The snippets above are fine for ten domains. For hundreds you need concurrency, per-server throttling, retries, sitemap parsing, CSV output and scheduling. If you'd rather not maintain that, I packaged these checks as hosted Apify Actors. You paste a list, get one CSV-ready row per site, and failed checks are not charged:

• Bulk WHOIS (RDAP) & Domain Expiry, DNS, SPF/DMARC Checker: expiry, registrar, availability, DNS and an email-security grade with reasons. $2.50 per 1,000 domains.
• Bulk PageSpeed Insights Checker: Lighthouse & Core Web Vitals: mobile and desktop in one row, real-user CWV pass/fail, top 3 fixes, sitemap input and a monitoring mode that only flags real changes. $5 per 1,000 audits.
• SEO & AI Search (GEO) Audit: on-page SEO, structured data, AI-crawler rules and llms.txt per page, with concrete fixes. $15 per 1,000 pages.

They use only the official sources described above, store no personal data, and also work as MCP tools for AI agents.

Disclosure: I built and maintain these Actors. The code in this article is free to use; feedback and bug reports are very welcome in the comments.

Comments

[Oleksandr]

The RDAP bootstrapping from data.iana.org is a really elegant way to handle TLDs without hardcoding a massive list. Also, the CLS x100 quirk in the CrUX data is a classic API trap that trips up so many developers. Great practical guide!

[Steven Mock]

This is the checklist I wish more small business owners saw before something breaks. The domain expiry one is so quiet and so expensive — a lapsed domain takes the site and the email down at once, and nobody notices until a client calls. The AI-crawler point is the one I'm watching closest now. A lot of decent sites are accidentally blocking GPTBot or ClaudeBot with a blanket disallow, so they're invisible in the places people are actually asking questions. Thanks for laying out the official-source approach instead of another scraping workaround.