DEV Community: Datawright Labs

Bulk meta tags and Open Graph checker in Python: titles, descriptions, canonicals and og:image (2026)

Datawright Labs — Mon, 26 Oct 2026 00:13:10 +0000

Meta tags are the part of a page nobody looks at until a link preview shows up blank in Slack, or Search Console says 400 pages have duplicate titles. Browser extensions check one page at a time. This post builds a script that checks a whole list and writes one CSV row per page, with a plain list of what's wrong.

What gets checked

Check	Why it matters
Missing or long `<title>` (over ~60 chars)	Search results truncate it, and it's still the strongest on-page signal.
Missing description, or outside ~70-160 chars	Google often rewrites it anyway, but a missing one means it picks random page text.
No canonical, or more than one different canonical	Duplicate URLs (tracking params, `www`, trailing slashes) split ranking signals.
`noindex` in meta robots or the `X-Robots-Tag` header	The classic "why did this page vanish" bug after a staging deploy.
Zero, several or empty `<h1>`	Not a ranking rule, but a reliable sign of a template problem.
Missing `og:title` / `og:image`, or an `og:image` that doesn't load	Blank or broken previews on Slack, LinkedIn, Facebook, iMessage and Discord.
No `twitter:card`	X falls back to a tiny summary card.
Invalid JSON-LD, duplicate titles across the list	Structured data that doesn't parse is ignored; duplicate titles compete with each other.

The script

# Usage: python3 meta_tags_check.py urls.txt
# Bulk SEO meta tags + Open Graph checker: fetches each page, extracts title, description, canonical, robots,
# h1, Open Graph / Twitter tags and JSON-LD types, flags common problems and writes meta_report.csv. Standard library only.
import csv, json, sys, time, urllib.request
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser
from urllib.parse import urljoin

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'

class Meta(HTMLParser):
    def __init__(self):
        super().__init__(convert_charrefs=True)
        self.d = {'title': '', 'h1': [], 'canonical': [], 'jsonld': []}
        self.stack, self.buf = [], ''
    def handle_starttag(self, tag, attrs):
        a = {k.lower(): (v or '') for k, v in attrs}
        if tag == 'svg': self.stack.append('svg')
        if tag in ('title', 'h1') and 'svg' not in self.stack or tag == 'script' and a.get('type', '').lower() == 'application/ld+json':
            self.stack.append(tag); self.buf = ''
        elif tag == 'meta':
            key = (a.get('property') or a.get('name') or '').lower()
            if key and key not in self.d: self.d[key] = a.get('content', '').strip()
        elif tag == 'link' and 'canonical' in a.get('rel', '').lower().split():
            self.d['canonical'].append(a.get('href', ''))
    def handle_data(self, data):
        if self.stack and self.stack[-1] in ('title', 'h1', 'script'): self.buf += data
    def handle_endtag(self, tag):
        if not self.stack or self.stack[-1] != tag: return
        self.stack.pop()
        text = ' '.join(self.buf.split())
        if tag == 'title' and not self.d['title']: self.d['title'] = text
        elif tag == 'h1': self.d['h1'].append(text)
        elif tag == 'script':
            try:
                items = json.loads(self.buf)
                for it in items if isinstance(items, list) else items.get('@graph', [items]):
                    t = it.get('@type') if isinstance(it, dict) else None
                    self.d['jsonld'] += t if isinstance(t, list) else [t] if t else []
            except ValueError:
                self.d['jsonld'].append('INVALID')

def image_ok(url):
    try:
        req = urllib.request.Request(url, method='HEAD', headers={'User-Agent': UA})
        with urllib.request.urlopen(req, timeout=10) as r:
            return r.status < 400 and r.headers.get('Content-Type', '').startswith('image/')
    except Exception:
        return False

def check(url):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers={'User-Agent': UA}), timeout=20) as r:
            final, html = r.url, r.read(2_000_000).decode(r.headers.get_content_charset() or 'utf-8', 'replace')
            xrobots = r.headers.get('X-Robots-Tag', '')
    except Exception as e:
        return {'url': url, 'error': f'{type(e).__name__}: {e}'[:120]}
    p = Meta(); p.feed(html); d = p.d
    title, desc, robots = d['title'], d.get('description', ''), (d.get('robots', '') + ' ' + xrobots).lower()
    og_image = urljoin(final, d.get('og:image', '')) if d.get('og:image') else ''
    problems = []
    if not title: problems.append('missing title')
    elif len(title) > 60: problems.append(f'title {len(title)} chars')
    if not desc: problems.append('missing description')
    elif not 70 <= len(desc) <= 160: problems.append(f'description {len(desc)} chars')
    if not d['canonical']: problems.append('no canonical')
    elif len(set(d['canonical'])) > 1: problems.append('multiple canonicals')
    if 'noindex' in robots: problems.append('noindex')
    if len(d['h1']) != 1: problems.append(f"{len(d['h1'])} h1")
    elif not d['h1'][0]: problems.append('empty h1')
    for k in ('og:title', 'og:image'):
        if not d.get(k): problems.append(f'missing {k}')
    if og_image and not image_ok(og_image): problems.append('og:image broken')
    if not d.get('twitter:card'): problems.append('no twitter:card')
    if 'INVALID' in d['jsonld']: problems.append('invalid JSON-LD')
    return {'url': url, 'final_url': final, 'title': title, 'description': desc, 'canonical': ' '.join(d['canonical']),
            'robots': robots.strip(), 'h1': ' | '.join(d['h1'])[:200], 'og_title': d.get('og:title', ''), 'og_image': og_image,
            'twitter_card': d.get('twitter:card', ''), 'jsonld_types': ' '.join(sorted(set(map(str, d['jsonld'])))),
            'problems': '; '.join(problems), 'error': ''}

if __name__ == '__main__':
    urls = [l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')]
    t = time.time()
    with ThreadPoolExecutor(10) as pool:
        rows = list(pool.map(check, urls))
    titles = {}
    for r in rows:
        if r.get('title'): titles.setdefault(r['title'], []).append(r['url'])
    for r in rows:
        if len(titles.get(r.get('title'), [])) > 1: r['problems'] = (r['problems'] + '; duplicate title').lstrip('; ')
    fields = ['url', 'final_url', 'title', 'description', 'canonical', 'robots', 'h1', 'og_title', 'og_image',
              'twitter_card', 'jsonld_types', 'problems', 'error']
    with open('meta_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, fields); w.writeheader(); w.writerows(rows)
    for r in rows:
        flag = 'ERR' if r['error'] else '!!' if r['problems'] else 'ok'
        print(f"{flag:3} {r['url'][:34]:34} {r['error'] or r['problems'] or r['jsonld_types']}"[:150])
    print(f'{len(rows)} URLs in {time.time() - t:.1f} s -> meta_report.csv')

Three choices matter. HTMLParser from the standard library is enough here: meta tags are flat, so there's no need for BeautifulSoup. <title> elements inside inline <svg> icons are ignored, because many sites have several of them and a naive parser picks up "Close" or "Search" as the page title. And Open Graph tags are read from either property= or name=, since real sites use both (MDN uses name="og:title").

Results

$ python3 meta_tags_check.py urls.txt
!!  https://www.python.org             description 52 chars; no canonical; 5 h1; no twitter:card
!!  https://github.com                 title 70 chars; description 186 chars
!!  https://example.com                missing description; no canonical; 0 h1; missing og:title; missing og:image; no twitter:card
!!  https://www.bbc.com/news           title 93 chars; description 249 chars; empty h1; missing og:image; no twitter:card
ok  https://developer.mozilla.org/en-U
!!  https://kubernetes.io              description 559 chars; no canonical
!!  https://www.theguardian.com/intern description 62 chars; 0 h1; missing og:title; missing og:image; no twitter:card
!!  https://stripe.com                 2 h1
ok  https://www.allrecipes.com/recipe/ NewsArticle Recipe
!!  https://docs.python.org/3/         description 34 chars; no twitter:card
ERR https://expired.badssl.com         URLError: <urlopen error [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_
11 URLs in 0.5 s -> meta_report.csv

What this sample shows:

Big sites ship "problems" on purpose. BBC News has a 93-character title and a 249-character description, and the Guardian's international front has no Open Graph tags at all. Treat the length rules as a review list, not a verdict; on your own site they're usually worth fixing.
Multiple <h1>s are common. python.org uses five (one per carousel slide), Stripe two. Fine for users, but on a template you control it's often accidental.
Missing canonicals show up on well-run sites. python.org and kubernetes.io have none on their home pages. Not fatal, but on pages that are reachable through several URLs it's the cheapest duplicate-content fix there is.
Structured data comes along for free. The Allrecipes recipe page declares Recipe and NewsArticle JSON-LD, which is what powers its rich result.

Gotchas I hit

Bots get different HTML. Some sites serve a challenge page or 403 to unknown user agents, and the Facebook or Googlebot user agents get blocked from cloud IPs more often than a browser one. Send a browser User-Agent and treat challenge pages as "unknown", not "missing tags".
Check that og:image actually loads. A tag pointing at a deleted file is worse than no tag, because the preview breaks silently. The script does a HEAD request and expects an image/* content type. Some CDNs reject HEAD; fall back to a ranged GET if you see false alarms.
Relative og:image URLs exist. They're not valid per the Open Graph spec, but they're common; resolve them against the final URL before testing.
Cap the download. Some pages are several MB of inline JS. The script reads 2 MB, which always includes the <head>.
JavaScript-rendered tags won't appear. Single-page apps that inject meta tags client-side look empty to a plain HTTP fetch, which is also roughly what most link-preview bots see. That's a real finding, not a script bug.
Don't point it at internal addresses. If you expose this as a service that takes user input, block private and reserved IPs, or it becomes an SSRF tool.

Running it at scale

For a few hundred URLs the script is enough. If you'd rather not handle redirect tracking, charset detection, hreflang, Google rich-result required properties per schema.org type, microdata and RDFa, bot-challenge detection, Googlebot/Facebook/mobile user agents and a duplicate-title/description summary yourself, I also published it as a cloud tool: Meta Tags & Open Graph Checker on Apify (disclosure: I built it). It can return only the pages with problems, and costs $1 per 1,000 URLs checked; unreachable pages, bot-challenge pages, 4xx/5xx pages and invalid inputs are free.

Either way, run it before a launch and after every template change. A broken link preview is the first thing people see when they share your page.

Bulk redirect checker in Python: 301 vs 302, redirect chains and site-migration QA (2026)

Datawright Labs — Sun, 25 Oct 2026 18:56:46 +0000

Redirects are invisible until they go wrong. A migration ships, someone maps 2,000 old URLs to new ones, and three months later you find out that half of them go through three hops, a few land on the homepage, and one section quietly answers with a 302 that search engines treat as temporary.

curl -sIL https://example.com shows you one chain. This post builds a script that takes a list of URLs (or old -> expected new pairs) and returns, for each one, every hop, the final URL and status, and a plain list of what's wrong.

What counts as a problem

Check	Why it matters
Chain longer than 1 hop	Every hop costs latency, and crawlers give up on long chains. Point the old URL straight at the final one.
`302`, `303` or `307`	Temporary redirects. For a permanent move you want `301` or `308`.
HTTPS to HTTP	A downgrade mid-chain undoes the point of HTTPS.
Loop or more than 10 hops	Browsers stop and show an error.
Final status 4xx/5xx	The redirect works, but it lands on a broken page.
Final page on plain HTTP	Usually a missing HTTPS redirect at the end of the chain.
Final URL is not the expected one	Migration QA: the old URL should land exactly where your redirect map says.

The script

# Usage: python3 redirect_check.py urls.txt
# Bulk redirect checker: follows every redirect by hand, records each hop, and flags SEO problems
# (long chains, temporary redirects, HTTPS->HTTP, loops, broken final pages, wrong destination).
# Lines may be "url" or "old_url -> expected_new_url" for migration QA. Standard library only.
import csv, sys, time, http.client, ssl
from urllib.parse import urlsplit, urljoin
from concurrent.futures import ThreadPoolExecutor

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'
CTX = ssl.create_default_context()
MAX_HOPS = 10

def norm(u):
    return u.rstrip('/').lower()

def hop(url):
    u = urlsplit(url)
    Conn = http.client.HTTPSConnection if u.scheme == 'https' else http.client.HTTPConnection
    kw = {'context': CTX} if u.scheme == 'https' else {}
    c = Conn(u.hostname, u.port, timeout=15, **kw)
    c.request('GET', (u.path or '/') + ('?' + u.query if u.query else ''), headers={'User-Agent': UA})
    r = c.getresponse(); r.read(100_000); c.close()
    return r.status, r.getheader('Location')

def check(line):
    src, _, expected = (p.strip() for p in line.partition('->'))
    url, chain, seen, problems = src, [], set(), []
    try:
        for _ in range(MAX_HOPS + 1):
            if url in seen:
                problems.append('redirect loop'); break
            seen.add(url)
            status, loc = hop(url)
            chain.append((status, url))
            if status in (301, 302, 303, 307, 308) and loc:
                nxt = urljoin(url, loc)
                if url.startswith('https://') and nxt.startswith('http://'):
                    problems.append('HTTPS -> HTTP downgrade')
                if status in (302, 303, 307):
                    problems.append(f'temporary {status}')
                url = nxt
                continue
            break
        else:
            problems.append(f'more than {MAX_HOPS} redirects')
    except Exception as e:
        return {'input': src, 'error': f'{type(e).__name__}: {e}'[:120]}
    hops = len(chain) - 1
    final_status = chain[-1][0]
    if hops > 1: problems.append(f'{hops}-hop chain')
    if final_status >= 400: problems.append(f'final {final_status}')
    if url.startswith('http://'): problems.append('final page on plain HTTP')
    if expected and norm(url) != norm(urljoin(src, expected)):
        problems.append(f'expected {expected}')
    return {'input': src, 'final_url': url, 'final_status': final_status, 'hops': hops,
            'chain': ' > '.join(f'{s} {u}' for s, u in chain),
            'problems': '; '.join(dict.fromkeys(problems))}

if __name__ == '__main__':
    lines = [l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')]
    t = time.time()
    with ThreadPoolExecutor(16) as ex:
        rows = list(ex.map(check, lines))
    cols = ['input', 'final_url', 'final_status', 'hops', 'problems', 'chain', 'error']
    with open('redirect_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, cols); w.writeheader(); w.writerows(rows)
    for r in rows:
        flag = 'ERR' if 'error' in r else ('!!' if r['problems'] else 'ok')
        print(f"{flag:3} {r.get('hops', ''):>2}  {r['input'][:38]:38} {r.get('problems') or r.get('error', '') or r['final_url']}"[:160])
    print(f'{len(lines)} URLs in {time.time() - t:.1f} s -> redirect_report.csv')

Two choices matter. Redirects are followed by hand with http.client instead of letting urllib or requests do it silently, because the hops are the whole point: you want to see 301 > 301 > 200, not only where you ended up. And each Location header is resolved with urljoin, because relative redirects (Location: /get) are common and legal.

Results

The input file mixes plain URLs and migration pairs:

https://github.com
https://bbc.com -> https://www.bbc.com/
https://google.com
https://python.org
https://twitter.com
https://httpbin.org/redirect/3
https://httpbin.org/redirect/12
https://apple.com/ipad -> https://www.apple.com/ipad/
https://mozilla.org/firefox -> https://www.mozilla.org/en-US/firefox/new/
https://wikipedia.org/wiki/Python -> https://en.wikipedia.org/wiki/Python
https://expired.badssl.com

$ python3 redirect_check.py urls.txt
ok   0  https://github.com                     https://github.com
ok   1  https://bbc.com                        https://www.bbc.com/
ok   1  https://google.com                     https://www.google.com/
ok   1  https://python.org                     https://www.python.org/
ok   1  https://twitter.com                    https://x.com/
!!   3  https://httpbin.org/redirect/3         temporary 302; 3-hop chain
!!  10  https://httpbin.org/redirect/12        temporary 302; more than 10 redirects; 10-hop chain
!!   2  https://apple.com/ipad                 2-hop chain
!!   2  https://mozilla.org/firefox            2-hop chain; expected https://www.mozilla.org/en-US/firefox/new/
!!   2  https://wikipedia.org/wiki/Python      2-hop chain
ERR     https://expired.badssl.com             SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: certificate has expired (_s
11 URLs in 1.4 s -> redirect_report.csv

What this sample shows:

Two-hop chains are everywhere. apple.com/ipad goes 301 to www.apple.com/ipad, then 301 again to add the trailing slash. Wikipedia goes apex, then www, then en. Each is harmless for one URL, but on a site you control, pointing every old URL straight at its final address is a free win.
The expected-destination check catches real drift. I expected mozilla.org/firefox to land on www.mozilla.org/en-US/firefox/new/, but Firefox now lives on its own domain (www.firefox.com). In a migration, that's exactly the kind of row you want flagged instead of assuming the map is right.
Temporary redirects show up as such. httpbin's test endpoints use 302, so they're flagged, along with the 12-redirect chain that hits the 10-hop limit.
Clean ones are clean. bbc.com and google.com take one 301 to www, and the BBC row matches its expected destination.

Gotchas I hit

Port 80 can be blocked. My test box times out on plain http:// URLs, so the sample uses https:// inputs. In real use, test the http:// versions too: the HTTP-to-HTTPS redirect is the one most often misconfigured.
Trailing slashes and case. https://www.bbc.com and https://www.bbc.com/ are the same page. The script normalizes both sides before comparing; decide whether your site should treat case as significant.
Bots get different answers. Some sites redirect unknown user agents to a challenge page or answer with 403. Send a browser User-Agent, and also try a Googlebot one if SEO is the reason you're checking.
Not every redirect is a 3xx. <meta http-equiv="refresh"> and JavaScript redirects don't show up in headers. Parse the final HTML if you need them.
Expired certificates throw before the first hop. The script reports them as errors, which is usually what you want.
Don't point it at internal addresses. If you expose this as a service that takes user input, block private and reserved IPs, or it becomes an SSRF tool.

Running it at scale

For a few hundred URLs the script is enough. If you'd rather not handle meta refresh, canonical and noindex checks on the destination, "deep URL dumped on the homepage" detection, Googlebot and mobile user agents and SSRF guards yourself, I also published it as a cloud tool: Redirect Checker on Apify (disclosure: I built it). It takes plain URLs or old -> new pairs (or a JSON redirect map), can return only the problem rows, and costs $0.50 per 1,000 URLs checked; unreachable URLs and invalid inputs are free.

Either way, run it before and after every migration. Redirect mistakes are cheap to fix the day you ship and expensive to find three months later.

Check security headers for a list of websites in Python: HSTS, CSP, X-Frame-Options and an A-F grade (2026)

Datawright Labs — Sun, 25 Oct 2026 00:12:31 +0000

Security headers are the cheapest hardening a website can get: a handful of response headers that tell the browser to stay on HTTPS, refuse to run injected scripts, refuse to be framed by another site and stop guessing content types. They're also easy to forget, and they quietly disappear when someone swaps a CDN, a load balancer or a framework.

Checking one site by hand is fine (curl -sI https://example.com). Checking a client list, every subdomain you own, or a few hundred prospects isn't. This post builds one script that takes a list of URLs and returns, for each one, the final URL after redirects, an A-F grade, a 0-100 score and a plain-English list of what's missing.

What the script checks

Header	Points	What "good" looks like
`Strict-Transport-Security`	25	`max-age` of at least 180 days
`Content-Security-Policy`	25	restricts scripts (`script-src` or `default-src`); minus points for `'unsafe-inline'` without a nonce or hash, and for `'unsafe-eval'`
Clickjacking	15	`X-Frame-Options` or CSP `frame-ancestors`
`X-Content-Type-Options`	15	`nosniff`
`Referrer-Policy`	10	present
`Permissions-Policy`	5	present
Cookies	5	every `Set-Cookie` has `Secure`

A final page on plain HTTP scores zero. 90+ is an A, 75+ a B, 60+ a C, 45+ a D, anything lower an F. The weights are my own judgement, not a standard, so change them to match what you care about.

The script

# Usage: python3 security_headers.py urls.txt
# Bulk security-headers audit: fetches each URL (following redirects by hand), scores HSTS, CSP,
# clickjacking, X-Content-Type-Options, Referrer-Policy and Permissions-Policy, grades A-F,
# and writes headers_report.csv. Standard library only.
import csv, sys, re, time, http.client, ssl
from urllib.parse import urlsplit, urljoin
from concurrent.futures import ThreadPoolExecutor

UA = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36'
CTX = ssl.create_default_context()

def fetch(url, max_hops=10):
    chain = []
    for _ in range(max_hops):
        u = urlsplit(url)
        Conn = http.client.HTTPSConnection if u.scheme == 'https' else http.client.HTTPConnection
        kw = {'context': CTX} if u.scheme == 'https' else {}
        c = Conn(u.hostname, u.port, timeout=15, **kw)
        c.request('GET', (u.path or '/') + ('?' + u.query if u.query else ''), headers={'User-Agent': UA, 'Accept': '*/*'})
        r = c.getresponse(); r.read(200_000); c.close()
        chain.append(f'{r.status} {url}')
        loc = r.getheader('Location')
        if r.status in (301, 302, 303, 307, 308) and loc:
            url = urljoin(url, loc); continue
        # header names are case-insensitive; Set-Cookie can repeat
        h = {k.lower(): v for k, v in r.getheaders()}
        h['set-cookie'] = [v for k, v in r.getheaders() if k.lower() == 'set-cookie']
        return url, r.status, h, chain
    raise RuntimeError('too many redirects')

def grade(final_url, h):
    score, notes = 0, []
    m = re.search(r'max-age=(\d+)', h.get('strict-transport-security', ''))
    if m and int(m.group(1)) >= 180 * 86400: score += 25
    else: notes.append('HSTS missing or max-age < 180 days')
    csp = h.get('content-security-policy', '')
    if 'script-src' in csp or 'default-src' in csp:
        score += 25
        if "'unsafe-inline'" in csp and not re.search(r"'(nonce|sha\d+)-", csp): score -= 8; notes.append("CSP allows 'unsafe-inline'")
        if "'unsafe-eval'" in csp: score -= 4; notes.append("CSP allows 'unsafe-eval'")
    elif csp: score += 5; notes.append('CSP does not restrict scripts')
    else: notes.append('no CSP')
    if h.get('x-frame-options') or 'frame-ancestors' in csp: score += 15
    else: notes.append('no clickjacking protection')
    if h.get('x-content-type-options', '').lower() == 'nosniff': score += 15
    else: notes.append('no X-Content-Type-Options: nosniff')
    if h.get('referrer-policy'): score += 10
    else: notes.append('no Referrer-Policy')
    if h.get('permissions-policy'): score += 5
    else: notes.append('no Permissions-Policy')
    insecure = [c.split('=')[0] for c in h['set-cookie'] if 'secure' not in c.lower()]
    if not insecure: score += 5
    else: notes.append('cookies without Secure: ' + ', '.join(insecure[:3]))
    if not urlsplit(final_url).scheme == 'https': score = 0; notes.append('final page is plain HTTP')
    letter = 'A' if score >= 90 else 'B' if score >= 75 else 'C' if score >= 60 else 'D' if score >= 45 else 'F'
    return score, letter, notes

def check(raw):
    url = raw if '://' in raw else 'https://' + raw
    try:
        final, status, h, chain = fetch(url)
        score, letter, notes = grade(final, h)
        return {'input': raw, 'final_url': final, 'status': status, 'grade': letter, 'score': score,
                'hops': len(chain) - 1, 'server': h.get('server', ''), 'issues': '; '.join(notes)}
    except Exception as e:
        return {'input': raw, 'error': f'{type(e).__name__}: {e}'[:200]}

if __name__ == '__main__':
    urls = list(dict.fromkeys(l.strip() for l in open(sys.argv[1]) if l.strip() and not l.startswith('#')))
    t = time.time()
    with ThreadPoolExecutor(10) as ex: rows = list(ex.map(check, urls))
    cols = ['input', 'final_url', 'status', 'grade', 'score', 'hops', 'server', 'issues', 'error']
    with open('headers_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, cols); w.writeheader(); w.writerows(rows)
    for r in sorted(rows, key=lambda r: -r.get('score', -1)):
        print(f"{r.get('grade', 'ERR'):3} {r.get('score', ''):>3}  {r['input']:22} {r.get('issues') or r.get('error', '')}"[:150])
    print(f'{len(urls)} URLs in {time.time() - t:.1f} s -> headers_report.csv')

Two details matter. First, redirects are followed by hand with http.client instead of letting urllib do it, because the headers you care about are the ones on the final page, and the redirect chain itself (http:// to https:// to www.) is worth seeing. Second, header names are lowercased, since HTTP headers are case-insensitive and servers disagree on capitalization.

Results

$ python3 security_headers.py urls.txt
A   100  bbc.co.uk
A    95  github.com             no Permissions-Policy
A    95  stripe.com             no Permissions-Policy
B    83  mozilla.org            CSP allows 'unsafe-inline'; CSP allows 'unsafe-eval'; no Permissions-Policy
C    73  apify.com              CSP allows 'unsafe-inline'; CSP allows 'unsafe-eval'; no Referrer-Policy; no Permissions-Policy
D    50  php.net                no CSP; no X-Content-Type-Options: nosniff; no Referrer-Policy
D    45  python.org             no CSP; no X-Content-Type-Options: nosniff; no Referrer-Policy; no Permissions-Policy
F    30  wikipedia.org          no CSP; no clickjacking protection; no X-Content-Type-Options: nosniff; no Referrer-Policy; no Permissions-Policy
F     5  example.com            HSTS missing or max-age < 180 days; no CSP; no clickjacking protection; ...
ERR      expired.badssl.com     SSLCertVerificationError: certificate has expired
10 URLs in 0.4 s -> headers_report.csv

A low grade isn't automatically a vulnerability. Wikipedia serves almost no user-specific content to anonymous readers and has its own reasons for its setup, and example.com is a placeholder page. The grade is a checklist, not a pentest. Where it earns its keep is on your own sites and your clients' sites, where a missing CSP or a dropped HSTS header after a migration is a real regression.

The most common gap in this sample is Permissions-Policy, which almost nobody sets. The most common CSP weakness is 'unsafe-inline', which keeps the policy from blocking injected inline scripts. The fix there is nonces or hashes, as the BBC's 'strict-dynamic' policy does.

Gotchas I hit

Expired or invalid certificates throw before you see any headers. The script reports them as errors, which is usually what you want. If you still want to grade the headers, retry with an unverified SSL context and mark the row F.
Redirect loops exist. Cap the hops (10 here) or one misconfigured site hangs a thread.
Content-Security-Policy-Report-Only doesn't enforce anything. It's a testing mode. Don't count it as a CSP; at most give it partial credit.
Check the HTTP-to-HTTPS redirect separately. HSTS only helps after the first HTTPS visit, so request http:// too and confirm it redirects. The script above starts on https:// to keep it short.
Some headers are deprecated. X-XSS-Protection and Expect-CT are ignored by modern browsers, so don't award points for them, and Feature-Policy was renamed Permissions-Policy.
Some sites serve different headers to bots. Send a browser User-Agent or you may be grading a bot-protection page.
Don't point it at internal addresses. If you run this as a service that accepts user input, block private and reserved IPs, or it becomes an SSRF tool.

Running it at scale

For a weekly cron over your own sites the script is enough. If you'd rather not handle the HTTP redirect check, invalid certificates, cookie flags, security.txt and SSRF guards yourself, I also published it as a cloud tool: Security Headers Checker on Apify (disclosure: I built it). It records the full redirect chain with timings, grades sites with broken TLS instead of skipping them, checks security.txt, and can return only sites below a grade you choose. It costs $1 per 1,000 URLs checked; unreachable URLs and invalid inputs are free.

Either way, it takes a few seconds to know which of your sites quietly lost a header, and that's worth a look before someone else finds out.

Bulk WHOIS lookup in Python with RDAP: domain age, expiry, registrar and availability (no scraping, 2026)

Datawright Labs — Sat, 24 Oct 2026 20:09:41 +0000

If you've ever tried to run WHOIS for a few hundred domains, you know the pain: every registry formats its text differently, port 43 servers throttle you, and half the fields are redacted anyway. The good news is that you don't need WHOIS any more. Generic TLD registries have been required to run RDAP (Registration Data Access Protocol) since 2019, and ICANN let them switch off the old WHOIS service in January 2025. RDAP is plain HTTPS returning JSON, with the same structure everywhere.

This post builds one script that takes a messy list of domains and returns, for each one: registrar, creation date, expiry date, domain age, days until expiry, EPP status codes, nameservers, and whether the domain is unregistered. It's useful for lead qualification (older domains are usually real businesses), brand and typo-squat monitoring, expiring-domain lists, and checking your own portfolio.

How RDAP lookup works

Find the right server. IANA publishes a bootstrap file, data.iana.org/rdap/dns.json, that maps each TLD to its registry's RDAP base URL. Download it once per run.
Ask for the domain. GET <base>/domain/example.com returns JSON. A 404 means the name isn't registered, which gives you an availability check for free.
Read the fields. Dates are in events (registration, expiration, last changed), the registrar is the entity with the registrar role (its name is in a jCard fn field), status codes are in status, and nameservers are in nameservers[].ldhName.

The script

# Usage: python3 whois_rdap_bulk.py domains.txt
# Bulk WHOIS replacement: looks up every domain over RDAP (JSON) in parallel and writes whois_report.csv
# with registrar, created/expires dates, domain age, days until expiry, EPP status, nameservers, and availability.
import csv, sys, json, time, datetime as dt, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor

UA = {'User-Agent': 'rdap-bulk-check/1.0 (+python)', 'Accept': 'application/rdap+json'}
MAX_WAIT = 10  # some registries answer 429 with Retry-After of several hours: give up instead of hanging
NOW = dt.datetime.now(dt.timezone.utc)
# Registries that run RDAP but are missing from IANA's bootstrap file.
EXTRA = {'io': 'https://rdap.identitydigital.services/rdap/', 'co': 'https://rdap.registry.co/co/',
         'us': 'https://rdap.nic.us/', 'me': 'https://rdap.identitydigital.services/rdap/'}

def get_json(url, tries=3):
    for i in range(tries):
        try:
            with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=15) as r:
                return 200, json.load(r)
        except urllib.error.HTTPError as e:
            wait = int(e.headers.get('Retry-After') or 2 * (i + 1)) if str(e.headers.get('Retry-After') or '1').isdigit() else 5
            if e.code in (429, 500, 502, 503) and i < tries - 1 and wait <= MAX_WAIT:
                time.sleep(wait); continue
            return e.code, None
        except Exception:
            if i == tries - 1: return 0, None
            time.sleep(1)

def load_bootstrap():
    _, data = get_json('https://data.iana.org/rdap/dns.json')
    servers = {tld: urls[0] for tlds, urls in data['services'] for tld in tlds}
    servers.update({k: v for k, v in EXTRA.items() if k not in servers})
    return servers

def normalize(line):
    d = line.strip().lower()
    for p in ('https://', 'http://'): d = d.removeprefix(p)
    d = d.split('/')[0].split('@')[-1].split(':')[0].removeprefix('www.').rstrip('.')
    return d if '.' in d else None

def parse_date(s):
    return dt.datetime.fromisoformat(s.replace('Z', '+00:00')) if s else None

def lookup(domain, servers):
    row = {'domain': domain}
    base = servers.get(domain.rsplit('.', 1)[-1])
    if not base:
        return {**row, 'result': 'no-rdap'}
    code, j = get_json(base.rstrip('/') + '/domain/' + domain)
    if code == 404:
        return {**row, 'result': 'available'}
    if code != 200 or not j:
        return {**row, 'result': 'rate-limited' if code == 429 else f'error {code}'}
    ev = {e.get('eventAction'): e.get('eventDate') for e in j.get('events', [])}
    created, expires = parse_date(ev.get('registration')), parse_date(ev.get('expiration'))
    registrar = next((e for e in j.get('entities', []) if 'registrar' in e.get('roles', [])), {})
    name = next((v[3] for v in (registrar.get('vcardArray') or [None, []])[1] if v[0] == 'fn'), '')
    return {**row, 'result': 'registered', 'registrar': name,
            'created': created.date() if created else '', 'expires': expires.date() if expires else '',
            'age_years': round((NOW - created).days / 365.25, 1) if created else '',
            'days_left': (expires - NOW).days if expires else '',
            'status': ' '.join(j.get('status', [])),
            'nameservers': ' '.join(sorted(n.get('ldhName', '').lower() for n in j.get('nameservers', [])))}

if __name__ == '__main__':
    domains = list(dict.fromkeys(filter(None, map(normalize, open(sys.argv[1])))))
    servers = load_bootstrap()
    t = time.time()
    with ThreadPoolExecutor(8) as pool:
        rows = list(pool.map(lambda d: lookup(d, servers), domains))
    cols = ['domain', 'result', 'registrar', 'created', 'expires', 'age_years', 'days_left', 'status', 'nameservers']
    with open('whois_report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, cols); w.writeheader(); w.writerows(rows)
    for r in rows:
        print(f"{r['result']:<11} {r['domain']:<26} {str(r.get('registrar',''))[:28]:<28} "
              f"{str(r.get('created','')):<11} {str(r.get('expires','')):<11} {r.get('days_left','')}")
    print(f'{len(rows)} domains in {time.time()-t:.1f} s -> whois_report.csv')

Put one domain per line in domains.txt. The normalize() step means you can paste URLs, www. hosts, email addresses or mailto: links and they still work, and duplicates are dropped.

What a real run returns

I ran it on 13 inputs (a mix of bare domains, a URL, a mailto: address, a duplicate, and one name I made up):

registered  github.com                 MarkMonitor Inc.             2007-10-09  2028-10-09  716
registered  python.org                 Gandi SAS                    1995-03-27  2033-03-28  2346
registered  wikipedia.org              MarkMonitor Inc.             2001-01-13  2027-01-13  80
registered  amazon.com                 MarkMonitor Inc.             1994-11-01  2027-10-31  371
registered  bbc.co.uk                  British Broadcasting Corpora 1994-12-13  2034-12-13  2971
registered  apify.com                  Amazon Registrar, Inc.       2009-06-02  2035-06-02  3142
registered  crawlee.dev                GoDaddy.com, LLC             2022-07-08  2028-07-08  622
registered  google.io                  MarkMonitor Inc.             2002-10-01  2027-09-30  340
no-rdap     notion.so
registered  vercel.com                 Amazon Registrar, Inc.       1999-10-04  2029-10-04  1076
registered  openai.com                 MarkMonitor Inc.             2007-01-19  2029-01-19  818
available   datawright-unregistered-test-8812.com
registered  stackoverflow.com          CSC Corporate Domains, Inc.  2003-12-26  2027-02-02  100
13 domains in 0.5 s -> whois_report.csv

The CSV also has age_years, the EPP status codes (for example client transfer prohibited, which means the domain is locked against transfers) and the full nameserver list, which tells you who hosts the DNS. github.com, for instance, splits its nameservers between NS1 and Amazon Route 53.

A few things stand out. Big brands cluster at MarkMonitor and CSC, which are corporate registrars, so a registrar column is a decent signal for "is this a large company". wikipedia.org and stackoverflow.com were both within about three months of expiry when I ran this; that's normal for domains on auto-renew, but it's exactly the window to watch if you manage domains for clients. And the made-up name came back as a clean 404, so the same script doubles as a bulk availability checker.

Gotchas I hit

A registry rate-limited me for almost five hours. After a handful of test runs, Nominet (the .uk registry) answered with HTTP 429 and Retry-After: 16949. My first version politely slept on Retry-After, so the whole script just hung. Cap the wait (the script uses 10 seconds) and report the row as rate-limited instead. Also keep concurrency per registry low; the thread pool here is fine for a few hundred domains, but for thousands, group by RDAP host and throttle each one.
The IANA bootstrap file is missing some big ccTLDs. .io, .co, .me and .us all run RDAP but aren't in the file, so the script adds them by hand in EXTRA. Others, like .so, .jp, .eu and .es, have no public RDAP at all, so they come back as no-rdap. For those you're back to port-43 WHOIS or the registry's website.
Some ccTLDs leave fields out. .de returns status and nameservers but no dates, and .ch has no expiry date. Don't treat a blank date as an error.
Send a real User-Agent. Some RDAP servers reject Python's default urllib agent, so the script sets one explicitly.
The domain normalizer is naive. It keeps shop.example.com as-is instead of reducing it to example.com, and the registry will return 404 for a subdomain, which looks like "available". For real input, reduce each host to its registrable domain with the Public Suffix List (the tldextract package does this).
Registrant contacts are gone. Since GDPR, almost every registry redacts the owner's name and email in RDAP, same as in WHOIS. You get the registrar and its abuse contact, not the person.

Running it on a schedule or at scale

The script is enough for a weekly cron job over your own domains. If you'd rather not handle the rate limits, the extra registry servers and the subdomain logic yourself, I also published this as a cloud tool: WHOIS Domain Lookup (RDAP) on Apify (disclosure: I built it). It reduces every input to its registrable domain with the Public Suffix List, throttles per registry, adds the registrar's IANA ID and abuse contact, DNSSEC, a DNS-provider guess from the nameservers, and filters like "only expiring soon" or "only available". It costs $1 per 1,000 domains looked up; failed lookups and TLDs without RDAP are free.

Either way, the old WHOIS scraping era is over. RDAP gives you the same data as clean JSON, and a few dozen lines of Python are enough to use it.

Give your AI agent web-data tools with Apify's MCP server (Python, no SDK, 2026)

Datawright Labs — Fri, 23 Oct 2026 23:58:17 +0000

Most AI agents can now talk to remote MCP servers. Apify runs one at https://mcp.apify.com, and any public Actor in Apify Store can be exposed through it as a tool your agent can call: SSL checks, DNS audits, broken-link crawls, screenshots, keyword research and thousands more.

This post shows what actually happens on the wire, with a ~50-line stdlib Python client, so you can see exactly what your agent sees and pays for.

1. Pick the tools with the URL

?tools= takes a comma-separated list of Actors:

https://mcp.apify.com?tools=datawright-labs/ssl-certificate-checker,datawright-labs/email-domain-dns-checker

Auth is your Apify API token as Authorization: Bearer <APIFY_TOKEN> (free account; Actors bill per result on your own account).

2. The client

"""Call Apify Store Actors as MCP tools from plain Python (stdlib only).
Usage: APIFY_TOKEN=... python3 mcp_agent_tools.py
"""
import json, os, urllib.request

TOOLS = "datawright-labs/ssl-certificate-checker,datawright-labs/email-domain-dns-checker"
URL = f"https://mcp.apify.com?tools={TOOLS}"
HEADERS = {
    "Authorization": "Bearer " + os.environ["APIFY_TOKEN"],
    "Content-Type": "application/json",
    "Accept": "application/json, text/event-stream",
}

def rpc(body, session=None):
    h = dict(HEADERS)
    if session:
        h["mcp-session-id"] = session
    req = urllib.request.Request(URL, data=json.dumps(body).encode(), headers=h)
    with urllib.request.urlopen(req, timeout=180) as r:
        text = r.read().decode()
        sid = r.headers.get("mcp-session-id") or session
    # Streamable HTTP may answer as SSE ("data: {...}") or plain JSON
    msgs = [json.loads(l[5:]) for l in text.splitlines() if l.startswith("data:")]
    return (msgs[-1] if msgs else (json.loads(text) if text.strip() else None)), sid

# 1. Handshake
_, sid = rpc({"jsonrpc": "2.0", "id": 1, "method": "initialize",
              "params": {"protocolVersion": "2025-03-26", "capabilities": {},
                         "clientInfo": {"name": "tutorial", "version": "1"}}})
rpc({"jsonrpc": "2.0", "method": "notifications/initialized"}, sid)

# 2. What tools does the agent see?
tools, _ = rpc({"jsonrpc": "2.0", "id": 2, "method": "tools/list"}, sid)
for t in tools["result"]["tools"]:
    print(f"- {t['name']}: {t.get('description', '')[:90]}")

# 3. Call one, exactly like an agent would
name = next(t["name"] for t in tools["result"]["tools"] if "ssl-certificate-checker" in t["name"])
res, _ = rpc({"jsonrpc": "2.0", "id": 3, "method": "tools/call",
              "params": {"name": name, "arguments": {"domains": ["example.com", "github.com", "expired.badssl.com"]}}}, sid)
for c in res["result"]["content"]:
    print(c.get("text", "")[:1500])

# 4. The call returns a run summary; the rows come from get-dataset-items (the agent's next step)
run = json.loads(res["result"]["content"][0]["text"])
ds = run["storages"]["datasets"]["default"]["id"]
items, _ = rpc({"jsonrpc": "2.0", "id": 4, "method": "tools/call",
                "params": {"name": "get-dataset-items",
                           "arguments": {"datasetId": ds, "limit": 20,
                                         "fields": "host,status,daysUntilExpiry,issuerOrganization,problems"}}}, sid)
for c in items["result"]["content"]:
    print(c.get("text", "")[:2000])

3. What the agent sees

tools/list returns one tool per Actor (datawright-labs--ssl-certificate-checker, datawright-labs--email-domain-dns-checker) plus four helpers: get-actor-run, get-dataset-items, get-key-value-store-record, abort-actor-run.

4. The two-step call (the part most guides skip)

Calling the Actor tool runs it and returns a run summary, not the rows: status, run time, item count, the field list and a nextStep hint telling the agent to call get-dataset-items. Good agents follow that hint; if yours stops at "SUCCEEDED", tell it to fetch the dataset items. Passing fields= keeps the tokens down:

{"host":"example.com","status":"ok","daysUntilExpiry":63,"problems":[]}
{"host":"github.com","status":"ok","daysUntilExpiry":37,"problems":[]}
{"host":"expired.badssl.com","status":"expired","daysUntilExpiry":-4212,"problems":["Certificate expired","Not trusted: CERT_HAS_EXPIRED"]}

That whole round trip took about 9 seconds and cost $0.003 (3 hosts at $1 per 1,000).

5. Plug the same URL into your agent

Any client that supports remote Streamable HTTP MCP servers with a header takes the same two values (URL + Bearer header). Two configs we tested end to end for connection and tool discovery:

Hermes (pip install "hermes-agent[mcp]"), ~/.hermes/config.yaml: mcp_servers: { datawright: { url: "...", headers: { Authorization: "Bearer ${APIFY_TOKEN}" } } }, then hermes mcp test datawright.
OpenClaw: openclaw mcp add datawright --no-probe --url "..." --transport streamable-http --header 'Authorization=Bearer ${APIFY_TOKEN}', then openclaw mcp probe datawright (the pre-save probe doesn't expand ${APIFY_TOKEN}, hence --no-probe).

6. Gotchas

Agents only see the top 10 results of Apify's search-actors, so pin the Actors you want with ?tools= instead of letting the agent search.
/sse is gone; use the Streamable HTTP endpoint above.
Keep the token in an env var or your client's secret store, never in a prompt.

Disclosure: the two example Actors are ours (Datawright Labs). Any Store Actor works the same way.

Check SPF, DMARC and DKIM for a list of domains in Python (bulk email DNS audit, 2026)

Datawright Labs — Fri, 23 Oct 2026 14:17:41 +0000

Since Google and Yahoo started enforcing sender requirements, a domain with a broken SPF record or no DMARC policy doesn't just look sloppy: its mail lands in spam or bounces. The annoying part is that every one of these settings is a public DNS record, so you can audit your own domains, your clients' domains, or a whole lead list without logging in anywhere.

This post builds one script that checks the four records that matter for deliverability across a list of domains, then looks at what a real run on well-known domains returns.

What to check

MX. No MX records means the domain can't receive mail (and replies to it bounce).
SPF (the v=spf1 TXT record on the domain). There must be exactly one, it shouldn't end in +all or ?all, and evaluating it must take 10 DNS lookups or fewer. Every include:, a, mx, ptr, exists: and redirect= counts, recursively through each include. Go over 10 and receivers return permerror, which most treat as an SPF fail.
DMARC (TXT at _dmarc.<domain>). p=none only monitors; p=quarantine or p=reject actually protects you. Without rua= you get no aggregate reports, so you're flying blind.
DKIM (TXT at <selector>._domainkey.<domain>). DNS has no way to list selectors, so you can only probe common ones (google, selector1/selector2 for Microsoft 365, k1 for Mailchimp, s1/s2 for SendGrid, and so on). A key with an empty p= is revoked.

Why DNS-over-HTTPS

You could use dnspython, but some networks (corporate, cloud sandboxes, CI runners) intercept or block plain UDP DNS for TXT queries. Google's JSON resolver at dns.google/resolve works anywhere HTTPS does and needs nothing beyond the standard library. Swap in dnspython if your network is fine with UDP; it's faster.

The script

"""Bulk SPF / DMARC / DKIM / MX check for a list of domains, over DNS-over-HTTPS.
Usage: python3 email_dns_check.py example.com github.com ...  (or a file with one domain per line)"""
import sys, re, json, pathlib, urllib.request, urllib.parse
from concurrent.futures import ThreadPoolExecutor

DOH = "https://dns.google/resolve"
SELECTORS = ["default", "google", "selector1", "selector2", "k1", "k2", "s1", "s2",
             "dkim", "mail", "smtp", "mandrill", "mxvault", "everlytickey1"]

def q(name, rtype, strict=False):
    url = DOH + "?" + urllib.parse.urlencode({"name": name, "type": rtype})
    req = urllib.request.Request(url, headers={"Accept": "application/dns-json"})
    for attempt in range(3):  # public DoH endpoints occasionally time out; retry
        try:
            data = json.load(urllib.request.urlopen(req, timeout=10))
            break
        except OSError:
            if attempt == 2:
                raise
    if data.get("Status") == 3:
        if not strict:
            return []
        raise LookupError("NXDOMAIN")
    out = []
    for a in data.get("Answer", []):
        if a["type"] == {"TXT": 16, "MX": 15}[rtype]:
            out.append(a["data"].replace('" "', "").strip('"') if rtype == "TXT" else a["data"])
    return out

def spf_lookups(domain, seen=None, depth=0):
    """Count DNS-querying SPF mechanisms recursively (RFC 7208 limit: 10)."""
    seen = seen or set()
    if domain in seen or depth > 10:
        return 0
    seen.add(domain)
    recs = [t for t in q(domain, "TXT") if t.lower().startswith("v=spf1")]
    if not recs:
        return 0
    n = 0
    for term in recs[0].split()[1:]:
        t = term.lstrip("+-~?").lower()
        if t.startswith(("include:", "redirect=")):
            n += 1 + spf_lookups(re.split("[:=]", t, maxsplit=1)[1], seen, depth + 1)
        elif t.startswith(("a", "mx", "ptr", "exists:")) and not t.startswith("all"):
            n += 1
    return n

def check(domain):
    r = {"domain": domain, "problems": []}
    try:
        mx = q(domain, "MX", strict=True)
    except LookupError:
        return {"domain": domain, "problems": ["domain does not exist"]}
    r["mx"] = sorted(mx, key=lambda m: int(m.split()[0]))
    if not mx:
        r["problems"].append("no MX records")
    spf = [t for t in q(domain, "TXT") if t.lower().startswith("v=spf1")]
    r["spf"] = spf[0] if spf else None
    if not spf:
        r["problems"].append("no SPF record")
    elif len(spf) > 1:
        r["problems"].append("multiple SPF records (permerror)")
    else:
        r["spf_lookups"] = spf_lookups(domain)
        if r["spf_lookups"] > 10:
            r["problems"].append(f"SPF needs {r['spf_lookups']} DNS lookups (limit 10)")
        if re.search(r"(^|\s)[+?]?all\b", spf[0]) and "-all" not in spf[0] and "~all" not in spf[0]:
            r["problems"].append("SPF ends in +all/?all (allows anyone)")
    dmarc = [t for t in q("_dmarc." + domain, "TXT") if t.lower().startswith("v=dmarc1")]
    r["dmarc"] = dmarc[0] if dmarc else None
    if not dmarc:
        r["problems"].append("no DMARC record")
    else:
        tags = dict(p.strip().split("=", 1) for p in dmarc[0].split(";") if "=" in p)
        r["dmarc_policy"] = tags.get("p")
        if tags.get("p") == "none":
            r["problems"].append("DMARC p=none (monitoring only)")
        if "rua" not in tags:
            r["problems"].append("DMARC has no rua (no aggregate reports)")
    r["dkim_selectors"] = [s for s in SELECTORS
                           if any(re.search(r"\bp=[A-Za-z0-9+/]", t) for t in q(f"{s}._domainkey.{domain}", "TXT"))]
    if not r["dkim_selectors"]:
        r["problems"].append("no DKIM key on common selectors (may use a custom one)")
    if r["spf"] and r["spf"].split() == ["v=spf1", "-all"] and r.get("dmarc_policy") == "reject":
        r["problems"] = []  # a locked-down domain that sends no mail: nothing to fix
        r["dkim_selectors"] = ["(non-mail domain)"]
    return r

if __name__ == "__main__":
    args = sys.argv[1:]
    if len(args) == 1 and pathlib.Path(args[0]).is_file():
        args = pathlib.Path(args[0]).read_text().split()
    domains = list(dict.fromkeys(a.lower().strip() for a in args))
    with ThreadPoolExecutor(8) as pool:
        for r in pool.map(check, domains):
            flag = "OK  " if not r["problems"] else "WARN"
            print(f"{flag} {r['domain']:<22} SPF lookups={r.get('spf_lookups','-'):<3} "
                  f"DMARC p={r.get('dmarc_policy','-'):<11} DKIM={','.join(r.get('dkim_selectors',[])) or '-'}")
            for p in r["problems"]:
                print("       -", p)

Run it with domains as arguments or a file with one per line:

python3 email_dns_check.py github.com apify.com stripe.com shopify.com fastmail.com \
  proton.me example.com mailchimp.com notion.so this-domain-does-not-exist-xyz123.com

What a real run returned

OK   github.com             SPF lookups=10  DMARC p=quarantine  DKIM=google,selector1,k1,k2,s1,s2
OK   apify.com              SPF lookups=10  DMARC p=reject      DKIM=google
OK   stripe.com             SPF lookups=7   DMARC p=reject      DKIM=google,s1,s2,mandrill
OK   shopify.com            SPF lookups=4   DMARC p=reject      DKIM=google,mail
WARN fastmail.com           SPF lookups=1   DMARC p=none        DKIM=-
       - DMARC p=none (monitoring only)
       - no DKIM key on common selectors (may use a custom one)
WARN proton.me              SPF lookups=2   DMARC p=quarantine  DKIM=-
       - DMARC has no rua (no aggregate reports)
       - no DKIM key on common selectors (may use a custom one)
OK   example.com            SPF lookups=0   DMARC p=reject      DKIM=(non-mail domain)
OK   mailchimp.com          SPF lookups=7   DMARC p=reject      DKIM=google,k1,k2,mandrill
WARN notion.so              SPF lookups=0   DMARC p=quarantine  DKIM=-
       - no MX records
       - no DKIM key on common selectors (may use a custom one)
WARN this-domain-does-not-exist-xyz123.com SPF lookups=-   DMARC p=-           DKIM=-
       - domain does not exist

Ten domains took about 6 seconds with 8 threads, almost all of it network wait. A few things worth noticing:

github.com and apify.com sit exactly at 10 SPF lookups. That's legal, but one more include: (a new email tool, a new CRM) tips them into permerror. This is the most common way a working setup breaks silently.
"No DKIM found" is not always a problem. fastmail.com and proton.me sign their mail; they just use selectors you can't guess. Treat that line as "verify by looking at a real message's DKIM-Signature: s= header", not as an error.
fastmail.com publishes p=none. Plenty of big domains still only monitor. For your own domains, the usual path is p=none with rua until reports look clean, then quarantine, then reject.
example.com is a locked-down non-mail domain (v=spf1 -all plus p=reject, and a revoked wildcard DKIM key). That's the right setup for any domain you own but never send from, because it stops spoofing.
notion.so has no MX. Its mail runs on another domain, which is fine, but on a lead list "no MX" usually means the address will bounce.

Gotchas I hit

Public DoH resolvers time out now and then. My third run failed on a single DKIM query until I added the 3-attempt retry in q().
Probing selectors hits NXDOMAIN constantly, which is normal and must not be treated as "domain doesn't exist". Only the MX lookup on the domain itself is strict.
Macros in SPF (exists:%{i}._spf.example.com) can't be resolved without a sending IP. Count them as a lookup and move on, which is what receivers effectively do for the limit.
TXT records over 255 characters come back as several quoted strings. Join them ('" "' replaced with nothing) before parsing, or long SPF and DKIM records look broken.
DMARC on a subdomain falls back to the organizational domain. The script checks only the exact domain; for subdomains, also look up _dmarc on the parent.

Running it on a schedule or at scale

The script is enough for a weekly cron job over your own domains. If you'd rather not host it, or you need to audit thousands of domains from a list, I also published this as a cloud tool: SPF, DMARC & DKIM Checker on Apify (disclosure: I built it). It adds the organizational-domain DMARC fallback, 19 DKIM selectors with key size, BIMI, MTA-STS and TLS-RPT, a 0-100 score per domain, and costs $1 per 1,000 domains checked. Domains that don't exist are free.

Either way, check the records before your customers' spam folders do it for you.

Check SSL certificate and domain expiry for a list of domains in Python (no WHOIS scraping, 2026)

Datawright Labs — Thu, 22 Oct 2026 02:15:08 +0000

Two things quietly take a website down: a TLS certificate that expires, and a domain registration that lapses. Auto-renewal makes both rarer, which is exactly why they still bite. The renewal fails once, nobody is watching, and you find out from customers.

You don't need a monitoring subscription to catch either. Python's standard library can read a server's certificate, and RDAP (the JSON successor to WHOIS that registries now have to run) gives you the domain expiry date without scraping WHOIS text. This post builds one script that does both for a whole list of hosts, then looks at what a real run returns.

The approach

Normalize the input. Accept bare domains, full URLs and host:port, and drop duplicates.
Open a verified TLS connection with ssl.create_default_context(). If the handshake fails verification, the exception already tells you why: expired, self-signed, untrusted chain or hostname mismatch. You get that diagnosis for free.
Read the certificate with getpeercert() and compute days until notAfter.
Look up domain expiry over RDAP. IANA publishes a bootstrap file that maps each TLD to its registry's RDAP server. Load it once, then request /domain/<name> and read the expiration event.
Run it in parallel with a thread pool, since nearly all the time is network wait.

The script

# Usage: python3 ssl_check.py domains.txt [warn_days]
# Checks the TLS certificate of every host (in parallel) plus domain expiry via RDAP, writes ssl_report.csv.
import csv, socket, ssl, sys, json, datetime as dt, urllib.request
from concurrent.futures import ThreadPoolExecutor

WARN = int(sys.argv[2]) if len(sys.argv) > 2 else 30
NOW = dt.datetime.now(dt.timezone.utc)

def parse_host(line):
    s = line.strip().split('//')[-1].split('/')[0]
    host, _, port = s.partition(':')
    return host.lower(), int(port or 443)

def cert_info(host, port):
    row = {'host': host, 'port': port, 'status': 'ok', 'problems': ''}
    ctx = ssl.create_default_context()
    try:
        with socket.create_connection((host, port), timeout=8) as sock:
            with ctx.wrap_socket(sock, server_hostname=host) as tls:
                c = tls.getpeercert()
                row['tls'] = tls.version()
    except ssl.SSLCertVerificationError as e:
        row.update(status='invalid', problems=e.verify_message)
        return row
    except Exception as e:
        row.update(status='error', problems=type(e).__name__)
        return row
    end = dt.datetime.fromtimestamp(ssl.cert_time_to_seconds(c['notAfter']), dt.timezone.utc)
    days = (end - NOW).days
    issuer = dict(x[0] for x in c['issuer'])
    row.update(valid_to=end.date().isoformat(), days_left=days,
               issuer=issuer.get('organizationName', issuer.get('commonName')),
               sans=' '.join(v for k, v in c.get('subjectAltName', []) if k == 'DNS'))
    if days < WARN:
        row.update(status='expiring', problems=f'Certificate expires in {days} days')
    return row

RDAP = {}
def rdap_base(tld):
    if not RDAP:
        boot = json.load(urllib.request.urlopen('https://data.iana.org/rdap/dns.json', timeout=20))
        for tlds, urls in boot['services']:
            for t in tlds:
                RDAP[t] = urls[0]
    return RDAP.get(tld)

def domain_expiry(host):
    parts = host.split('.')
    domain = '.'.join(parts[-2:])   # naive: fine for .com/.org/.io, not for .co.uk
    base = rdap_base(parts[-1])
    if not base:
        return {}
    try:
        req = urllib.request.Request(base.rstrip('/') + '/domain/' + domain, headers={'Accept': 'application/rdap+json', 'User-Agent': 'ssl-check/1.0'})
        d = json.load(urllib.request.urlopen(req, timeout=15))
    except Exception:
        return {}
    for ev in d.get('events', []):
        if ev.get('eventAction') == 'expiration':
            exp = dt.datetime.fromisoformat(ev['eventDate'].replace('Z', '+00:00'))
            return {'domain_expires': exp.date().isoformat(), 'domain_days_left': (exp - NOW).days}
    return {}

def check(line):
    host, port = parse_host(line)
    row = cert_info(host, port)
    row.update(domain_expiry(host))
    return row

hosts = list(dict.fromkeys(l.strip() for l in open(sys.argv[1]) if l.strip()))
rdap_base('com')  # load the IANA bootstrap file once, before the threads start
with ThreadPoolExecutor(20) as pool:
    rows = list(pool.map(check, hosts))
rows.sort(key=lambda r: (r['status'] == 'ok', r.get('days_left', 9999)))
cols = ['host', 'port', 'status', 'days_left', 'valid_to', 'issuer', 'tls', 'domain_expires', 'domain_days_left', 'problems', 'sans']
with open('ssl_report.csv', 'w', newline='') as f:
    w = csv.DictWriter(f, cols, extrasaction='ignore'); w.writeheader(); w.writerows(rows)
for r in rows:
    print(f"{r['status']:9} {r['host']:28} cert {str(r.get('days_left', '-')):>4} d  domain {str(r.get('domain_days_left', '-')):>5} d  {r['problems']}")

Save your hosts one per line in domains.txt and run python3 ssl_check.py domains.txt 30. The second argument is the warning window in days.

A real run

I ran it on 11 hosts: seven normal sites plus four of the deliberately broken test hosts from badssl.com. It finished in 0.6 seconds, output sorted so problems come first (October 22, 2026):

invalid   expired.badssl.com           cert    - d  domain   167 d  certificate has expired
invalid   self-signed.badssl.com       cert    - d  domain   167 d  self-signed certificate
invalid   wrong.host.badssl.com        cert    - d  domain   167 d  Hostname mismatch, certificate is not valid for 'wrong.host.badssl.com'.
invalid   untrusted-root.badssl.com    cert    - d  domain   167 d  self-signed certificate in certificate chain
ok        github.com                   cert   38 d  domain   718 d
ok        example.com                  cert   64 d  domain   295 d
ok        www.wikipedia.org            cert   72 d  domain    82 d
ok        stackoverflow.com            cert   81 d  domain   103 d
ok        apify.com                    cert   86 d  domain  3145 d
ok        python.org                   cert  115 d  domain  2349 d
ok        crawlee.dev                  cert  179 d  domain   625 d

A few things worth noticing:

The verification error is the diagnosis. SSLCertVerificationError.verify_message says "certificate has expired" or "Hostname mismatch" directly, so you don't have to re-implement chain or hostname checks.
Short certificate lifetimes are normal now. GitHub's certificate had 38 days left and that's fine; CAs are moving toward much shorter validity periods. A 30-day warning window is reasonable today, but expect to tighten it as lifetimes shrink.
Domain expiry can be closer than the certificate's. wikipedia.org's registration had 82 days left, and stackoverflow.com's 103, both sooner than some certificates on the list. Checking only certificates would miss that.

Gotchas I hit

Some RDAP servers reject Python's default User-Agent. My first run got no data for .org domains with urllib, while curl worked fine. Sending any explicit User-Agent header fixed it.
Old TLS versions show up as errors, not warnings. Modern OpenSSL builds refuse TLS 1.0 and 1.1 handshakes, so a server that only speaks those fails to connect. If you need to report the protocol instead of failing, use a separate context with a lower minimum_version.
The registrable-domain logic is naive. Taking the last two labels works for .com, .org and .dev, but turns shop.example.co.uk into co.uk. Use the tldextract package (Public Suffix List) for real input.
Non-standard ports may be blocked by your network. A test host on port 1011 timed out from my machine, which the script reports as error, not invalid.
RDAP isn't universal for country-code TLDs. Many ccTLDs aren't in the IANA bootstrap file yet; the script just leaves domain expiry blank for those.

Running it on a schedule

The script is enough for a cron job that emails you ssl_report.csv when anything isn't ok. If you'd rather not host that, I also published this as a cloud tool: SSL Certificate Checker on Apify (disclosure: I built it). It does the same checks plus SANs, chain, key type, cipher and handshake time, runs on a daily schedule with Slack or email alerts, and costs $1 per 1,000 hosts checked. Unreachable hosts are free.

Either way, the point is to have something look at the dates before they matter.

Bulk website screenshots in Python with Playwright: full-page, mobile, PDF and no cookie banners (2026)

Datawright Labs — Wed, 21 Oct 2026 18:13:41 +0000

Taking one screenshot with Playwright is a two-liner. Taking a few hundred that actually look right is where it gets fiddly: cookie banners cover half the page, lazy-loaded images come out blank, news homepages produce 60,000-pixel PNGs, and some sites never reach "network idle".

Here's the script I ended up with, plus what happened when I ran it on six sites on October 21, 2026 (example.com, python.org, crawlee.dev, MDN, BBC and the Guardian).

Setup

pip install playwright
playwright install chromium   # or pass --chrome /usr/bin/google-chrome to reuse an installed Chrome

The script

"""Bulk website screenshots with Playwright: viewport, full-page, mobile and PDF, with cookie banners hidden.
Usage: python3 screenshots.py urls.txt out/ [--mobile] [--full] [--pdf] [--workers 4]
"""
import argparse, asyncio, csv, pathlib, re, time
from playwright.async_api import async_playwright

HIDE_CSS = """
#onetrust-banner-sdk, #onetrust-consent-sdk, #CybotCookiebotDialog, .cc-window, .cookie-banner,
#cookie-banner, #cookie-notice, .fc-consent-root, #truste-consent-track, .qc-cmp2-container,
[id^="sp_message_container"], .sp-message-open #sp_message_iframe, #didomi-host, #usercentrics-root,
[id*="cookie" i][class*="banner" i], [class*="cookie-consent" i], [aria-label*="cookie" i][role="dialog"]
{ display: none !important; }
html, body, html.sp-message-open { overflow: auto !important; position: static !important; }
"""

def slug(url):
    return re.sub(r'[^a-zA-Z0-9]+', '-', url.split('://', 1)[-1]).strip('-')[:80]

async def shoot(browser, url, out, a):
    ctx_args = dict(viewport={'width': 1280, 'height': 800}, device_scale_factor=1)
    if a.mobile:
        ctx_args = dict(viewport={'width': 390, 'height': 844}, device_scale_factor=3, is_mobile=True, has_touch=True,
                        user_agent='Mozilla/5.0 (iPhone; CPU iPhone OS 17_0 like Mac OS X) AppleWebKit/605.1.15 '
                                   '(KHTML, like Gecko) Version/17.0 Mobile/15E148 Safari/604.1')
    ctx = await browser.new_context(**ctx_args)
    page = await ctx.new_page()
    t0 = time.time()
    try:
        resp = await page.goto(url, wait_until='domcontentloaded', timeout=30000)
        try:
            await page.wait_for_load_state('networkidle', timeout=8000)
        except Exception:
            pass  # sites with analytics beacons never go idle
        await page.add_style_tag(content=HIDE_CSS)
        # trigger lazy-loaded images before a full-page capture
        if a.full:
            await page.evaluate("""async () => { for (let y = 0; y < document.body.scrollHeight; y += 600) {
                window.scrollTo(0, y); await new Promise(r => setTimeout(r, 100)); } window.scrollTo(0, 0); }""")
            await page.wait_for_timeout(500)
        path = out / (slug(url) + ('.pdf' if a.pdf else '.png'))
        if a.pdf:
            await page.emulate_media(media='screen')
            await page.pdf(path=str(path), format='A4', print_background=True)
        else:
            clip = None
            if a.full:  # cap very long pages (news homepages can run 60,000+ px)
                h = await page.evaluate('Math.max(document.body.scrollHeight, document.documentElement.scrollHeight)')
                if h > a.max_height:
                    clip = dict(x=0, y=0, width=page.viewport_size['width'], height=a.max_height)
            await page.screenshot(path=str(path), full_page=a.full, clip=clip)
        return dict(url=url, status=resp.status if resp else '', file=path.name,
                    bytes=path.stat().st_size, seconds=round(time.time() - t0, 1), error='')
    except Exception as e:
        return dict(url=url, status='', file='', bytes=0, seconds=round(time.time() - t0, 1), error=str(e)[:120])
    finally:
        await ctx.close()

async def main():
    p = argparse.ArgumentParser()
    p.add_argument('urls'); p.add_argument('out')
    p.add_argument('--mobile', action='store_true'); p.add_argument('--full', action='store_true')
    p.add_argument('--pdf', action='store_true'); p.add_argument('--max-height', type=int, default=15000); p.add_argument('--workers', type=int, default=4)
    p.add_argument('--chrome', default=None, help='path to an installed Chrome (else Playwright Chromium)')
    a = p.parse_args()
    urls = [u.strip() for u in open(a.urls) if u.strip() and not u.startswith('#')]
    out = pathlib.Path(a.out); out.mkdir(parents=True, exist_ok=True)
    sem = asyncio.Semaphore(a.workers)
    async with async_playwright() as pw:
        browser = await pw.chromium.launch(executable_path=a.chrome)
        async def one(u):
            async with sem:
                return await shoot(browser, u, out, a)
        t0 = time.time()
        rows = await asyncio.gather(*(one(u) for u in urls))
        await browser.close()
    with open(out / 'report.csv', 'w', newline='') as f:
        w = csv.DictWriter(f, fieldnames=list(rows[0])); w.writeheader(); w.writerows(rows)
    ok = sum(1 for r in rows if not r['error'])
    print(f"{ok}/{len(rows)} captured in {time.time() - t0:.1f}s -> {out}/report.csv")
    for r in rows: print(r)

if __name__ == '__main__':
    asyncio.run(main())

Run it with a text file of URLs, one per line:

python3 screenshots.py urls.txt shots/                  # 1280x800 desktop viewport
python3 screenshots.py urls.txt shots/ --mobile --full  # iPhone viewport, whole page
python3 screenshots.py urls.txt shots/ --pdf            # A4 PDF with backgrounds

Every run also writes report.csv with the HTTP status, file size, seconds per page and any error, so failures don't disappear silently.

What I learned testing it

1. Generic cookie selectors miss the big consent platforms. My first version hid the usual suspects (OneTrust, Cookiebot, .cookie-banner), and the BBC came out clean. The Guardian's screenshot was 80% consent wall. It uses Sourcepoint, which injects #sp_message_container_* with an iframe and also locks scrolling with an sp-message-open class on <html>. Hiding the container and resetting overflow/position on html fixed it. If a site still shows a banner, open DevTools, find the outermost fixed-position element and add its selector to HIDE_CSS. Hiding with CSS is safer than clicking "Accept": you don't consent on anyone's behalf, and it doesn't break when the button text is translated.

2. Don't rely on networkidle alone. News sites keep analytics and ad beacons firing forever, so wait_until="networkidle" can hang until the timeout. The script loads on domcontentloaded, then gives networkidle up to 8 seconds and moves on regardless.

3. Scroll before a full-page capture. Lazy-loaded images only load when they enter the viewport. A quick scroll from top to bottom in 600 px steps before the screenshot fills them in.

4. Cap the height. The BBC homepage on an iPhone viewport at 3x scale produced a 1170 x 60,114 px PNG, 22 MB, and Pillow refused to open it without a "decompression bomb" warning. With --max-height 15000 (CSS pixels), it's 1170 x 45,000 and 16 MB. Measure with Math.max(document.body.scrollHeight, document.documentElement.scrollHeight): on the Guardian, the <html> height alone was too small and the cap didn't kick in until I checked both.

5. PDFs of busy pages are huge. The BBC homepage as an A4 PDF was 40.7 MB and took 34 seconds; the Guardian was 0.5 MB in 2 seconds. Use emulate_media("screen") so you get the page as people see it instead of its print stylesheet.

Timings (4 parallel contexts, one shared browser):

Mode	6 sites	Slowest page
Desktop viewport PNG	6.3 s	BBC, 4.7 s
Mobile full-page PNG	18.7 s	Guardian, 14.1 s
A4 PDF	35.8 s	BBC, 34.4 s

Reusing one browser and opening a fresh context per URL is the main speed trick: launching Chrome costs more than most screenshots.

Scaling it up

This works well for dozens or a few hundred URLs from your laptop. Past that you start dealing with memory (each full-page capture of a long page holds the whole bitmap), retries, IP blocks and somewhere to store thousands of files.

If you'd rather not run the browsers yourself, we packaged the same approach (full-page, mobile, PDF, cookie-banner hiding) as the Website Screenshot API on Apify. Disclosure: it's ours. It costs $1.50 per 1,000 viewport screenshots and $5 per 1,000 full-page captures or PDFs, and Apify's free $5 monthly credit covers about 3,300 viewport shots. You can call it over REST or as an MCP tool from an AI agent.

Either way, the five lessons above are the part that saves the most time.

App Store top charts in Python: Apple's free RSS feeds, genres, countries and ratings (2026)

Datawright Labs — Wed, 21 Oct 2026 16:19:08 +0000

If you want to know which apps are climbing the App Store, you don't need a paid app-intelligence subscription to get started. Apple publishes the top charts as free, keyless JSON feeds, and the iTunes Lookup API fills in ratings, versions and update dates. The catch is that there are two feed systems, they don't cover the same charts, and neither is well documented.

This post covers what each feed returns, the limits I ran into when testing on October 21, 2026, and a script that writes daily snapshots to a CSV you can diff over time.

The two feeds

1. The legacy iTunes RSS feed (still live):

https://itunes.apple.com/{country}/rss/topfreeapplications/limit=100/genre=6015/json

Charts: topfreeapplications, toppaidapplications, topgrossingapplications, and the iPad versions (topfreeipadapplications, toppaidipadapplications, topgrossingipadapplications).
Genre filter: yes, with /genre={id}. 6015 is Finance, 6014 Games, 6007 Productivity, 7001-7019 are the game subgenres.
Fields: name, developer, category, price, release date, icon and store URL.
Max 100 entries. I asked for limit=200 and got 100 back.

2. The newer marketing-tools feed:

https://rss.marketingtools.apple.com/api/v2/{country}/apps/top-free/100/apps.json

Cleaner JSON (feed.results[] with id, name, artistName, genres, releaseDate) and an updated timestamp.
Only top-free and top-paid. top-grossing returned a 404 when I tested it, and there's no genre filter.

So if you care about top grossing (usually the most interesting chart, because it's a revenue proxy) or about one category, the legacy feed is the one that works.

Adding ratings and versions

Neither feed includes ratings. The iTunes Lookup API does, and it accepts many ids at once:

https://itunes.apple.com/lookup?id=407558537,6448311069&country=us&entity=software

You get averageUserRating, userRatingCount, version, currentVersionReleaseDate, file size, content rating and more. Pass the same country as the chart, because ratings are per storefront.

The script

Standard library only. It pulls each country and chart combination, enriches each entry with Lookup data, and appends rows to a CSV with the date, so running it daily (cron, GitHub Actions) builds a rank history.

"""App Store top charts in Python: Apple's public RSS feeds + iTunes Lookup.
Usage: python3 app_store_charts.py --countries us,gb,ca --charts top-free,top-grossing --genre 6015 --out charts.csv
Append each day's run to the same CSV and diff ranks between dates to track movers."""
import argparse, csv, datetime, json, time, urllib.request

FEEDS = {
    'top-free': 'topfreeapplications', 'top-paid': 'toppaidapplications', 'top-grossing': 'topgrossingapplications',
    'top-free-ipad': 'topfreeipadapplications', 'top-paid-ipad': 'toppaidipadapplications',
    'top-grossing-ipad': 'topgrossingipadapplications',
}

def get_json(url, tries=4):
    for i in range(tries):
        try:
            req = urllib.request.Request(url, headers={'User-Agent': 'Mozilla/5.0 (chart-tracker)'})
            with urllib.request.urlopen(req, timeout=30) as r:
                return json.load(r)
        except Exception as e:
            if i == tries - 1:
                raise
            time.sleep(2 ** i)

def chart(country, name, genre=None, limit=100):
    url = f'https://itunes.apple.com/{country}/rss/{FEEDS[name]}/limit={limit}'
    if genre:
        url += f'/genre={genre}'
    entries = get_json(url + '/json').get('feed', {}).get('entry', [])
    if isinstance(entries, dict):  # a 1-entry feed is an object, not a list
        entries = [entries]
    rows = []
    for rank, e in enumerate(entries, 1):
        rows.append({
            'rank': rank,
            'app_id': e['id']['attributes']['im:id'],
            'name': e['im:name']['label'],
            'developer': e['im:artist']['label'],
            'category': e['category']['attributes']['label'],
            'price': e['im:price']['attributes']['amount'],
            'release_date': e.get('im:releaseDate', {}).get('label', '')[:10],
        })
    return rows

def lookup(ids, country):
    """Ratings, version and size for up to ~150 app ids per call."""
    out = {}
    for i in range(0, len(ids), 150):
        url = f'https://itunes.apple.com/lookup?id={",".join(ids[i:i+150])}&country={country}&entity=software'
        for r in get_json(url).get('results', []):
            out[str(r['trackId'])] = r
    return out

def main():
    ap = argparse.ArgumentParser()
    ap.add_argument('--countries', default='us')
    ap.add_argument('--charts', default='top-free')
    ap.add_argument('--genre', type=int)
    ap.add_argument('--out', default='charts.csv')
    a = ap.parse_args()
    today = datetime.date.today().isoformat()
    fields = ['date', 'country', 'chart', 'genre', 'rank', 'app_id', 'name', 'developer', 'category',
              'price', 'release_date', 'rating', 'rating_count', 'version', 'last_updated']
    n = 0
    with open(a.out, 'a', newline='') as f:
        w = csv.DictWriter(f, fieldnames=fields)
        if f.tell() == 0:
            w.writeheader()
        for c in a.countries.split(','):
            for ch in a.charts.split(','):
                rows = chart(c, ch, a.genre)
                meta = lookup([r['app_id'] for r in rows], c)
                for r in rows:
                    m = meta.get(r['app_id'], {})
                    w.writerow({**r, 'date': today, 'country': c, 'chart': ch, 'genre': a.genre or '',
                                'rating': round(m.get('averageUserRating', 0), 2), 'rating_count': m.get('userRatingCount', ''),
                                'version': m.get('version', ''), 'last_updated': m.get('currentVersionReleaseDate', '')[:10]})
                n += len(rows)
                print(f'{c} {ch}: {len(rows)} apps, {len(meta)} with lookup data')
    print(f'wrote {n} rows to {a.out}')

if __name__ == '__main__':
    main()

Test run on October 21, 2026: US, UK and Canada, top free plus top grossing, took 16 seconds for 600 rows, and every app had Lookup data. The US Finance top free chart (--genre 6015) took 2.5 seconds: Capital One, Kalshi, the Treasury's Trump Accounts app, PayPal and Cash App were the top five.

Things that will bite you

A one-entry feed is an object, not a list. Small genre charts in small storefronts can return a single entry dict. The script normalizes it.
Charts update a few times a day. Pick a fixed time for your snapshot, or you'll be comparing a morning chart with an evening one.
Rank is position in the list. The feed has no rank field, so count it yourself, and don't sort the entries before you do.
Country codes are storefronts, not languages. gb is the UK, not uk. Names and prices come back localized.
Apple deprecates things quietly. The legacy feed has outlived several "replacements" already, but wrap your calls in retries and alert on empty charts, so a silent change doesn't leave gaps in your history.

To track movers, load two dates from the CSV into pandas, merge on country, chart and app_id, and subtract the ranks. Apps present today but not yesterday are your new entries.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: App Store Top Charts Scraper. It covers all six iPhone and iPad charts, any country and genre, adds the rank change since the previous run, and can run on a schedule. It costs $1 per 1,000 chart entries ($0.10 for a top-100 chart), so Apify's free monthly credit covers daily tracking of a few charts. The script above does the core job too, so use whichever fits.

How to check Core Web Vitals for 1,000 URLs in Python (PageSpeed Insights API, CrUX field data, quotas and retries)

Datawright Labs — Wed, 21 Oct 2026 14:20:31 +0000

Checking one page in PageSpeed Insights is easy. Checking every template, landing page and product page on a site, every week, is where people end up with a spreadsheet of copy-pasted scores and no idea which numbers they can trust.

This post covers what the PageSpeed Insights (PSI) API actually returns, the difference between lab and field data (and which one Google uses for Core Web Vitals), the quota rules that will stop you halfway through a list, and a script that handles all of it and writes a CSV.

Lab data vs field data

One runPagespeed call gives you two very different things:

Lab data (lighthouseResult): Google loads the page once in Lighthouse on a throttled, emulated phone (or desktop) and measures it. You get the 0-100 performance score, LCP, CLS, TBT, FCP and Speed Index. It's repeatable-ish and works for any URL, including pages nobody visits.
Field data (loadingExperience for the URL, originLoadingExperience for the whole origin): the 75th percentile of what real Chrome users experienced over the last 28 days, from the Chrome UX Report (CrUX). This is where INP lives, because interaction latency needs real users clicking things. Lab data has no INP; TBT is the lab stand-in.

The Core Web Vitals assessment (the "passed/failed" you see in Search Console) is based on field data: LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1 at p75. The lab performance score is a diagnostic tool, not a ranking signal. If you only have budget for one column in your report, make it the field one.

Three details that trip people up:

Low-traffic URLs have no page-level field data. loadingExperience comes back without metrics, and PSI may fall back to origin-level data. Record which one you got, or you'll compare a page against a whole site without knowing it.
CrUX sends CLS multiplied by 100. A CUMULATIVE_LAYOUT_SHIFT_SCORE percentile of 5 means CLS 0.05.
Google says it plans to remove CrUX data from the PSI API. The PSI getting-started page currently says: "We plan to discontinue including real-world data from the Chrome User Experience Report in this API" and points to the CrUX API instead. More on that below.

Quotas: you need an API key

The docs say a key is "recommended for frequent, automated queries". In practice it's mandatory. When I tested on October 21, 2026, a single keyless request from a fresh machine came back immediately with:

HTTP 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per day'
of service 'pagespeedonline.googleapis.com' for consumer 'project_number:583797351490'.

Every keyless call worldwide counts against that one shared project, so it's exhausted. Create a key in Google Cloud Console (APIs & Services → enable "PageSpeed Insights API" → Credentials → API key), restrict it to the PSI API, and pass it as key=. It's free.

With a key, the commonly cited default is 25,000 queries per day per Cloud project, plus a short-window rate limit. Published numbers for that window differ a lot, so check yours under APIs & Services → PageSpeed Insights API → Quotas. What matters for code:

Each call takes roughly 10-30 seconds because Google really loads the page. 1,000 URLs one at a time is several hours; with 4-8 in parallel it's well under an hour.
A daily-quota 429 is fatal, a per-minute 429 is not. Retrying the daily one just burns time until midnight Pacific. The message text ("per day") tells you which one you hit.
500/503 happen, especially on heavy pages, and usually succeed on retry. A 400 usually means the page itself failed (DNS error, 404, not HTML). Log it and move on, except NO_FCP, which is often transient.
Mobile and desktop are separate calls. 1,000 URLs × 2 strategies = 2,000 queries.

The script

Only dependency: pip install requests. It reads a file with one URL per line, runs them in parallel, retries with exponential backoff and jitter, stops cleanly on a daily-quota or bad-key error, and writes pagespeed.csv with lab metrics and field data side by side. The --local flag runs Lighthouse on your own machine instead (npm i -g lighthouse, needs Chrome), which returns the same lighthouseResult JSON. Handy when you have no key yet or want to test staging sites PSI can't reach. It's lab-only, though: field data only comes from Google.

"""Bulk Core Web Vitals / PageSpeed checker.
Usage:  PSI_API_KEY=... python3 pagespeed_bulk.py urls.txt [mobile|desktop] [workers]
        python3 pagespeed_bulk.py urls.txt mobile 2 --local   # no key: run Lighthouse on this machine
Writes pagespeed.csv (one row per URL) with lab metrics + CrUX field data (p75)."""
import csv, json, os, random, subprocess, sys, time
from concurrent.futures import ThreadPoolExecutor
import requests

API = "https://www.googleapis.com/pagespeedonline/v5/runPagespeed"
KEY = os.environ.get("PSI_API_KEY")
LIGHTHOUSE = os.environ.get("LIGHTHOUSE_BIN", "lighthouse")  # npm i -g lighthouse

class Fatal(Exception):
    pass

def psi(url, strategy):
    params = {"url": url, "strategy": strategy.upper(), "category": "PERFORMANCE"}
    if KEY:
        params["key"] = KEY
    for attempt in range(5):
        try:
            r = requests.get(API, params=params, timeout=120)
        except requests.RequestException as e:
            err = f"network: {type(e).__name__}"
        else:
            if r.ok:
                return r.json()
            msg = r.json().get("error", {}).get("message", r.text[:200]) if r.content else f"HTTP {r.status_code}"
            if r.status_code == 429 and "per day" in msg:
                raise Fatal("daily quota exhausted: " + msg)  # retrying won't help until tomorrow
            if r.status_code == 403 or "API key not valid" in msg:
                raise Fatal("API key problem: " + msg)
            if r.status_code == 400 and "NO_FCP" not in msg:
                return {"error": msg[:200]}  # page itself failed to load (DNS, 404, not HTML...)
            if r.status_code not in (429, 500, 502, 503, 504) and r.status_code != 400:
                return {"error": f"HTTP {r.status_code}: {msg[:200]}"}
            err = f"HTTP {r.status_code}"
        time.sleep(2 ** attempt * 5 + random.random() * 3)  # 5s, 10s, 20s, 40s ... with jitter
    return {"error": f"gave up after retries ({err})"}

def local_lighthouse(url, strategy):
    cmd = [LIGHTHOUSE, url, "--output=json", "--quiet", "--only-categories=performance",
           "--chrome-flags=--headless=new --no-sandbox"]
    if strategy == "desktop":
        cmd.append("--preset=desktop")
    p = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
    if p.returncode or not p.stdout.strip():
        return {"error": (p.stderr.strip().splitlines() or ["lighthouse failed"])[-1][:200]}
    return {"lighthouseResult": json.loads(p.stdout)}  # same shape PSI returns

def field(exp):
    m = (exp or {}).get("metrics") or {}
    get = lambda k: (m.get(k) or {}).get("percentile")
    cls = get("CUMULATIVE_LAYOUT_SHIFT_SCORE")
    return {"lcp_p75_ms": get("LARGEST_CONTENTFUL_PAINT_MS"), "inp_p75_ms": get("INTERACTION_TO_NEXT_PAINT"),
            "cls_p75": cls / 100 if cls is not None else None,  # CrUX sends CLS x100
            "cwv": exp.get("overall_category") if m else None}

def row(url, data):
    if "error" in data:
        return {"url": url, "error": data["error"]}
    lhr = data["lighthouseResult"]
    a = lhr["audits"]
    val = lambda k: a.get(k, {}).get("numericValue")
    page, origin = field(data.get("loadingExperience")), field(data.get("originLoadingExperience"))
    src = "page" if page["cwv"] else "origin" if origin["cwv"] else "none"
    f = page if src == "page" else origin
    score = lhr["categories"]["performance"]["score"]
    return {"url": url, "final_url": lhr.get("finalDisplayedUrl"),
            "perf_score": round(score * 100) if score is not None else None,
            "lab_lcp_ms": round(val("largest-contentful-paint") or 0), "lab_cls": round(val("cumulative-layout-shift") or 0, 3),
            "lab_tbt_ms": round(val("total-blocking-time") or 0), "lab_fcp_ms": round(val("first-contentful-paint") or 0),
            "field_source": src, "field_cwv": f["cwv"], "field_lcp_p75_ms": f["lcp_p75_ms"],
            "field_inp_p75_ms": f["inp_p75_ms"], "field_cls_p75": f["cls_p75"], "error": ""}

if __name__ == "__main__":
    args = [a for a in sys.argv[1:] if not a.startswith("--")]
    urls = list(dict.fromkeys(u.strip() for u in open(args[0]) if u.strip().startswith("http")))
    strategy = args[1] if len(args) > 1 else "mobile"
    workers = int(args[2]) if len(args) > 2 else 4
    run = local_lighthouse if "--local" in sys.argv else psi
    t, stop = time.time(), []
    def one(u):
        if stop:
            return {"url": u, "error": "skipped: " + stop[0]}
        try:
            r = row(u, run(u, strategy))
        except Fatal as e:
            stop.append(str(e)[:120]); r = {"url": u, "error": str(e)[:200]}
        print(f"{r.get('perf_score')!s:>4}  {r.get('lab_lcp_ms')!s:>6} ms  {r.get('field_cwv') or '-':<8} {u} {r['error']}", flush=True)
        return r
    with ThreadPoolExecutor(workers) as pool:
        rows = list(pool.map(one, urls))
    cols = ["url", "final_url", "perf_score", "lab_lcp_ms", "lab_cls", "lab_tbt_ms", "lab_fcp_ms", "field_source",
            "field_cwv", "field_lcp_p75_ms", "field_inp_p75_ms", "field_cls_p75", "error"]
    with open("pagespeed.csv", "w", newline="") as fh:
        w = csv.DictWriter(fh, cols); w.writeheader(); w.writerows(rows)
    print(f"{len(rows)} URLs in {time.time() - t:.0f} s, {sum(1 for r in rows if r['error'])} errors -> pagespeed.csv")

Run it:

export PSI_API_KEY=your-key
python3 pagespeed_bulk.py urls.txt mobile 6

Results from a test run

My test machine had no PSI key (that's how I found the keyless 429 above), so I tested the full pipeline (parsing, concurrency, CSV) with --local: Lighthouse 12.8.2 and headless Chrome 154 on an 8-core Linux box, mobile emulation, six public sites. Same code path, same lighthouseResult format PSI returns.

Run 1, two in parallel (42 s total):

URL	Score	LCP	CLS	TBT
example.com	100	839 ms	0	67 ms
python.org	77	4,336 ms	0.003	6 ms
developer.mozilla.org/en-US/	88	3,309 ms	0	0 ms
crawlee.dev	37	11,971 ms	0	2,507 ms
wikipedia.org	76	3,977 ms	0.001	117 ms
web.dev	82	2,864 ms	0.12	206 ms

Run 2, same URLs, one at a time a few minutes later (78 s total): example.com 100, python.org 77, MDN 90, crawlee.dev 47 with LCP 4,692 ms, wikipedia.org 82, web.dev 74.

That second run is the most useful result here:

Lab numbers are noisy. crawlee.dev's LCP went from 12.0 s to 4.7 s and its score from 37 to 47 between two runs; web.dev moved 8 points. TBT stayed around 2.5 s both times, so the JavaScript cost is real; the LCP swing is not. For decisions, run each URL 3-5 times and take the median, or lean on field data.
Parallelism on one machine skews local results. Running two Lighthouse instances at once competes for CPU, which inflates TBT and LCP. That's fine with the PSI API (Google's machines do the work, so concurrency is just about quota), but with --local keep workers low.
CLS was the most stable metric. web.dev showed 0.12 in both runs, just over the 0.1 "good" threshold, which is exactly the kind of thing worth a ticket.

Getting field data without PSI

Since PSI's field data is scheduled to go away, it's worth querying CrUX directly. Same Google key (enable the "Chrome UX Report API" on the project), one POST per URL or origin, and it's much faster than PSI because nothing gets loaded:

r = requests.post(
    "https://chromeuxreport.googleapis.com/v1/records:queryRecord",
    params={"key": KEY},
    json={"url": "https://web.dev/", "formFactor": "PHONE"},  # or {"origin": "https://web.dev"}
    timeout=30,
)
m = r.json().get("record", {}).get("metrics", {})
lcp = m.get("largest_contentful_paint", {}).get("percentiles", {}).get("p75")
inp = m.get("interaction_to_next_paint", {}).get("percentiles", {}).get("p75")
cls = m.get("cumulative_layout_shift", {}).get("percentiles", {}).get("p75")  # a string like "0.05"

A 404 means CrUX doesn't have enough traffic for that URL; retry with origin instead. Without a key the endpoint answers 403 PERMISSION_DENIED ("Method doesn't allow unregistered callers"), as I confirmed on the same day.

Things that will bite you

Normalize and dedupe URLs first. http://, https://, trailing slashes and ?utm_ variants all count as separate queries. The script dedupes exact duplicates only.
Check final_url. If a URL redirects to a login page or the homepage, you're measuring the wrong page. The script keeps the final URL so you can spot it.
Sample by template. For a 50,000-page store, 1,000 URLs spread across templates (home, category, product, article) tells you nearly as much as all 50,000, and it fits in the daily quota with room for mobile and desktop.
Save the raw JSON if you'll want opportunities later. The CSV keeps the headline metrics; lighthouseResult.audits has the fix-it list (unused JavaScript, image sizes, render-blocking resources).

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Bulk PageSpeed Insights & Core Web Vitals Checker. You paste a list of URLs and get scores, lab metrics and the top opportunities per URL and device. It works without a Google key (it runs its own Lighthouse, $0.015 per audit) or with your own PSI key ($0.004 per audit, and that adds the CrUX field data), can run on a schedule, and is callable by API. The script above is the whole idea, though, so use whichever fits.

Find broken links on any website with Python, grouped by the page you need to fix (2026)

Datawright Labs — Wed, 21 Oct 2026 02:16:10 +0000

A broken-link report is only useful if it tells you where to go to fix things. "This URL returns 404" is half the answer. "These three pages link to it" is the part you can act on. Most quick scripts check one page at a time and lose that mapping, or they check the same footer link 50 times.

This post builds a small crawler that does it properly: crawl the site, collect every outgoing URL once, check each one once, and write a CSV that maps every broken URL back to the pages that contain it. Then it looks at a real run, because the interesting part of link checking is the false positives.

The approach

Breadth-first crawl of same-site pages. Start at the homepage, follow <a href> links that stay on the same host, stop at a page limit.
Collect every outgoing URL with its source pages. Links, images, scripts and stylesheets all count. A missing image or script is as broken as a dead link. Strip #fragments so /docs#install and /docs#usage are one check.
Check each unique URL once, in parallel. Use GET with stream=True instead of HEAD. Plenty of servers answer HEAD with 405 or 404 even when the page is fine, and some just hang on it. A streamed GET reads only the headers and closes the connection.
Retry once before calling something broken. Transient timeouts are common.

The script

# Usage: python3 broken_links.py https://example.com [max_pages]
# Crawls same-site pages (BFS), checks every link and image once, writes broken.csv grouped by source page.
import csv, sys, time, collections
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin, urldefrag, urlparse
import requests
from bs4 import BeautifulSoup

start = sys.argv[1]
max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 50
host = urlparse(start).netloc
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (compatible; link-check-tutorial/1.0)"

def check(url):
    # GET with stream=True reads headers only; many servers answer HEAD wrongly (405/404) or hang.
    for attempt in range(2):
        try:
            r = s.get(url, stream=True, timeout=15, allow_redirects=True)
            r.close()
            return r.status_code, r.url
        except requests.RequestException as e:
            err = type(e).__name__
            time.sleep(2)
    return err, url

sources = collections.defaultdict(set)   # link -> pages that contain it
queue, seen_pages, i = [start], {start}, 0
t = time.time()
while i < len(queue) and i < max_pages:
    page = queue[i]; i += 1
    try:
        r = s.get(page, timeout=20)
    except requests.RequestException:
        continue
    if "text/html" not in r.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    for tag, attr in (("a", "href"), ("img", "src"), ("link", "href"), ("script", "src")):
        for el in soup.find_all(tag):
            raw = el.get(attr)
            if not raw or raw.startswith(("mailto:", "tel:", "javascript:", "data:", "#")):
                continue
            link = urldefrag(urljoin(r.url, raw))[0]
            if not link.startswith("http"):
                continue
            sources[link].add(page)
            if tag == "a" and urlparse(link).netloc == host and link not in seen_pages:
                seen_pages.add(link); queue.append(link)

links = sorted(sources)
with ThreadPoolExecutor(10) as pool:
    results = dict(zip(links, pool.map(check, links)))

broken = [(u, st) for u, (st, _) in results.items() if not (isinstance(st, int) and st < 400)]
with open("broken.csv", "w", newline="") as f:
    w = csv.writer(f); w.writerow(["source_page", "broken_url", "status"])
    for u, st in sorted(broken):
        for p in sorted(sources[u]):
            w.writerow([p, u, st])
print(f"{i} pages crawled, {len(links)} unique URLs checked, {len(broken)} broken in {time.time() - t:.0f} s -> broken.csv")
for u, st in sorted(broken, key=lambda x: str(x[1]))[:15]:
    print(f"{str(st):<16} {u}  (on {len(sources[u])} page(s))")

Install the two dependencies with pip install requests beautifulsoup4 and run python3 broken_links.py https://your-site.com 200.

Results on crawlee.dev

I ran python3 broken_links.py https://crawlee.dev 50 on October 20, 2026:

50 pages crawled, 3,008 unique URLs checked in 245 seconds, 10 checks in parallel.
35 URLs flagged. That number is misleading, which is the main lesson of this post.

Here is what the 35 actually were:

Status	Count	Really broken?
403	14	No. Stack Overflow, npm, Crunchbase, Upwork, OpenAI and others block scripted requests. The Stack Overflow tag link alone appears on 37 pages.
999	2	No. LinkedIn's "go away, bot" status.
Connect timeout	7	No. All were plain `http://` links, and my test machine's network blocks outbound port 80. Worth knowing if you run this from a locked-down server or CI runner.
404	8	7 yes, 1 no (see below).
SSL error	4	Yes.

So 11 of the 35 were genuinely broken, and every one of them is a real typo or a bad link:

https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator on a blog post, with "fingeprint" missing an "r".
Three links like https://github.com//github.com/apify/crawlee/pull/1890/..., where a full URL got pasted into a field that already prefixed https://github.com/.
https://www.en.wikipedia.org/wiki/Robots.txt and https://www.developer.mozilla.org/...: an extra www. in front of hosts that don't have one, so the TLS certificate doesn't match.
A github.com/renovate%5Bbot%5D profile link (bot accounts have no profile page) and two SwaggerHub API docs that no longer exist.

The one 404 that wasn't broken was https://5JC94MPMLY-dsn.algolia.net, a <link rel="preconnect"> to the search provider. Its root path 404s, but nobody ever navigates there.

Things that will bite you

Treat 401, 403, 429 and 999 as "blocked", not "broken". Report them in a separate column, or re-check them from a real browser. If you count them as broken, your report is mostly noise and people stop reading it.
Skip preconnect and dns-prefetch links, or at least don't treat them as pages. They point at hosts, not documents.
Check from a normal network. Corporate proxies and CI runners often block plain HTTP or whole domains, which shows up as timeouts.
Be polite on your own site. This script checks external URLs 10 at a time but has no per-host limit. On a big site most links point at your own host, so add a per-host concurrency cap and respect robots.txt before you crawl thousands of pages.
Watch the redirect chains. allow_redirects=True hides a 301 to a 404. The final status is what matters for users, but r.history tells you which links are worth updating to skip the redirect.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Broken Link Checker. It adds robots.txt handling, a per-host concurrency limit, a spending cap per run, and source-page grouping for every broken link, image and script. It costs $0.50 per 1,000 URLs checked, so Apify's free monthly credit covers small and medium sites, and you can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.

Google autocomplete for keyword research in Python: A-Z, question and YouTube suggestions in bulk (2026)

Datawright Labs — Tue, 20 Oct 2026 18:20:02 +0000

Autocomplete is the closest thing to free search demand data you can get. Every suggestion Google, YouTube or Bing shows is something real people type often enough for the engine to offer it, and the endpoints that power those dropdowns return plain JSON with no API key. Tools like AnswerThePublic and keyword.io are mostly this idea plus a nice UI.

This post covers the three endpoints, the expansion trick that turns 10 suggestions into hundreds, and a small script that merges everything into one ranked CSV.

The endpoints

Google (and YouTube and Google Shopping, same host):

https://suggestqueries.google.com/complete/search?client=chrome&hl=en&gl=us&q=standing+desk

client=chrome returns JSON shaped like ["query", ["suggestion 1", ...], [descriptions], [], {"google:suggesttype": ["QUERY", "NAVIGATION", ...], "google:suggestrelevance": [...]}]. The type list lets you drop navigational results (URLs) and keep real queries.
ds=yt switches to YouTube suggestions, ds=sh to Google Shopping. YouTube ideas skew toward "how to", reviews and comparisons, which is exactly what you want for video or tutorial content.
hl is the language and gl the country. Suggestions differ a lot by market, so set both.
Decode the body as UTF-8 yourself. With client=chrome the response can come back without a charset, and non-English suggestions get mangled if you trust the default.

Bing:

https://api.bing.com/osjson.aspx?query=standing+desk&mkt=en-US

This is the OpenSearch suggestion format: ["query", ["suggestion 1", ...]]. Bing's list is noticeably different from Google's, so it's worth a second source.

Turning 10 suggestions into 1,000

Each request only returns up to about 10 suggestions. The standard trick is to query variations of the seed and merge the results:

A-Z: standing desk a, standing desk b, ... standing desk z. Each letter surfaces a different slice of long-tail phrases.
Question prefixes: how standing desk, is standing desk, why standing desk. These give you the "is a standing desk worth it" style queries that make good FAQ sections and blog posts.
Modifier suffixes: vs, for, without, near me surface comparisons, audiences and local intent.

Then dedupe (lowercase and collapse whitespace) and keep track of which engines suggested each idea. A phrase that Google, YouTube and Bing all suggest is a stronger signal than one that shows up once.

The script

Only dependency: pip install requests.

import csv, json, string, sys, time
import requests

UA = {"User-Agent": "Mozilla/5.0 (autocomplete-tutorial/1.0)"}
PREFIXES = ["how", "what", "why", "can", "is", "best"]  # question-style ideas
SUFFIXES = ["vs", "for", "without", "near me"]

def google(q, hl="en", gl="us", ds=None):
    p = {"client": "chrome", "ie": "utf-8", "oe": "utf-8", "hl": hl, "gl": gl, "q": q}
    if ds:
        p["ds"] = ds  # "yt" = YouTube, "sh" = Google Shopping
    r = requests.get("https://suggestqueries.google.com/complete/search", params=p, headers=UA, timeout=20)
    r.raise_for_status()
    data = json.loads(r.content.decode("utf-8"))
    types = (data[4] if len(data) > 4 else {}).get("google:suggesttype", [])
    return [s for i, s in enumerate(data[1]) if i >= len(types) or types[i] == "QUERY"]

def bing(q, mkt="en-US"):
    r = requests.get("https://api.bing.com/osjson.aspx", params={"query": q, "mkt": mkt}, headers=UA, timeout=20)
    r.raise_for_status()
    return r.json()[1]

ENGINES = {"google": google, "youtube": lambda q: google(q, ds="yt"), "bing": bing}

def expand(seed):
    yield seed
    for c in string.ascii_lowercase:
        yield f"{seed} {c}"
    for w in PREFIXES:
        yield f"{w} {seed}"
    for w in SUFFIXES:
        yield f"{seed} {w}"

def ideas(seed, engines=("google", "youtube", "bing"), pause=0.3):
    seen = {}
    for engine in engines:
        for q in expand(seed):
            try:
                results = ENGINES[engine](q)
            except requests.RequestException as e:
                print(f"  {engine} {q!r}: {e}", file=sys.stderr)
                time.sleep(5)
                continue
            for pos, s in enumerate(results, 1):
                k = " ".join(s.lower().split())
                row = seen.setdefault(k, {"keyword": k, "engines": set(), "best_position": pos, "hits": 0})
                row["engines"].add(engine)
                row["best_position"] = min(row["best_position"], pos)
                row["hits"] += 1
            time.sleep(pause)
    return sorted(seen.values(), key=lambda r: (-len(r["engines"]), -r["hits"], r["best_position"]))

if __name__ == "__main__":
    seed = " ".join(sys.argv[1:]) or "standing desk"
    t = time.time()
    rows = ideas(seed)
    with open("keywords.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["keyword", "engines", "hits", "best_position"])
        for r in rows:
            w.writerow([r["keyword"], "+".join(sorted(r["engines"])), r["hits"], r["best_position"]])
    print(f"{len(rows)} unique ideas for {seed!r} in {time.time() - t:.1f} s -> keywords.csv")
    for r in rows[:10]:
        print(f'{r["keyword"]:<45} {"+".join(sorted(r["engines"])):<20} hits={r["hits"]}')

Results

I ran python3 autocomplete_ideas.py standing desk on October 20, 2026. That's 37 queries per engine, 111 requests in total with a 0.3 s pause between them:

1,013 unique keyword ideas in 57 seconds.
73 of them were suggested by all three engines, for example "standing desk with drawers", "standing desk topper", "standing desk treadmill" and "standing desk benefits".
Each engine contributed a lot of its own: 286 ideas only from Google, 275 only from YouTube and 212 only from Bing. Using just one engine would have missed roughly two thirds of the list.
The question prefixes produced usable content ideas straight away: "is standing desk worth it", "is standing desk better than sitting", "is standing desk hsa eligible", "what standing desk to buy".
One Bing request failed with a transient SSL error. The script logs it, waits 5 seconds and moves on, which is why the error handling is there.

Things that will bite you at scale

Rate limits: a few hundred requests from one IP is fine. Thousands in a row from a single IP will start getting HTTP 429 or a captcha page instead of JSON (the response stops parsing as JSON, which is the easiest way to detect it). Slow down, add retries with backoff, or spread requests across IPs.
Personalization: suggestions vary with language, country and sometimes location. Pin hl, gl and mkt so runs are comparable over time.
No volume numbers: autocomplete tells you a query is popular enough to suggest, not how many searches it gets. Use the engine count, the number of expansions an idea appears in, and its position as a relative signal, then check the top candidates in a volume tool if you need numbers.
Duplicates across expansions: "standing desk with drawers" comes back from the seed, from "w" and from "with". Merge before you count, otherwise your favourite ideas look more important than they are.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this (Google, YouTube, Google Shopping and Bing, A-Z, 0-9 and question expansions, any language and country, recursive depth, must-contain and exclude-word filters, retries and proxy rotation for big seed lists) as a hosted tool on Apify: Autocomplete Keyword Scraper. It costs $0.20 per 1,000 unique suggestions, with duplicates merged before you pay, so Apify's free monthly credit covers most small jobs. You can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.