Bulk Core Web Vitals with the free PageSpeed Insights API (and a small hosted option)

Innovative Cookie — Mon, 09 Nov 2026 12:54:42 +0000

Core Web Vitals are easiest to fix when you look at many URLs at once. Google’s PageSpeed Insights (PSI) API v5 is free with your own API key and returns lab Lighthouse scores plus real-user CrUX field data in one call.

1. Get a free Google API key

In Google Cloud, enable “PageSpeed Insights API”, create an API key, and check the Quotas page for your daily limit.

2. One URL, one call

curl "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?url=https://example.com&strategy=mobile&category=performance&key=YOUR_KEY"

Useful JSON paths:

Lab: lighthouseResult.audits (largest-contentful-paint, cumulative-layout-shift, total-blocking-time)
Field (when enough traffic): loadingExperience.metrics; else originLoadingExperience

3. Bulk loop (keep it polite)

import csv, json, time, urllib.parse, urllib.request

KEY = "YOUR_KEY"
URLS = ["https://example.com/", "https://example.com/pricing"]

def psi(url, strategy="mobile"):
    q = urllib.parse.urlencode({
        "url": url, "strategy": strategy,
        "category": "performance", "key": KEY,
    })
    with urllib.request.urlopen(
        "https://www.googleapis.com/pagespeedonline/v5/runPagespeed?" + q,
        timeout=90,
    ) as r:
        return json.load(r)

rows = []
for u in URLS:
    data = psi(u)
    audits = data["lighthouseResult"]["audits"]
    rows.append({
        "url": u,
        "perf": data["lighthouseResult"]["categories"]["performance"]["score"],
        "lcp_ms": audits["largest-contentful-paint"]["numericValue"],
        "cls": audits["cumulative-layout-shift"]["numericValue"],
        "tbt_ms": audits["total-blocking-time"]["numericValue"],
        "fetched_at": data.get("analysisUTCTimestamp"),
    })
    time.sleep(0.5)  # stay under quota; back off on HTTP 429

with open("cwv.csv", "w", newline="") as f:
    w = csv.DictWriter(f, fieldnames=rows[0].keys())
    w.writeheader(); w.writerows(rows)
print(len(rows), "rows → cwv.csv")

Start from failing field metrics, then use lab audits to find the cause. Re-run after fixes; lab scores move between runs.

Hosted option (optional)

I built a small Apify Actor that runs the same PSI calls in bulk from a URL list, a sitemap or a domain, and exports CSV/JSON. You still bring your own free Google key. Disclosure: I am the creator.

https://apify.com/innovative_cookie/pagespeed-bulk

No Actor fee for now — only Apify platform usage (Free plan includes monthly credit).

Written with AI assistance.

How to extract every URL from a website's sitemaps and check their status codes (Python, free)

Innovative Cookie — Mon, 09 Nov 2026 10:58:33 +0000

Before you crawl a site, look at what it tells search engines it has: its XML sitemaps. Starting from the sitemap gives you the list of pages the owner wants indexed, and checking their status codes finds the 404s, 5xx and redirects that should not be in there.

This tutorial does it in two small Python scripts, using only requests and the standard library.

Step 1: find the sitemaps

Most sites declare them in robots.txt:

curl -s https://example.com/robots.txt | grep -i '^sitemap:'

If nothing is declared, try the usual locations: /sitemap.xml, /sitemap_index.xml (Yoast), /wp-sitemap.xml (WordPress core) and /sitemap.xml.gz.

Step 2: get every URL, including nested indexes and .gz files

A sitemap is either a urlset (a list of pages) or a sitemapindex (a list of other sitemaps). Indexes can point to more indexes, and some files are gzipped whatever their extension says, so the script checks the gzip magic bytes and recurses. Save it as sitemap_urls.py:

import gzip
import xml.etree.ElementTree as ET

import requests

NS = {"s": "http://www.sitemaps.org/schemas/sitemap/0.9"}
HEADERS = {"User-Agent": "my-sitemap-script/1.0"}


def sitemap_urls(sitemap_url):
    """Yield every page URL from a sitemap or sitemap index (recursively)."""
    r = requests.get(sitemap_url, headers=HEADERS, timeout=20)
    r.raise_for_status()
    data = r.content
    if data[:2] == b"\x1f\x8b":  # gzip magic bytes, whatever the extension says
        data = gzip.decompress(data)
    root = ET.fromstring(data)
    if root.tag.endswith("sitemapindex"):
        for loc in root.findall("s:sitemap/s:loc", NS):
            yield from sitemap_urls(loc.text.strip())
    else:
        for loc in root.findall("s:url/s:loc", NS):
            yield loc.text.strip()


if __name__ == "__main__":
    urls = list(sitemap_urls("https://crawlee.dev/sitemap.xml"))
    print(len(urls), "URLs")
    print(urls[:3])

On https://crawlee.dev/sitemap.xml this printed 3877 URLs when I ran it on 29 October 2026.

A few traps:

One sitemap file holds at most 50,000 URLs and 50 MB, so big sites always use an index.
Some servers return an HTML page with status 200 for a missing sitemap. ET.fromstring then fails or finds no <url>; treat that as an error, not as "0 pages".
Deduplicate: the same URL often appears in several sitemaps.

Step 3: check the status code of every URL

Send a HEAD request (no page body), fall back to GET when the server refuses HEAD with 405 or 501, follow redirects, and record the final URL and the redirect hops. Keep it polite: one request at a time with a short pause. Save this as check_status.py next to the first file and run python3 check_status.py:

import csv
import time

import requests

from sitemap_urls import HEADERS, sitemap_urls


def check(url, session):
    """Return (status, final_url). HEAD first, GET if the server refuses HEAD."""
    try:
        r = session.head(url, headers=HEADERS, allow_redirects=True, timeout=15)
        if r.status_code in (405, 501):
            r = session.get(url, headers=HEADERS, allow_redirects=True, timeout=15, stream=True)
            r.close()
        hops = [h.status_code for h in r.history]
        return r.status_code, r.url, hops
    except requests.RequestException as e:
        return None, None, [type(e).__name__]


with requests.Session() as s, open("status.csv", "w", newline="") as f:
    out = csv.writer(f)
    out.writerow(["url", "status", "final_url", "redirect_hops"])
    for i, url in enumerate(sitemap_urls("https://crawlee.dev/sitemap.xml")):
        if i >= 50:  # keep the demo small
            break
        status, final, hops = check(url, s)
        out.writerow([url, status, final if final != url else "", " > ".join(map(str, hops))])
        time.sleep(0.25)  # be polite: one request every 250 ms

print(open("status.csv").read()[:400])

On the first 50 URLs of crawlee.dev it took about 20 seconds and found 50 × 200. Here is what the same check() function returns for a URL that redirects twice and for one that does not exist:

https://crawlee.dev/docs -> (200, 'https://apify.github.io/crawlee/js/docs/', [301, 301])
https://crawlee.dev/this-does-not-exist-xyz -> (404, 'https://crawlee.dev/this-does-not-exist-xyz', [])

How to read status.csv:

4xx / 5xx: remove the URL from the sitemap or fix the page.
A non-empty redirect_hops (like 301 > 301): put the final URL in the sitemap instead. Sitemaps should list canonical, final URLs.
Errors (timeouts, DNS): check again later before acting.

Step 4: make it a habit

Run it after every migration or big content change, or weekly from cron or a CI job, and diff the CSV with the previous run.

Please respect the sites you check: keep the delay, use a clear user agent, and only check sites you own or have permission to audit.

Questions or edge cases your sitemaps hit? Tell me in the comments.

This article was written with the help of AI. I ran and checked every code snippet and result before publishing.

DEV Community: Innovative Cookie