How to check Core Web Vitals for 1,000 URLs in Python (PageSpeed Insights API, CrUX field data, quotas and retries)

Checking one page in PageSpeed Insights is easy. Checking every template, landing page and product page on a site, every week, is where people end up with a spreadsheet of copy-pasted scores and no idea which numbers they can trust.

This post covers what the PageSpeed Insights (PSI) API actually returns, the difference between lab and field data (and which one Google uses for Core Web Vitals), the quota rules that will stop you halfway through a list, and a script that handles all of it and writes a CSV.

Lab data vs field data

One runPagespeed call gives you two very different things:

• Lab data (lighthouseResult): Google loads the page once in Lighthouse on a throttled, emulated phone (or desktop) and measures it. You get the 0-100 performance score, LCP, CLS, TBT, FCP and Speed Index. It's repeatable-ish and works for any URL, including pages nobody visits.
• Field data (loadingExperience for the URL, originLoadingExperience for the whole origin): the 75th percentile of what real Chrome users experienced over the last 28 days, from the Chrome UX Report (CrUX). This is where INP lives, because interaction latency needs real users clicking things. Lab data has no INP; TBT is the lab stand-in.

The Core Web Vitals assessment (the "passed/failed" you see in Search Console) is based on field data: LCP ≤ 2.5 s, INP ≤ 200 ms and CLS ≤ 0.1 at p75. The lab performance score is a diagnostic tool, not a ranking signal. If you only have budget for one column in your report, make it the field one.

Three details that trip people up:

• Low-traffic URLs have no page-level field data. loadingExperience comes back without metrics, and PSI may fall back to origin-level data. Record which one you got, or you'll compare a page against a whole site without knowing it.
• CrUX sends CLS multiplied by 100. A CUMULATIVE_LAYOUT_SHIFT_SCORE percentile of 5 means CLS 0.05.
• Google says it plans to remove CrUX data from the PSI API. The PSI getting-started page currently says: "We plan to discontinue including real-world data from the Chrome User Experience Report in this API" and points to the CrUX API instead. More on that below.

Quotas: you need an API key

The docs say a key is "recommended for frequent, automated queries". In practice it's mandatory. When I tested on October 21, 2026, a single keyless request from a fresh machine came back immediately with:

HTTP 429: Quota exceeded for quota metric 'Queries' and limit 'Queries per day'
of service 'pagespeedonline.googleapis.com' for consumer 'project_number:583797351490'.

Every keyless call worldwide counts against that one shared project, so it's exhausted. Create a key in Google Cloud Console (APIs & Services → enable "PageSpeed Insights API" → Credentials → API key), restrict it to the PSI API, and pass it as key=. It's free.

With a key, the commonly cited default is 25,000 queries per day per Cloud project, plus a short-window rate limit. Published numbers for that window differ a lot, so check yours under APIs & Services → PageSpeed Insights API → Quotas. What matters for code:

• Each call takes roughly 10-30 seconds because Google really loads the page. 1,000 URLs one at a time is several hours; with 4-8 in parallel it's well under an hour.
• A daily-quota 429 is fatal, a per-minute 429 is not. Retrying the daily one just burns time until midnight Pacific. The message text ("per day") tells you which one you hit.
• 500/503 happen, especially on heavy pages, and usually succeed on retry. A 400 usually means the page itself failed (DNS error, 404, not HTML). Log it and move on, except NO_FCP, which is often transient.
• Mobile and desktop are separate calls. 1,000 URLs × 2 strategies = 2,000 queries.

The script

Only dependency: pip install requests. It reads a file with one URL per line, runs them in parallel, retries with exponential backoff and jitter, stops cleanly on a daily-quota or bad-key error, and writes pagespeed.csv with lab metrics and field data side by side. The --local flag runs Lighthouse on your own machine instead (npm i -g lighthouse, needs Chrome), which returns the same lighthouseResult JSON. Handy when you have no key yet or want to test staging sites PSI can't reach. It's lab-only, though: field data only comes from Google.

"""Bulk Core Web Vitals / PageSpeed checker.
Usage:  PSI_API_KEY=... python3 pagespeed_bulk.py urls.txt [mobile|desktop] [workers]
        python3 pagespeed_bulk.py urls.txt mobile 2 --local   # no key: run Lighthouse on this machine
Writes pagespeed.csv (one row per URL) with lab metrics + CrUX field data (p75)."""
import csv, json, os, random, subprocess, sys, time
from concurrent.futures import ThreadPoolExecutor
import requests

API = "https://www.googleapis.com/pagespeedonline/v5/runPagespeed"
KEY = os.environ.get("PSI_API_KEY")
LIGHTHOUSE = os.environ.get("LIGHTHOUSE_BIN", "lighthouse")  # npm i -g lighthouse

class Fatal(Exception):
    pass

def psi(url, strategy):
    params = {"url": url, "strategy": strategy.upper(), "category": "PERFORMANCE"}
    if KEY:
        params["key"] = KEY
    for attempt in range(5):
        try:
            r = requests.get(API, params=params, timeout=120)
        except requests.RequestException as e:
            err = f"network: {type(e).__name__}"
        else:
            if r.ok:
                return r.json()
            msg = r.json().get("error", {}).get("message", r.text[:200]) if r.content else f"HTTP {r.status_code}"
            if r.status_code == 429 and "per day" in msg:
                raise Fatal("daily quota exhausted: " + msg)  # retrying won't help until tomorrow
            if r.status_code == 403 or "API key not valid" in msg:
                raise Fatal("API key problem: " + msg)
            if r.status_code == 400 and "NO_FCP" not in msg:
                return {"error": msg[:200]}  # page itself failed to load (DNS, 404, not HTML...)
            if r.status_code not in (429, 500, 502, 503, 504) and r.status_code != 400:
                return {"error": f"HTTP {r.status_code}: {msg[:200]}"}
            err = f"HTTP {r.status_code}"
        time.sleep(2 ** attempt * 5 + random.random() * 3)  # 5s, 10s, 20s, 40s ... with jitter
    return {"error": f"gave up after retries ({err})"}

def local_lighthouse(url, strategy):
    cmd = [LIGHTHOUSE, url, "--output=json", "--quiet", "--only-categories=performance",
           "--chrome-flags=--headless=new --no-sandbox"]
    if strategy == "desktop":
        cmd.append("--preset=desktop")
    p = subprocess.run(cmd, capture_output=True, text=True, timeout=180)
    if p.returncode or not p.stdout.strip():
        return {"error": (p.stderr.strip().splitlines() or ["lighthouse failed"])[-1][:200]}
    return {"lighthouseResult": json.loads(p.stdout)}  # same shape PSI returns

def field(exp):
    m = (exp or {}).get("metrics") or {}
    get = lambda k: (m.get(k) or {}).get("percentile")
    cls = get("CUMULATIVE_LAYOUT_SHIFT_SCORE")
    return {"lcp_p75_ms": get("LARGEST_CONTENTFUL_PAINT_MS"), "inp_p75_ms": get("INTERACTION_TO_NEXT_PAINT"),
            "cls_p75": cls / 100 if cls is not None else None,  # CrUX sends CLS x100
            "cwv": exp.get("overall_category") if m else None}

def row(url, data):
    if "error" in data:
        return {"url": url, "error": data["error"]}
    lhr = data["lighthouseResult"]
    a = lhr["audits"]
    val = lambda k: a.get(k, {}).get("numericValue")
    page, origin = field(data.get("loadingExperience")), field(data.get("originLoadingExperience"))
    src = "page" if page["cwv"] else "origin" if origin["cwv"] else "none"
    f = page if src == "page" else origin
    score = lhr["categories"]["performance"]["score"]
    return {"url": url, "final_url": lhr.get("finalDisplayedUrl"),
            "perf_score": round(score * 100) if score is not None else None,
            "lab_lcp_ms": round(val("largest-contentful-paint") or 0), "lab_cls": round(val("cumulative-layout-shift") or 0, 3),
            "lab_tbt_ms": round(val("total-blocking-time") or 0), "lab_fcp_ms": round(val("first-contentful-paint") or 0),
            "field_source": src, "field_cwv": f["cwv"], "field_lcp_p75_ms": f["lcp_p75_ms"],
            "field_inp_p75_ms": f["inp_p75_ms"], "field_cls_p75": f["cls_p75"], "error": ""}

if __name__ == "__main__":
    args = [a for a in sys.argv[1:] if not a.startswith("--")]
    urls = list(dict.fromkeys(u.strip() for u in open(args[0]) if u.strip().startswith("http")))
    strategy = args[1] if len(args) > 1 else "mobile"
    workers = int(args[2]) if len(args) > 2 else 4
    run = local_lighthouse if "--local" in sys.argv else psi
    t, stop = time.time(), []
    def one(u):
        if stop:
            return {"url": u, "error": "skipped: " + stop[0]}
        try:
            r = row(u, run(u, strategy))
        except Fatal as e:
            stop.append(str(e)[:120]); r = {"url": u, "error": str(e)[:200]}
        print(f"{r.get('perf_score')!s:>4}  {r.get('lab_lcp_ms')!s:>6} ms  {r.get('field_cwv') or '-':<8} {u} {r['error']}", flush=True)
        return r
    with ThreadPoolExecutor(workers) as pool:
        rows = list(pool.map(one, urls))
    cols = ["url", "final_url", "perf_score", "lab_lcp_ms", "lab_cls", "lab_tbt_ms", "lab_fcp_ms", "field_source",
            "field_cwv", "field_lcp_p75_ms", "field_inp_p75_ms", "field_cls_p75", "error"]
    with open("pagespeed.csv", "w", newline="") as fh:
        w = csv.DictWriter(fh, cols); w.writeheader(); w.writerows(rows)
    print(f"{len(rows)} URLs in {time.time() - t:.0f} s, {sum(1 for r in rows if r['error'])} errors -> pagespeed.csv")

Run it:

export PSI_API_KEY=your-key
python3 pagespeed_bulk.py urls.txt mobile 6

Results from a test run

My test machine had no PSI key (that's how I found the keyless 429 above), so I tested the full pipeline (parsing, concurrency, CSV) with --local: Lighthouse 12.8.2 and headless Chrome 154 on an 8-core Linux box, mobile emulation, six public sites. Same code path, same lighthouseResult format PSI returns.

Run 1, two in parallel (42 s total):

URL	Score	LCP	CLS	TBT
example.com	100	839 ms	0	67 ms
python.org	77	4,336 ms	0.003	6 ms
developer.mozilla.org/en-US/	88	3,309 ms	0	0 ms
crawlee.dev	37	11,971 ms	0	2,507 ms
wikipedia.org	76	3,977 ms	0.001	117 ms
web.dev	82	2,864 ms	0.12	206 ms

Run 2, same URLs, one at a time a few minutes later (78 s total): example.com 100, python.org 77, MDN 90, crawlee.dev 47 with LCP 4,692 ms, wikipedia.org 82, web.dev 74.

That second run is the most useful result here:

• Lab numbers are noisy. crawlee.dev's LCP went from 12.0 s to 4.7 s and its score from 37 to 47 between two runs; web.dev moved 8 points. TBT stayed around 2.5 s both times, so the JavaScript cost is real; the LCP swing is not. For decisions, run each URL 3-5 times and take the median, or lean on field data.
• Parallelism on one machine skews local results. Running two Lighthouse instances at once competes for CPU, which inflates TBT and LCP. That's fine with the PSI API (Google's machines do the work, so concurrency is just about quota), but with --local keep workers low.
• CLS was the most stable metric. web.dev showed 0.12 in both runs, just over the 0.1 "good" threshold, which is exactly the kind of thing worth a ticket.

Getting field data without PSI

Since PSI's field data is scheduled to go away, it's worth querying CrUX directly. Same Google key (enable the "Chrome UX Report API" on the project), one POST per URL or origin, and it's much faster than PSI because nothing gets loaded:

r = requests.post(
    "https://chromeuxreport.googleapis.com/v1/records:queryRecord",
    params={"key": KEY},
    json={"url": "https://web.dev/", "formFactor": "PHONE"},  # or {"origin": "https://web.dev"}
    timeout=30,
)
m = r.json().get("record", {}).get("metrics", {})
lcp = m.get("largest_contentful_paint", {}).get("percentiles", {}).get("p75")
inp = m.get("interaction_to_next_paint", {}).get("percentiles", {}).get("p75")
cls = m.get("cumulative_layout_shift", {}).get("percentiles", {}).get("p75")  # a string like "0.05"

A 404 means CrUX doesn't have enough traffic for that URL; retry with origin instead. Without a key the endpoint answers 403 PERMISSION_DENIED ("Method doesn't allow unregistered callers"), as I confirmed on the same day.

Things that will bite you

• Normalize and dedupe URLs first. http://, https://, trailing slashes and ?utm_ variants all count as separate queries. The script dedupes exact duplicates only.
• Check final_url. If a URL redirects to a login page or the homepage, you're measuring the wrong page. The script keeps the final URL so you can spot it.
• Sample by template. For a 50,000-page store, 1,000 URLs spread across templates (home, category, product, article) tells you nearly as much as all 50,000, and it fits in the daily quota with room for mobile and desktop.
• Save the raw JSON if you'll want opportunities later. The CSV keeps the headline metrics; lighthouseResult.audits has the fix-it list (unused JavaScript, image sizes, render-blocking resources).

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Bulk PageSpeed Insights & Core Web Vitals Checker. You paste a list of URLs and get scores, lab metrics and the top opportunities per URL and device. It works without a Google key (it runs its own Lighthouse, $0.015 per audit) or with your own PSI key ($0.004 per audit, and that adds the CrUX field data), can run on a schedule, and is callable by API. The script above is the whole idea, though, so use whichever fits.