"Is there a tool to run PageSpeed Insights on all my pages at once?" comes up constantly in SEO and web dev forums. The answer is yes, Google gives you an API for exactly this. The harder part is everything around it: finding the pages, staying inside quota, handling 40-second responses, and reading the results without being fooled by noise.
These are my notes from building a bulk auditor on top of the PageSpeed Insights (PSI) and Chrome UX Report (CrUX) APIs. At the end there is the hosted version I published on Apify. Disclosure: I built it.
The API call
One audit is one GET:
GET https://www.googleapis.com/pagespeedonline/v5/runPagespeed
?url=https://web.dev/
&strategy=mobile
&category=performance&category=seo&category=accessibility&category=best-practices
&key=YOUR_KEY
Things that are not obvious from the docs:
- One strategy per call. Mobile and desktop are two audits. Mobile is what Google uses for ranking, so start there.
-
Categories must be listed. Without
categoryparams you only get performance. -
The response is huge. A full Lighthouse result runs to megabytes. Use the
fieldsparameter to keep only what you need, for exampleid,loadingExperience,originLoadingExperience,lighthouseResult(finalUrl,categories,audits,runtimeError). It cuts transfer and parse time a lot. - It is slow. Each call runs a real Lighthouse audit on Google's side: 10 to 40 seconds, and up to 2 minutes for heavy pages. Set your HTTP timeout accordingly and run audits in parallel.
- Get a key. Without one you get throttled quickly. With a free key from Google Cloud you typically get 25,000 queries a day. Check your project's quota page for the actual limits.
Lab data and field data are different things
Each PSI response has two kinds of data, and mixing them up is the most common mistake I see in bulk reports.
Lab data is lighthouseResult: the Lighthouse performance score (0 to 100) and metrics like LCP, CLS, TBT, FCP and Speed Index, measured in one simulated load on Google's servers. Every page gets lab data.
Field data is loadingExperience (this URL) and originLoadingExperience (the whole site): 75th percentile values for LCP, INP and CLS from real Chrome users over the last 28 days. This is what the Core Web Vitals assessment uses. Many pages have none, because Google only publishes it for pages with enough Chrome traffic. When a URL has too little, you often still get origin-level data.
Two consequences for a bulk report:
- Do not compute "passes Core Web Vitals" from lab scores. The assessment is: LCP at most 2.5 s, INP at most 200 ms and CLS at most 0.1, all at p75 from field data. If any of the three is missing, the honest answer is "no data", not "pass".
- Record which scope the field data came from (
urlororigin). Otherwise a quiet product page "passes" because the homepage is fast.
Lab scores move between runs
Run the same URL twice and the performance score can differ by 5 to 10 points. Network timing and server response vary, even on Google's own sites. For one-off reports, that is fine if you say so. For monitoring, it matters a lot: a naive "alert if the score dropped" rule fires every day.
What worked for me in monitor mode:
- Alert on a performance drop only above a threshold (10 points by default).
- Alert when a page flips from passing to failing the field-data assessment.
- Alert when a field metric gets clearly worse and is no longer "good".
- Alert when a page that worked last time now errors.
Field data changes slowly (it is a 28-day window), so field-based alerts are much less noisy than lab-based ones.
Finding the pages
"Audit my site" means you need the URL list first. The reliable order:
-
robots.txt, forSitemap:lines. -
/sitemap.xmlif robots.txt lists nothing. - Follow sitemap indexes into child sitemaps, including
.xml.gzfiles.
Then sample. A 20,000-URL sitemap is mostly the same few templates (product, category, blog post). Auditing 25 to 50 pages spread across the site's sections, always including the homepage, tells you more per dollar than auditing the first 50 URLs in the sitemap, which are often all from one section.
Errors that look like results
Some PSI responses have scores but describe the wrong page:
- The URL returns 404 or 500. Lighthouse happily audits the error page.
- The URL redirects to a different site (an expired domain, a login wall).
- Lighthouse fails with a
runtimeErrorlikeNO_FCP(no content painted). That one is sometimes transient and worth one retry.
I turn all of these into error rows with the reason instead of a score. A bulk report where 30 of 200 rows are secretly 404 pages is worse than useless.
Field data only, at 100 URLs a minute
If you only need real-user Core Web Vitals, skip Lighthouse entirely and call the CrUX API (chromeuxreport.googleapis.com/v1/records:queryRecord). It answers in well under a second, so you can check about 100 URLs a minute. Useful for screening a large list before running full audits on the failing pages.
Making the output usable
The rows people actually filter on, in my experience:
-
performanceScore,seoScore,accessibilityScore,bestPracticesScore -
cwvVerdict(pass,fail,no-data) andoriginCwvVerdict -
fieldLcpP75Ms,fieldInpP75Ms,fieldClsP75with their ratings -
topOpportunity: the single biggest Lighthouse fix by estimated time saved -
psiReportUrl: a link to Google's own report for that page, for whoever has to fix it
Plus one summary row per site: average and median score, Core Web Vitals pass rate, the 5 worst pages and the most common fixes across pages. That is usually the part that ends up in a client deck.
The hosted version
The actor is Bulk PageSpeed Insights & Core Web Vitals Checker on Apify. Give it URLs or a domain. It does sitemap discovery, parallel audits, the field data logic above, error rows, a site summary, and optional monitoring with regression alerts.
Pricing: $5 per 1,000 audits on the shared Google key, $2 per 1,000 with your own free key (which also gives you your own daily quota), $1 per 1,000 for field-only checks. Errors, summaries and monitor comparisons are free. Runs on the shared key are capped at 1,000 audits each.
from apify_client import ApifyClient
client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("zrk1212/pagespeed-bulk-auditor").call(run_input={
"startDomains": ["example.com"],
"maxPagesPerSite": 25,
"strategy": "mobile",
"includeFieldData": True,
"includeSiteSummary": True,
# "googleApiKey": "YOUR_GOOGLE_KEY", # optional: lower price, your own quota
})
for row in client.dataset(run["defaultDatasetId"]).iterate_items():
if row["rowType"] == "audit":
print(row["performanceScore"], row["cwvVerdict"], row["requestedUrl"], row.get("topOpportunity"))
elif row["rowType"] == "summary":
print("site pass rate:", row["cwvPassRatePct"], "%")
If you want to build your own instead, everything above works with a free key and a thread pool. The parts worth copying are the error handling and the field data rules. Those are where most bulk reports go wrong.
Top comments (0)