DEV Community

Cover image for Audit Core Web Vitals for a whole sitemap (and stop confusing page and origin data)
Fjord Labs
Fjord Labs

Posted on

Audit Core Web Vitals for a whole sitemap (and stop confusing page and origin data)

#webdev #python #seo #performance

PageSpeed Insights is great for one URL. For a 200-page site, it's an afternoon of copy-pasting. Here's how I audit an entire sitemap in one go, mobile and desktop, and get a spreadsheet of Lighthouse scores, real-user Core Web Vitals and the top fixes per page.

A trap first: page vs origin field data

When Google has no real-user (CrUX) data for a specific page, PageSpeed Insights quietly shows the origin's data instead. So a slow, low-traffic page can look "fast" because the homepage is fast. Keep the two apart:

  • fieldDataPage: real users of this URL (often null on small pages)
  • fieldDataOrigin: real users across the whole site

The code

I use a hosted Actor I maintain on Apify (Bulk PageSpeed & Core Web Vitals Audit). With the Python client (pip install apify-client pandas):

from apify_client import ApifyClient
import pandas as pd

client = ApifyClient("<YOUR_APIFY_TOKEN>")
run = client.actor("fjordlabs/pagespeed-audit").call(run_input={
    "sitemapUrls": ["https://www.example.com/sitemap.xml"],
    "maxUrls": 200,
    "strategies": ["mobile", "desktop"],
    "alertBelowPerformanceScore": 50,
})

rows = list(client.dataset(run["defaultDatasetId"]).iterate_items())
df = pd.json_normalize(rows)
cols = ["url", "strategy", "scores.performance", "lab.lcpMs", "lab.cls",
        "fieldDataPage.overall", "fieldDataOrigin.overall", "coreWebVitalsPassed"]
print(df[cols].sort_values("scores.performance").head(15))
Enter fullscreen mode Exit fullscreen mode

Sitemap indexes are followed and URLs de-duplicated, so you can point it at the root sitemap.

Find the fixes that matter most

Each row has opportunities with estimated savings. Summing them across the site shows which single fix helps the most pages:

opps = pd.json_normalize(rows, "opportunities", ["url"])
print(opps.groupby("title")["savingsMs"].agg(["count", "sum"]).sort_values("sum", ascending=False).head())
Enter fullscreen mode Exit fullscreen mode

Usually it's "Reduce unused JavaScript" or "Eliminate render-blocking resources" on a shared template, so one fix lifts dozens of pages.

Weekly monitoring

Schedule it weekly. Pages under your threshold come back with alert: true, which you can route to Slack or email with Apify's integrations.

Cost: $0.004 per page audit, and failed audits are free. A 200-page site on mobile + desktop is $1.60.

Disclosure: I built and maintain this Actor. Happy to answer questions in the comments. Links to Apify include my referral code.

Top comments (0)