How to run Lighthouse on every page in your sitemap (without a PageSpeed API key)

Lighthouse is great for one page at a time. When you're responsible for a whole site, say 50 docs pages, a blog, or a client's product catalogue, the questions change. Which pages are slow? Which ones are missing a meta description? Did last week's release make things worse?

Doing that by hand in DevTools gets old quickly. This post covers three ways to audit every URL in a sitemap, what each costs in time and money, and what the output looks like from a real run.

Option 1: Lighthouse CLI in a loop (free, local)

If you have Node and Chrome installed, you can do it yourself:

npm install -g lighthouse
curl -s https://www.example.com/sitemap.xml \
  | grep -o '<loc>[^<]*' | sed 's/<loc>//' | head -20 \
  | while read url; do
      lighthouse "$url" --quiet --chrome-flags="--headless=new" \
        --only-categories=performance,seo \
        --output=json --output-path="./reports/$(echo "$url" | md5sum | cut -c1-8).json"
    done

This works and costs nothing. The catches:

• Sitemap indexes (a sitemap of sitemaps) and .gz sitemaps need extra handling.
• You end up with a folder of 300–500 KB JSON files and still have to pull the scores and metrics out of each one.
• Scores depend on your machine and network. Your laptop on fibre isn't a mid-range phone on 4G.
• Run audits one at a time. Running several in parallel on one machine makes them compete for CPU and drags performance scores down.

For a one-off check of a small site, this is honestly the best option.

Option 2: PageSpeed Insights API (free, with limits)

Google's PSI API runs Lighthouse for you and also returns Chrome UX Report field data when a page has enough traffic. You need an API key, you're rate-limited, and heavy use runs into quota errors. It's a good choice for a few hundred URLs a day if you're happy to write the sitemap parsing and the retry logic.

Option 3: a hosted bulk auditor

If you want a table of results without maintaining scripts, you can run Lighthouse in the cloud. Full disclosure: I built a small tool for this on Apify, the Lighthouse Bulk Auditor. It's paid per successful audit (currently US$0.015 per page per device), so weigh that against Options 1 and 2. The rest of this walkthrough uses it because that's where the sample output below comes from.

Step 1: Point it at a sitemap

Open the Actor, click Try for free, and switch to the JSON input. This is the exact input I used for the sample below:

{
  "sitemapUrl": "https://docs.apify.com/sitemap.xml",
  "maxSitemapUrls": 3,
  "strategies": ["desktop"],
  "maxOpportunities": 3
}

• sitemapUrl can be a normal sitemap, a sitemap index or a .gz file.
• maxSitemapUrls caps how many pages are audited. Start small, check the output, then raise it.
• strategies is mobile, desktop or both. Both doubles the number of audits.
• maxOpportunities is how many "top fixes" to keep per page.

That run audited 3 pages in about 98 seconds in total. A single audit usually takes 10–40 seconds depending on the page.

Step 2: Read the output

You get one row per page per device. This is a real row from that run, trimmed slightly:

{
  "url": "https://docs.apify.com/api",
  "strategy": "desktop",
  "status": "ok",
  "scores": { "performance": 71, "accessibility": 80, "bestPractices": 69, "seo": 85 },
  "firstContentfulPaintMs": 570,
  "largestContentfulPaintMs": 1366,
  "totalBlockingTimeMs": 455,
  "cumulativeLayoutShift": 0.005,
  "speedIndexMs": 2355,
  "serverResponseTimeMs": 17,
  "topOpportunities": [
    { "id": "mainthread-work-breakdown", "title": "Minimize main-thread work", "estimatedSavingsMs": 450 },
    { "id": "bootup-time", "title": "Reduce JavaScript execution time", "estimatedSavingsMs": 450 },
    { "id": "render-blocking-insight", "title": "Render-blocking requests", "estimatedSavingsMs": 350 }
  ],
  "failedSeoOrA11yAudits": [
    "Document does not have a meta description",
    "Image elements do not have `[alt]` attributes",
    "Background and foreground colors do not have a sufficient contrast ratio."
  ],
  "lighthouseVersion": "13.6.0"
}

The fields I find most useful:

• largestContentfulPaintMs, cumulativeLayoutShift, totalBlockingTimeMs: the lab versions of the Core Web Vitals. TBT stands in for INP in lab testing.
• topOpportunities: Lighthouse's own estimate of what each fix would save, sorted by time saved. It's a decent to-do list, but treat the numbers as estimates.
• failedSeoOrA11yAudits: the plain-English names of failing binary checks. This is often the quickest win, because a missing meta description takes a minute to fix.
• status / errorMessage: pages that couldn't be loaded show up as error rows with the reason, so they don't silently drop out.

Export to CSV or Excel and sort by largestContentfulPaintMs or scores.performance to find your worst pages.

Step 3: Narrow it down when you only care about one thing

A full audit isn't always what you need. Limiting categories makes runs faster and the output easier to scan.

Performance only, with a pass/fail budget:

{
  "startUrls": [{ "url": "https://apify.com" }, { "url": "https://docs.apify.com" }],
  "strategies": ["mobile"],
  "categories": ["performance"],
  "minPerformanceScore": 50
}

Each row gets passedPerformanceBudget: true/false. In my test, one page scored 64 and passed. The other scored 35 with a 23-second LCP on emulated slow 4G and failed. That's exactly the kind of page this is meant to surface.

SEO only ("categories": ["seo"]) or accessibility only ("categories": ["accessibility"]) took about 10–12 seconds per page in my runs. Columns for categories you didn't ask for come back as null.

Step 4: Automate it

The useful version of this is a scheduled run:

1. Save your input as a task.
2. Add a schedule, for example weekly.
3. Set minPerformanceScore and add a webhook or Slack integration so you hear about pages that fail the budget.

You can also call it from a script or CI with Apify's run-sync-get-dataset-items endpoint, which starts a run and returns the rows in one request. Synchronous calls time out after 5 minutes, so use the normal asynchronous run endpoint for large sitemaps.

Caveats worth knowing (whichever option you pick)

• Scores vary between runs by a few points, because of network jitter, ads and third-party scripts. Compare trends over several runs, not single numbers.
• Lab data isn't field data. The CrUX numbers at the top of PageSpeed Insights come from real Chrome users. Lighthouse lab data is reproducible and works for pages with no traffic, but the two will differ.
• Location matters. Cloud audits run from wherever the servers are (US data centres in this case), so serverResponseTimeMs for a site hosted in Australia or Europe will look worse than from a local machine.
• Only audit sites you own or have permission to test, and keep volumes reasonable.
• Lighthouse isn't a full accessibility audit. It catches common, automatable issues such as contrast, alt text and labels, but not everything WCAG covers.

Wrapping up

For a handful of pages, run the Lighthouse CLI locally. For a few hundred URLs a day when you're happy to write some glue code, the PSI API is free. When you want a whole sitemap turned into a spreadsheet, or a scheduled check that alerts you, a hosted runner saves the plumbing.

If you try the Actor and something doesn't work, or there's a field you wish it had, open an issue on its Issues tab. I'd genuinely like to know.