Compare PageSpeed Insights before and after a deploy, and fail CI on regressions

A release swaps a theme, a plugin or a script bundle, and nobody notices a page got slower until a client says so. Audit the same pages before and after, diff the numbers, and fail the pipeline on a regression. The hard part is defining "worse" when no two Lighthouse runs are identical.

Pick a URL list you can keep

• Use one page per template: home, a listing, a detail page, an article, any public funnel step.
• Pin specific items (one product, one post). A "latest posts" page changes between runs.
• Commit the list. Both runs need the same URLs, device strategy and categories.
• Keep it public. Google fetches pages from its own servers, so localhost and pages behind a login fail with http-error. Audit production or a public preview URL.

A pinned list has a blind spot: it cannot cover pages the release adds. Reader @omyvnss pointed this out. To catch them, extract the site's sitemap URLs before and after the release and compare. The Sitemap URL Extractor, which I also built, does the extraction. Add one page for each new kind of URL, such as a new path prefix. This only works if the sitemap updates with the deploy, and extraction is billed per URL ($0.0002), so on a large site it can cost more than the audits themselves.

Lighthouse is noisy, so set a tolerance

Two runs of the same page can differ by a few points with no deploy in between. Flag every drop and people soon ignore the check. Three habits keep it honest:

1. Measure the noise. Audit the unchanged site three times, note the largest swing per metric on your own pages, and set each tolerance above it. The numbers in the script below are placeholders, not recommendations.
2. Use absolute differences, not percentages: a percentage of a near-zero CLS baseline means nothing.
3. Confirm before failing. A breach is a candidate: rerun the list once and fail only if the same metric breaches again.

A tolerance hides real changes smaller than itself, so keep each release's result files and glance at the trend.

Gate on lab metrics, check field data later

Lab metrics come from Google's test device, so runs are comparable and they move the moment you deploy. Gate on labMetrics.lcpMs, labMetrics.tbtMs, labMetrics.cls and scores.performance.

Field data is real users from the Chrome UX Report, a 28-day 75th percentile, so a release from ten minutes ago has barely moved it. It cannot gate a deploy. It is null for low-traffic pages, and originFallback: true means origin-level numbers. Use it weeks later: rerun the list and compare fieldData.overallCategory.

Run the audits with the Actor

PageSpeed Insights Audit is an Apify Actor that calls the official PageSpeed Insights API v5 and returns one record per URL and strategy, with output fields documented as stable. Save this as input.json:

{
    "urls": [
        "https://www.example.com/",
        "https://www.example.com/products/",
        "https://www.example.com/blog/a-pinned-post"
    ],
    "strategy": "mobile",
    "categories": ["performance"]
}

The diff needs only the performance category. With "strategy": "both" each URL gets a mobile and a desktop record, compared separately. Pages Google cannot audit come back as success: false records with an errorType (dns, timeout, rate-limited and others), unbilled. The script must never read them as "no regression".

The diff script

Save this as diff.mjs (Node only). Exit codes: 0 clean, 1 confirmed regression, 2 some audits could not be compared, 64 bad usage. A metric regresses only if every successful "after" file is past its tolerance, which is how the rerun applies.

import { readFileSync } from "node:fs";

// Placeholder tolerances: replace them with what your own unchanged-site runs show.
const checks = [
  // label, value, which way is worse (1 = higher, -1 = lower), tolerance
  ["score", (r) => r.scores?.performance, -1, 5],
  ["LCP ms", (r) => r.labMetrics?.lcpMs, 1, 500],
  ["TBT ms", (r) => r.labMetrics?.tbtMs, 1, 200],
  ["CLS", (r) => r.labMetrics?.cls, 1, 0.05],
];

const [beforeFile, ...afterFiles] = process.argv.slice(2);
if (!afterFiles.length) {
  console.error("usage: node diff.mjs before.json after.json [rerun.json ...]");
  process.exit(64);
}
const load = (file) =>
  new Map(JSON.parse(readFileSync(file, "utf8")).map((r) => [`${r.strategy} ${r.url}`, r]));
const before = load(beforeFile);
const afters = afterFiles.map(load);
if (!before.size) {
  console.error("the before file has no records");
  process.exit(2);
}

let regressed = 0;
let noData = 0;
for (const [key, b] of before) {
  const runs = afters.map((m) => m.get(key));
  const good = runs.filter((r) => r?.success);
  const usable =
    b.success && good.length && checks.every(([, get]) => [b, ...good].every((r) => Number.isFinite(get(r))));
  if (!usable) {
    noData++;
    const why = [b, ...runs].map((r) => r?.errorType).filter(Boolean).join(" ");
    console.log(`NO DATA     ${key} ${why || "missing"}`);
    continue;
  }
  const bad = checks.filter(([, get, dir, tol]) => good.every((a) => dir * (get(a) - get(b)) > tol));
  for (const [label, get] of bad) console.log(`REGRESSION  ${key} ${label}: ${get(b)} -> ${get(good.at(-1))}`);
  if (bad.length) regressed++;
}
console.log(`${before.size} audits compared, ${regressed} regressed, ${noData} without data`);
process.exit(regressed ? 1 : noData ? 2 : 0);

Run it in CI

A short list fits in one synchronous call. Save this as audit.sh:

set -euo pipefail
curl -sSf -X POST \
  "https://api.apify.com/v2/acts/josh99smith~pagespeed-insights-audit/run-sync-get-dataset-items" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d @input.json -o "$1"

The synchronous endpoint gives up after 300 seconds. The Actor's docs estimate about 10 minutes per 100 URLs, so that covers roughly 50 audits. For longer lists, start the run with POST /v2/acts/josh99smith~pagespeed-insights-audit/runs, poll it, then read its dataset. In GitHub Actions, with the token stored as a secret:

jobs:
  release:
    runs-on: ubuntu-latest
    env:
      APIFY_TOKEN: ${{ secrets.APIFY_TOKEN }}
    steps:
      - uses: actions/checkout@v4
      - run: bash audit.sh before.json
      - run: ./deploy.sh # your own deploy step
      - run: bash audit.sh after.json
      - name: Diff, with one rerun to confirm
        run: |
          node diff.mjs before.json after.json || {
            bash audit.sh rerun.json
            node diff.mjs before.json after.json rerun.json
          }

If the first diff fails, the job audits again and diffs with both "after" files: a one-off spike passes, a repeat fails. Exit 2 also triggers the rerun, which clears transient quota or timeout failures; if it fails again the build stays red.

Cost and limits

• Price: $0.004 per successfully audited URL and strategy. Failed pages, invalid URLs and quota errors are free, and each run stops at the maximum cost you set.
• Worked example: 20 URLs on mobile, before and after, is 40 audits: $0.16. A confirmation rerun adds 20 audits ($0.08), so $0.24 at most. Adding desktop doubles both figures.
• Quota: the default built-in Google API key, and its daily quota, are shared by all users. When it runs out you get free rate-limited records, which the script reports as exit 2, not a pass. For CI, create your own free key (25,000 audits a day) and pass it as apiKey, built into input.json from a CI secret, not committed.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built this Actor. Source: github.com/josh99smith/pagespeed-insights-audit. PageSpeed Insights and Lighthouse are Google trademarks; the Actor is not affiliated with Google.

Comments

[Om Yaduvanshi]

the pinned URL list is the piece i'd be least confident about. one page per template covers the pages you already know, and the releases that hurt performance most tend to ship pages that didn't exist when the list was committed. a second check that diffs the sitemap before and after and flags anything the list doesn't cover would close that gap nicely.

[Joshua Smith]

Fair point, thanks. A pinned list only covers pages you knew about when you committed it, and a brand-new template is exactly where a regression can hide. Diffing the sitemap before and after the release is a good fix: flag any new URL pattern the list doesn't cover, then add one page per new template. I might write that up as a follow-up. (I also maintain a small sitemap-extractor Actor, if you'd rather not script the URL listing.)