Run PageSpeed Insights on hundreds of URLs and get one spreadsheet back

Joshua Smith — Wed, 23 Sep 2026 13:09:47 +0000

pagespeed.web.dev is fine for one page. For a site audit you want every template, every landing page, mobile and desktop, in a table you can sort. Google exposes the same engine as an API; here is how to use it in bulk without babysitting a script.

The API behind pagespeed.web.dev

The PageSpeed Insights API v5 takes a URL and a strategy (mobile or desktop), runs Lighthouse in Google's data centre and returns a large JSON report: category scores, lab metrics (LCP, CLS, TBT, FCP, Speed Index, TTI), CrUX field data when the page has enough real-user traffic, and every audit with its details. It is free with an API key, up to 25,000 requests a day.

The parts that turn into work: each call takes 10 to 30 seconds, so you need concurrency and retries; the report is several hundred kilobytes and the useful numbers are buried five levels deep; quota errors and pages Lighthouse cannot load need handling; and you want mobile and desktop side by side.

The two-minute version

PageSpeed Insights Audit is an open-source Apify Actor that wraps the official API and reduces each report to the numbers people actually use.

Paste your URLs into Page URLs, one per line.
Pick the Device strategy: mobile (what Google ranks on), desktop, or both.
Optionally switch on Include audit details for the top opportunities and failing audits. Click Start.

100 URLs on mobile finish in about ten minutes. Export as CSV or Excel, or push to Google Sheets.

From code

curl -X POST "https://api.apify.com/v2/acts/josh99smith~pagespeed-insights-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://example.com", "https://www.wikipedia.org"], "strategy": "both", "includeAuditDetails": true }'

What you get back

One record per URL and strategy (trimmed):

{
    "url": "https://example.com",
    "strategy": "mobile",
    "success": true,
    "scores": { "performance": 72, "accessibility": 88, "bestPractices": 96, "seo": 100 },
    "labMetrics": { "fcpMs": 2144, "lcpMs": 3413, "cls": 0.042, "tbtMs": 420, "speedIndexMs": 3187, "ttiMs": 4902 },
    "fieldData": {
        "lcpMs": { "percentile": 2371, "category": "FAST" },
        "inpMs": { "percentile": 214, "category": "AVERAGE" },
        "cls": { "percentile": 0.05, "category": "FAST" },
        "overallCategory": "FAST",
        "originFallback": false
    },
    "opportunities": [
        { "id": "render-blocking-resources", "title": "Eliminate render-blocking resources", "savingsMs": 780 },
        { "id": "unused-javascript", "title": "Reduce unused JavaScript", "savingsMs": 450, "savingsBytes": 120832 }
    ],
    "lighthouseVersion": "12.8.2"
}

fieldData.originFallback: true tells you Google had no page-level CrUX data and used origin-level numbers instead, which matters when you compare pages.

Cost and limits

$0.004 per successful audit (100 URLs on both strategies = 200 audits = $0.80). Pages Google cannot load, invalid URLs and quota errors cost nothing; each run stops at the cost cap you set. Two Lighthouse runs of the same page can differ by a few points; that is Lighthouse, not the Actor, and it is why the field data column exists.

Use it from an AI agent

Add https://mcp.apify.com?tools=josh99smith/pagespeed-insights-audit to your MCP client and ask "audit example.com on mobile and list the top opportunities".

Disclosure: I built this Actor. Source: github.com/josh99smith/pagespeed-insights-audit. PageSpeed Insights and Lighthouse are Google trademarks; the Actor is not affiliated with Google.

Find out what any website is built with, in bulk, without a Wappalyzer subscription

Joshua Smith — Tue, 22 Sep 2026 13:30:12 +0000

You have a list of a few hundred domains and you want to know which run WordPress, which are on Shopify, who uses HubSpot, who is behind Cloudflare. The browser extensions do this one tab at a time. Wappalyzer's API is subscription-only and BuiltWith starts at $295 a month. Here is a way to do it for less than a cent per site.

How technology detection works

Every technology leaves traces in a page: a Set-Cookie: PHPSESSID, a <meta name="generator" content="WordPress 6.6">, a script URL under /wp-content/, a Server: cloudflare header, an x-shopify-stage header. The open-source webappanalyzer project maintains 7,600 such fingerprints (it is the community continuation of the original Wappalyzer rule set). Matching them is mostly regular expressions against headers, cookies, meta tags, script sources and the DOM, plus a step that resolves implied technologies (WordPress implies PHP and MySQL) and removes contradictions.

Doing it yourself means fetching each page politely, handling redirects and timeouts, keeping the fingerprint bundle fresh, and implementing the implication logic. That is a weekend of work and then maintenance.

The two-minute version

Website Tech Stack Detector is an Apify Actor that does exactly the above over plain HTTP (no browser), and I published the source under GPL-3.0 at github.com/josh99smith/tech-stack-detector.

Open the Actor page and paste your URLs into Website URLs, one per line. The scheme is optional.
Optionally turn on Include detection evidence to see which header, cookie or script triggered each match.
Click Start. Download the results as JSON, CSV or Excel, or send them to Google Sheets from the Integrations tab.

From code

One HTTP call runs the Actor and returns the dataset:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~tech-stack-detector/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{ "urls": ["https://www.shopify.com", "wordpress.org", "https://www.nytimes.com"], "detectViaDns": true }'

Python, with the official client:

from apify_client import ApifyClient

client = ApifyClient("<YOUR_API_TOKEN>")
run = client.actor("josh99smith/tech-stack-detector").call(
    run_input={"urls": ["https://www.shopify.com", "wordpress.org"], "detectViaDns": True}
)
for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["url"], item.get("byCategory", {}).get("CMS"), item.get("byCategory", {}).get("CDN"))

What you get back

One record per site (trimmed):

{
    "url": "https://wordpress.org/",
    "success": true,
    "statusCode": 200,
    "title": "Blog Tool, Publishing Platform, and CMS",
    "technologyCount": 13,
    "technologies": [
        { "name": "WordPress", "categories": ["CMS", "Blogs"], "version": "7.2", "confidence": 100 },
        { "name": "Nginx", "categories": ["Web servers", "Reverse proxies"], "version": null, "confidence": 100 },
        { "name": "Google Tag Manager", "categories": ["Tag managers"], "version": null, "confidence": 100 }
    ],
    "byCategory": { "CMS": ["WordPress"], "Web servers": ["Nginx"], "Tag managers": ["Google Tag Manager"] },
    "server": "nginx",
    "responseTimeMs": 474
}

Sites that cannot be reached come back as { "success": false, "errorType": "dns" | "timeout" | "blocked" | ... } so nothing silently disappears from your list.

Cost and limits

Pricing is pay-per-event: $0.008 per successfully analysed site, nothing for failures, no start fee, and a cost cap you set per run. A thousand domains cost about $8. Apify's free plan includes $5 of monthly usage, which covers roughly 600 sites before you pay anything.

The honest limitation: detection is HTTP-only. Technologies that only reveal themselves after JavaScript runs in a real browser are missed. In practice CMS, ecommerce platform, CDN, hosting, analytics and tag managers are almost always visible in the raw HTML and headers; some client-side widgets are not.

Use it from an AI agent

The Actor is a tool in the Apify MCP server. Add https://mcp.apify.com?tools=josh99smith/tech-stack-detector to Claude Desktop, Cursor or Claude Code and ask "what is shopify.com built with?".

Disclosure: I built this Actor. Bugs and misdetections go in the Issues tab on the Actor page or as GitHub issues; fingerprint fixes belong upstream at enthec/webappanalyzer.

DEV Community: Joshua Smith

Run PageSpeed Insights on hundreds of URLs and get one spreadsheet back

The API behind pagespeed.web.dev

The two-minute version

From code

What you get back

Cost and limits

Use it from an AI agent

Find out what any website is built with, in bulk, without a Wappalyzer subscription

How technology detection works

The two-minute version

From code

What you get back

Cost and limits

Use it from an AI agent