<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:dc="http://purl.org/dc/elements/1.1/">
  <channel>
    <title>DEV Community: Websites for Tradespeople</title>
    <description>The latest articles on DEV Community by Websites for Tradespeople (@websitesfortradespeople).</description>
    <link>https://dev.to/websitesfortradespeople</link>
    <image>
      <url>https://media2.dev.to/dynamic/image/width=90,height=90,fit=cover,gravity=auto,format=auto/https:%2F%2Fdev-to-uploads.s3.us-east-2.amazonaws.com%2Fuploads%2Fuser%2Fprofile_image%2F4250645%2F02e64f5a-c114-4ac2-836d-75f44e48fabf.png</url>
      <title>DEV Community: Websites for Tradespeople</title>
      <link>https://dev.to/websitesfortradespeople</link>
    </image>
    <atom:link rel="self" type="application/rss+xml" href="https://dev.to/feed/websitesfortradespeople"/>
    <language>en</language>
    <item>
      <title>Auditing 43,738 sites with the PageSpeed Insights API: what to store, and what we threw away</title>
      <dc:creator>Websites for Tradespeople</dc:creator>
      <pubDate>Tue, 17 Nov 2026 22:31:14 +0000</pubDate>
      <link>https://dev.to/websitesfortradespeople/auditing-43738-sites-with-the-pagespeed-insights-api-what-to-store-and-what-we-threw-away-3hla</link>
      <guid>https://dev.to/websitesfortradespeople/auditing-43738-sites-with-the-pagespeed-insights-api-what-to-store-and-what-we-threw-away-3hla</guid>
      <description>&lt;p&gt;This summer we used Google's PageSpeed Insights API to run Lighthouse against 43,738 UK trade business websites (plumbers, electricians, roofers and so on), on both mobile and desktop settings. The results became a public report. This post is about the plumbing: what we stored, what we did not, and what we would do differently.&lt;/p&gt;

&lt;h2&gt;
  
  
  The setup
&lt;/h2&gt;

&lt;p&gt;Each site got two requests to the &lt;a href="https://developers.google.com/speed/docs/insights/rest/v5/pagespeedapi/runpagespeed" rel="noopener noreferrer"&gt;runPagespeed endpoint&lt;/a&gt; of the v5 API, one with &lt;code&gt;strategy=mobile&lt;/code&gt; and one with &lt;code&gt;strategy=desktop&lt;/code&gt;, asking for the Performance, Accessibility, Best Practices and SEO categories, with a 90 second timeout on our side. After a seven-site test on 30 July 2026, collection ran in two passes, 3 to 5 August and 30 August to 1 September. 31,030 of the 43,738 sites returned a usable result on at least one strategy.&lt;/p&gt;

&lt;p&gt;The collector was deliberately lean. From each response it kept the four category scores and six metric values (LCP, CLS, TBT, FCP, Speed Index and TTI), and nothing else.&lt;/p&gt;

&lt;p&gt;That was the main mistake.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 1: keep the whole response, or at least the metadata
&lt;/h2&gt;

&lt;p&gt;When we came to write the methodology, we wanted to state the exact test conditions. We could not, because we had thrown them away:&lt;/p&gt;

&lt;ul&gt;
&lt;li&gt;
&lt;strong&gt;Lighthouse version.&lt;/strong&gt; Never stored. The service moved to Lighthouse 13.0 in October 2025, and a spot check in September 2026 came back as 13.4.1, so our audits ran on some 13.x release. A point release can change how a score is calculated.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;The page actually audited.&lt;/strong&gt; We sent each listed address and Google followed redirects, but we never stored where it landed. In a later spot check, one of two sites redirected to a different host, with a run warning we would not have kept.&lt;/li&gt;
&lt;li&gt;
&lt;strong&gt;Device emulation.&lt;/strong&gt; We know which strategy each request used, not which emulated device and screen Google applied.&lt;/li&gt;
&lt;/ul&gt;

&lt;p&gt;At minimum, persist these per response:&lt;br&gt;
&lt;/p&gt;

&lt;div class="highlight js-code-highlight"&gt;
&lt;pre class="highlight plaintext"&gt;&lt;code&gt;analysisUTCTimestamp
lighthouseResult.lighthouseVersion
lighthouseResult.fetchTime
lighthouseResult.requestedUrl
lighthouseResult.mainDocumentUrl
lighthouseResult.finalDisplayedUrl   (older responses and the API reference use finalUrl)
lighthouseResult.runWarnings
lighthouseResult.runtimeError
lighthouseResult.configSettings
lighthouseResult.environment          (includes benchmarkIndex, a rough measure of the host's speed)
lighthouseResult.categories[*].score
lighthouseResult.audits[&amp;lt;metric&amp;gt;].numericValue
loadingExperience
originLoadingExperience
&lt;/code&gt;&lt;/pre&gt;

&lt;/div&gt;



&lt;p&gt;Better still, keep the raw response, gzipped. Most of a &lt;code&gt;lighthouseResult&lt;/code&gt; by size is base64 screenshots (the &lt;code&gt;final-screenshot&lt;/code&gt; and &lt;code&gt;screenshot-thumbnails&lt;/code&gt; audits and the top-level &lt;code&gt;fullPageScreenshot&lt;/code&gt;), so stripping those first keeps storage manageable; measure your own sizes rather than trusting anyone's estimate.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 2: we threw away free field data
&lt;/h2&gt;

&lt;p&gt;The two fields at the bottom of that list are the ones we most regret. Every PageSpeed Insights response can include &lt;code&gt;loadingExperience&lt;/code&gt; and &lt;code&gt;originLoadingExperience&lt;/code&gt;: Chrome UX Report field data for the URL and origin, where Google has enough real-visitor data to publish it. Our collector discarded both, so our report is lab data only, and we cannot say how many of these sites had field data at all.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 3: some settings are not in the response at all
&lt;/h2&gt;

&lt;p&gt;Even a complete response would not have told us the throttling setup. The &lt;code&gt;configSettings&lt;/code&gt; block the API returns carries form factor, locale, channel and the category list, with no throttling method, CPU slowdown or network values.&lt;/p&gt;

&lt;p&gt;What Google publishes instead: the &lt;a href="https://developers.google.com/speed/docs/insights/v5/about" rel="noopener noreferrer"&gt;About PageSpeed Insights&lt;/a&gt; page describes the mobile test as "a mid-tier device (Moto G4) device on a mobile network" and the desktop test as "an emulated-desktop with a wired connection". Lighthouse's &lt;a href="https://github.com/GoogleChrome/lighthouse/blob/main/docs/throttling.md" rel="noopener noreferrer"&gt;throttling documentation&lt;/a&gt; says its default simulated throttling "matches the setup of PageSpeed Insights". If that is what ran, the timing figures are model estimates for a target connection, not load times over a real one. That is the provider's description, not something we observed.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 4: an error is about your request, not the website
&lt;/h2&gt;

&lt;p&gt;We logged three kinds of failure: timeouts, connection errors and HTTP 500s. None says anything reliable about the site. A timeout means our own call did not finish in 90 seconds, the connection errors came in a few short bursts on two days, and the body of a 500 was not kept, so nothing is known about its cause.&lt;/p&gt;

&lt;p&gt;The API's own 400-class responses (FAILED_DOCUMENT_REQUEST, NO_FCP, NOT_HTML) are different: Google could not fetch, render or parse the page. Those went to a separate sheet: 11,037 of the 43,765 rows written across both sheets, or 25.2%. (There are more rows than sites because of the retries and repeats in Lesson 5.)&lt;/p&gt;

&lt;p&gt;On quotas: rate-limited requests were dropped and retried in a later run. If you do this at scale, check your Google Cloud project's Quotas page for the current daily and per-minute limits rather than relying on figures quoted online, and retry 429s with backoff.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 5: small input bugs make duplicates
&lt;/h2&gt;

&lt;p&gt;An "already audited" check stopped repeats, but five listed addresses had a stray space after the scheme separator (&lt;code&gt;http:// example&lt;/code&gt;), slipped past it and were audited again on later runs. Deduplication kept the most recent row per domain, 27 rows in all. Normalise URLs before you build a skip set.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 6: one run per site is noisy, and batches differ
&lt;/h2&gt;

&lt;p&gt;Every figure rests on a single Lighthouse run per strategy. Lighthouse's &lt;a href="https://github.com/GoogleChrome/lighthouse/blob/main/docs/variability.md" rel="noopener noreferrer"&gt;variability guide&lt;/a&gt; says "The median Lighthouse score of 5 runs is twice as stable as 1 run". Across 30,000 sites a median is robust to that noise; for any single site, or a site sitting near a threshold, it is not.&lt;/p&gt;

&lt;p&gt;Our two passes also differed in date, city set and trade mix at once. So before anything became a headline, we recalculated it inside each pass separately and kept only figures that held in both. Trade in particular was so tangled with pass (several trades sit almost entirely in one) that per-trade figures could not be tested that way, so the report shows them only as a descriptive table and quotes none.&lt;/p&gt;

&lt;p&gt;Other rules we applied: medians rather than means for timing metrics, because of the long right tails; one quantile definition pinned (R type 7, which is also numpy and pandas' default "linear" method) so a figure cannot move with a library change; no median shown for fewer than 100 sites; and the sample size printed beside every figure.&lt;/p&gt;

&lt;h2&gt;
  
  
  Lesson 7: lab is not field
&lt;/h2&gt;

&lt;p&gt;Many published speed studies use the &lt;a href="https://developer.chrome.com/docs/crux" rel="noopener noreferrer"&gt;Chrome UX Report&lt;/a&gt;, which is field data from real Chrome users and covers only pages with enough traffic. Ours is one lab run per strategy on long-tail local business sites, many of which may not meet CrUX's eligibility bar; we did not check. The two kinds of number are not comparable, and neither corrects the other.&lt;/p&gt;

&lt;h2&gt;
  
  
  What came out
&lt;/h2&gt;

&lt;p&gt;With those caveats: in the mobile lab test, half of the 30,399 sites that returned a mobile load time took 6.7 seconds or more to reach largest contentful paint, and 72.5% took over 4 seconds. Of the 29,969 sites with an LCP on both strategies, 99.5% recorded a higher one on mobile, a median of 5.1 seconds apart. That last figure is close to what the test design predicts, since the mobile profile simulates a slower device and network; it is a confirmation more than a discovery.&lt;/p&gt;

&lt;p&gt;The full method, including a table of which settings were retained, partly retained or lost, and the anonymised row-level CSV under CC BY 4.0, are in the report: &lt;a href="https://www.websitesfortradespeople.co.uk/research/state-of-uk-trades-websites-2026" rel="noopener noreferrer"&gt;The State of UK Trades Websites 2026&lt;/a&gt;.&lt;/p&gt;

&lt;p&gt;Written with AI assistance.&lt;/p&gt;

</description>
      <category>webperf</category>
      <category>lighthouse</category>
      <category>webdev</category>
      <category>datascience</category>
    </item>
  </channel>
</rss>
