DEV Community

Cover image for Audit every page template on a site from its sitemap, without auditing thousands of pages
Joshua Smith
Joshua Smith

Posted on

Audit every page template on a site from its sitemap, without auditing thousands of pages

#webperf #seo #python #webdev

A shop with 2,000 URLs may have six layouts: home, category, product, blog post, blog index and static page. Every product page shares the same scripts and image gallery, so if one is slow, roughly all of them are. Auditing all 2,000 tells you the same thing 2,000 times; auditing three per template tells you which layout to fix first and how many pages the fix reaches. This post gets the URLs from the sitemaps, groups them into templates by path pattern, audits a sample and ranks the templates. Public pages and public data only.

Why this is harder than it looks

  • URLs do not label their template. /pages/about and /pages/stores/london may be different layouts. Path patterns are a first guess, not the truth.
  • Sitemaps are uneven. On the Shopify store I tested, category pages were 61 percent of the sitemap and the blog index had two URLs. A fixed share per group means hundreds of category audits and no blog-index audit, so sample a fixed number per template.
  • A URL cap can starve a template. Sitemap files are read in order. My first run, capped at 1,500 URLs, stopped inside the second collections file and returned no product pages at all.
  • Lab scores vary by a few points between runs. One page is an anecdote; three give a median and show whether the template or one heavy page is the problem.

The method

  1. Pull every URL from the sitemaps.
  2. Template = first path segment plus one * per remaining segment: /products/runner is /products/*, /blogs/news/launch is /blogs/*/*. Add rules where the platform nests layouts (on Shopify /collections/*/products/* is a product page).
  3. Read the groups and split any that mix two layouts.
  4. Audit three pages per template: the most recently modified (the template as it is today) and two random ones.
  5. Rank by median mobile performance score, weighed by how many pages each template covers.

Step 1: get the URLs

Sitemap URL Extractor finds the sitemaps through robots.txt and common paths, follows indexes and .xml.gz files, and returns one record per URL with its lastmod. Paste the site into Website or sitemap URLs, or send:

{
    "urls": ["https://shop.example"],
    "maxUrlsPerSite": 20000,
    "excludePatterns": ["*.pdf", "*.xml"]
}
Enter fullscreen mode Exit fullscreen mode

maxUrlsPerSite defaults to 5,000; set it above the site's size so no template is cut off. If you do not know the size, a run with List sitemap files only (outputSitemapsOnly: true) returns each sitemap file with its urlCount. From the API, with the JSON saved as sitemap-input.json:

curl -X POST "https://api.apify.com/v2/acts/josh99smith~sitemap-url-extractor/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" -d @sitemap-input.json -o sitemap.json
Enter fullscreen mode Exit fullscreen mode

Step 2: group into templates and pick a sample

Save this as templates.py (Python 3.8 or later, no packages). sample prints each template with its page count and example paths, and writes templates.json (the grouping) and input.json (the URLs to audit). report comes later.

import json
import random
import re
import sys
from collections import Counter, defaultdict
from statistics import median
from urllib.parse import urlsplit

PER_TEMPLATE = 3  # pages to audit per template
RULES = [  # (regex on the path, template name); first match wins, checked before the default
    (r"^/collections/[^/]+/products/", "/collections/*/products/*"),
]
SKIP = re.compile(r"\.(xml|txt|md|pdf|jpe?g|png|gif|webp|svg|zip)$", re.I)


def template(url):
    path = urlsplit(url).path.rstrip("/") or "/"
    for pattern, name in RULES:
        if re.search(pattern, path):
            return name
    parts = path.strip("/").split("/")
    if path == "/":
        return "/"
    if len(parts) == 1:
        return "/*"  # top-level pages: /about, /contact, ...
    return "/" + parts[0] + "/*" * (len(parts) - 1)


def sample(sitemap_file):
    groups = defaultdict(list)
    for r in json.load(open(sitemap_file, encoding="utf-8")):
        if r.get("success") is not False and not SKIP.search(urlsplit(r["url"]).path):
            groups[template(r["url"])].append(r)
    rng, picked, sizes = random.Random(42), {}, {}
    for name, rows in sorted(groups.items(), key=lambda kv: -len(kv[1])):
        rows.sort(key=lambda r: r["lastmod"] or "", reverse=True)
        chosen = rows[:1] + rng.sample(rows[1:], min(PER_TEMPLATE - 1, len(rows) - 1))
        sizes[name] = len(rows)
        picked.update({r["url"]: name for r in chosen})
        print(f"{name:28} {len(rows):6} pages, auditing {len(chosen)}, e.g. "
              + " ".join(urlsplit(r["url"]).path for r in chosen[:2]))
    json.dump({"sizes": sizes, "pages": picked}, open("templates.json", "w"), indent=1)
    json.dump({"urls": list(picked), "strategy": "mobile", "categories": ["performance"],
               "includeAuditDetails": True}, open("input.json", "w"), indent=4)
    print(f"\n{len(picked)} URLs written to input.json from {sum(sizes.values())} pages")


def report(results_file):
    t = json.load(open("templates.json", encoding="utf-8"))
    by_t, failed = defaultdict(list), []
    for r in json.load(open(results_file, encoding="utf-8")):
        name = t["pages"].get(r["url"])
        if name and r["success"]:
            by_t[name].append(r)
        elif name:
            failed.append(f"{r['url']} ({r['errorType']})")
    rows = []
    for name, recs in by_t.items():
        lab = lambda k: median(r["labMetrics"][k] for r in recs if r["labMetrics"][k] is not None)
        field = [r["fieldData"] for r in recs if r["fieldData"] and not r["fieldData"]["originFallback"]]
        fixes = Counter(o["title"] for r in recs for o in r.get("opportunities", [])[:3])
        rows.append((median(r["scores"]["performance"] for r in recs), name, len(recs),
                     t["sizes"][name], lab("lcpMs"), lab("tbtMs"),
                     max(r["labMetrics"]["cls"] or 0 for r in recs),
                     f"{sum(f['overallCategory'] != 'FAST' for f in field)}/{len(field)}",
                     ", ".join(f"{k} ({v}/{len(recs)})" for k, v in fixes.most_common(2) if v > 1)))
    print(f"{'template':32} {'perf':>4} {'n':>2} {'pages':>6} {'LCP ms':>7} {'TBT ms':>7} {'CLS':>5} field  shared fixes")
    for perf, name, n, size, lcp, tbt, cls, fld, fx in sorted(rows):
        print(f"{name:32} {perf:4.0f} {n:2} {size:6} {lcp:7.0f} {tbt:7.0f} {cls:5.2f} {fld:5}  {fx or '-'}")
    slow = [r for r in sorted(rows) if r[0] < 50]
    print(f"\n{len(slow)} template(s) with median mobile performance below 50, covering "
          f"{sum(r[3] for r in slow)} of {sum(t['sizes'].values())} sitemap pages.")
    print("Not audited:", ", ".join(failed) or "none")


if __name__ == "__main__":
    {"sample": sample, "report": report}[sys.argv[1]](sys.argv[2])
Enter fullscreen mode Exit fullscreen mode

Run python templates.py sample sitemap.json. On the test store it printed:

/collections/*                 1349 pages, auditing 3, e.g. /collections/all-shoes /collections/womens-spring
/products/*                     368 pages, auditing 3, e.g. /products/mens-runner /products/mens-trail-runner
/pages/*/*                      241 pages, auditing 3, e.g. /pages/menu-image-list/nav-tile-womens /pages/stores/city-outdoor-gear
/pages/*                        174 pages, auditing 3, e.g. /pages/materials /pages/newsletter
/blogs/*/*                       80 pages, auditing 3, e.g. /blogs/news/spring-launch /blogs/journal/everyday-shoes
/blogs/*                          2 pages, auditing 2, e.g. /blogs/news /blogs/journal
/                                 1 pages, auditing 1, e.g. /

18 URLs written to input.json from 2215 pages
Enter fullscreen mode Exit fullscreen mode

Read this before auditing. /pages/*/* mixes navigation tiles with store-locator pages, two layouts in one group; adding (r"^/pages/stores/", "/pages/stores/*") to RULES splits them. Rerun until each group is one layout.

Step 3: audit the sample

PageSpeed Insights Audit calls Google's official PageSpeed Insights API and returns one record per URL and strategy. input.json is already in its input format: mobile, performance category only, and includeAuditDetails on, so each record lists failing performance audits in opportunities.

curl -X POST "https://api.apify.com/v2/acts/josh99smith~pagespeed-insights-audit/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" -d @input.json -o results.json
Enter fullscreen mode Exit fullscreen mode

Or paste the URLs into Page URLs in the Console and download the dataset as JSON. For long lists use the apify-client packages instead of the synchronous endpoint.

Step 4: rank the templates

Run python templates.py report results.json in the same folder. Per template: median performance, pages audited (n) and in the sitemap, median lab LCP and TBT, worst CLS, audited pages failing page-level field data, and fixes in the top three opportunities of more than one sampled page.

template                         perf  n  pages  LCP ms  TBT ms   CLS field  shared fixes
/blogs/*                           37  2      2   10989    1392  0.00 0/0    Reduce JavaScript execution time (2/2), Minimize main-thread work (2/2)
/collections/*                     39  3   1349    9045    1499  0.00 0/1    Reduce JavaScript execution time (3/3), Minimize main-thread work (3/3)
/                                  40  1      1   20710     695  0.00 1/1    -
/blogs/*/*                         43  3     80   18073     596  0.00 0/0    Reduce unused JavaScript (3/3), Reduce JavaScript execution time (3/3)
/pages/*                           48  3    174   12101     498  0.00 0/0    Reduce unused JavaScript (3/3), Reduce JavaScript execution time (2/3)
/products/*                        50  3    368   19974     490  0.00 0/2    Reduce unused JavaScript (2/3), Improve image delivery (2/3)
/pages/*/*                         59  3    241    4031     447  0.00 0/0    Reduce unused JavaScript (3/3), Reduce JavaScript execution time (3/3)

5 template(s) with median mobile performance below 50, covering 1606 of 2215 sitemap pages.
Not audited: none
Enter fullscreen mode Exit fullscreen mode

How to read the result

Sort order is by score, but the order of work comes from the pages column:

  • /collections/* is the headline: a median of 39 across 1,349 pages. All three sampled pages share the same two fixes, JavaScript execution time and main-thread work, with TBT near 1.5 seconds. One change to that template reaches more pages than every other row combined.
  • /blogs/* scores lowest but has two pages. A look, not the first job.
  • /products/* has the worst LCP after the home page and "Improve image delivery" on two of three pages: start with the product gallery.
  • A fix shared by every sampled page is a template fix. If only one of three has it, that page probably has unusual content.
  • Small n is a lead, not a finding. Raise PER_TEMPLATE where the scores spread widely.
  • field counts pages failing page-level real-user data, out of those that have it. Most pages here fell back to origin-level data, which describes the whole site and is left out.

Lab LCP looks extreme because Lighthouse simulates a slow phone; compare templates with each other, not with the 2.5-second field threshold. The store is anonymised: the numbers are from the real run, the example paths are changed.

Cost and limits

Sitemap URL Extractor charges $0.0002 per extracted URL, and PageSpeed Insights Audit $0.004 per successfully audited URL and strategy. For the test store:

  • Sitemap: 2,216 URLs x $0.0002 = $0.443 (one non-page URL, skipped by the script, is still extracted).
  • Audits: 18 x $0.004 = $0.072, mobile only.
  • Total: about 52 cents.

Auditing every page would be 2,215 x $0.004 = $8.86 per strategy, and several hours at the README's estimate of 10 minutes per 100 URLs, for the same seven rows. Adding desktop doubles the audits to $0.144. After a template change, re-audit the same sample for $0.072; templates.json is reused.

Sites with no loadable sitemap and pages Google cannot audit are recorded as failures and cost nothing, and a run stops at the maximum cost you set. Limits:

  • Only URLs in the sitemaps are returned; nothing is crawled. Add unlisted templates (search results, filtered views) by hand.
  • maxUrlsPerSite goes up to 200,000; the default maxSitemapsPerSite of 500 files can cut off large publishers.
  • Pages behind a login fail with http-error.
  • The audit Actor's default Google API key is shared; when its quota runs out you get free rate-limited records. Your own free key in Google API key gives you 25,000 audits a day.

If this saves you time, a review on the Store page helps other people find it, and a star on GitHub helps too.

Disclosure: I built both Actors, Sitemap URL Extractor and PageSpeed Insights Audit. Source: github.com/josh99smith/sitemap-url-extractor and github.com/josh99smith/pagespeed-insights-audit. PageSpeed Insights and Lighthouse are Google trademarks, and Shopify is a trademark of Shopify Inc.; the Actors are not affiliated with Google or Shopify.

Top comments (1)

Collapse
 
gautamtalksdev profile image
Gautam K •

Three per template instead of all 2,000 is the right trade, and admitting path patterns are only a first guess is the honest part. One cheap check on a guessed group is to compare the script list of the sampled pages; if two URLs in the same group load different bundles, the group is probably two templates. How did you handle paths that looked alike but rendered different layouts, like the stores example?