Every agency has the same Monday ritual: open the desktop crawler, wait for it to chew through a client site, export a giant spreadsheet, and copy numbers into a report nobody reads past page two. The data in that workflow is fine. The shape of it is the problem.
An audit is more useful as one JSON row per page with a computed issues array. Here is why, and how to build the checks.
The row, not the report
{
"url": "https://example.com/pricing",
"title": "Pricing",
"titleLength": 7,
"metaDescription": null,
"h1Count": 2,
"wordCount": 96,
"redirectHops": 0,
"brokenLinkCount": 1,
"issues": ["title_too_short", "missing_meta_description",
"multiple_h1", "thin_content", "broken_internal_links"]
}
Once a page is a row, everything downstream is trivial: filter issueCount > 0, group by issue type, diff this week against last week, pipe into a dashboard, alert on regressions. The "report" becomes a query.
The checks worth automating
Most of a technical audit is a handful of deterministic checks per page:
-
Title: missing, under ~10 chars, over ~60 chars, duplicated across pages. The duplicate check must run across the whole crawl, not per page — keep a
Mapof title → first URL and flag later occurrences. - Meta description: missing, over ~160 chars, duplicated (same cross-crawl map trick).
- Headings: zero H1s or more than one.
-
Indexability:
noindexin robots meta, missing canonical. -
Redirect chains: follow redirects manually (
redirect: 'manual') so you can count hops and record each status. A 301 → 302 → 200 chain is invisible if your HTTP client silently follows it. - Broken internal links: collect every internal href during the crawl, HEAD-check each unique URL exactly once, cache the status run-wide. Fall back to a 1-byte ranged GET when a server rejects HEAD with 405.
- Content: word count after stripping nav/header/footer/scripts — thin-content thresholds are debatable, but under ~150 words is rarely a page that deserves to rank.
- Images: count how many are missing alt text; don't fail the page, report the number.
None of this needs a browser. Marketing sites, blogs, docs, and stores are server-rendered; a plain HTTP fetch plus an HTML parser audits them accurately, and the whole crawl runs in seconds instead of minutes.
The scheduling trick that makes it an agency product
A one-off audit is a snapshot. The valuable thing is the diff: schedule the crawl weekly per client, store each run's rows, and compare issues arrays. "3 new broken links, 2 pages went noindex after Thursday's deploy" is a report clients act on — and it is 10 lines of comparison code once your audit is rows instead of a spreadsheet.
I packaged this as an Apify actor this week: SEO Site Audit Scraper does everything above (BFS crawl with include/exclude filters, sitemap seeding, manual redirect chains, cross-crawl duplicate detection, free HEAD-checked broken links) and returns one row per page, pay per page, no seat license. The first 2 pages of every run are free.
The desktop crawlers are good tools. But if your audit ends up in a dashboard, a client report, or a CI check, it should have been JSON from the start.
Top comments (0)