TL;DR: Reading a crawl as one flat list averages out the problems that live inside a single template, and it makes every analysis pass run against pages it should ignore. Segment by template and role first. On one audit that cut AI analysis calls from roughly 312 to 111, a 64 percent reduction, with zero out-of-scope pages.
Run this first. One match rule per template, then a count check that proves the rule caught every page it should:
# One segment rule per template, matched on the URL path
Guides Address matches regex ^https://example\.com/guides/
Services Address matches regex ^https://example\.com/services/
Articles Address matches regex ^https://example\.com/blog/
# Validate the binding before trusting any per-segment number
Expected == Crawl segment count
Guides 5 == 5
Services 5 == 5
Articles 16 == 16
# A short count means a slug slipped the pattern. Fix the rule first.
That config is the whole move. One match rule per template, then a validation step so you know the rules caught every page they should. Everything below is why it changes your analysis and how far it scales.
The problem it solves is the default way most people read a crawl. A crawl of any real site returns one long list of issues, and reading it that way averages out the bugs that live inside a single template. A title-length warning on a deep archive page and the same warning on a core service page collapse into one undifferentiated line. The count went up. Which template caused it stays hidden.
Why a flat crawl hides your worst problems
When an issue lives inside one template, the duplicates a single template creates, a duplicate H2 from a shared component, schema emitted twice by one block, a flat view buries it among hundreds of unrelated rows. You triage by guessing and fix the loudest symptom instead of the template producing it.
There is a second cost that only shows up at scale. Every validation pass and every model prompt runs against every URL, including pages the check was never meant to evaluate. On a small crawl that waste is invisible. On a large one it decides whether the analysis is affordable.
Segmenting in Screaming Frog, three ways
Segmentation groups URLs by template or role so every report, issue count, and score reads per template instead of averaged sitewide. It is the same logic as testing by template cohort: stop treating the site as one list, start treating it as a set of templates that each have their own contract.
In Screaming Frog this is the Segments feature, added in version 19.0. Define segments once and the tool adds a coloured segments column to every tab and a segments bar to the Issues tab. You can set a segment up at the start, during, or at the end of a crawl, so you can segment a crawl you already have without re-running it.
Three ways to define one:
- URL pattern. The common case, templates encoded in the path. One regex match per template against the Address field, as in the block above. This only holds if your URL structure is predictable. If one template is served under several path shapes, a single rule catches one and misses the rest.
-
Custom extraction. When the template is not in the URL, pull an on-page signal that identifies it, a body class or the schema
@type, and segment on that value. - API data. Screaming Frog joins GA4 and Search Console data to the crawl, so you can segment by a metric like sessions or impressions, not just by structure.
Database storage mode for large crawls
Segmentation pays off most on large crawls, and large crawls need database storage mode, which writes the crawl to disk instead of holding it all in memory. That is what lets you save a crawl, reopen it, and compare it against a later one. Screaming Frog's hardware guidance: 8GB of RAM is generally enough under 200,000 URLs, and crawling over one million URLs wants an SSD and 16GB or more, on a 64-bit operating system with at least 4GB as the floor.
A real four-segment model
From a sanitized end-to-end review of a production WordPress site. Four segments by role, each judged on different criteria, because the question you ask an article is not the one you ask a service page.
| Segment | URLs | Judged on |
|---|---|---|
| Articles | 16 | Technical depth, information gain, claim accuracy |
| Guides | 5 | Hub purpose, audience, routing to the right pages |
| Service pages | 5 | Scope clarity, proof, CTA alignment |
| Trust core | 5 | Evidenced authority, credentials that verify |
The counts are not the point. A title warning on an article and a missing proof block on a service page are different problems with different fixes, and a flat crawl flattens them into one list. Per-segment reading also fed the schema work, since schema that breaks at the template level is exactly the kind of issue that hides in a flat view and snaps into focus when you read it per template.
The failure mode: rules skip slugs silently
URL-pattern rules have a quiet failure mode. A slug that does not follow the dominant naming convention is skipped. No error, no warning. The page falls out of the segment, and every per-segment number after that is wrong by however many slipped through. Validate the binding with the count check in the lead block before you trust anything. On the audit above, the binding was validated against the segment counts before anything ran, so every intended page was scored and none were missed.
Segmentation is cost control
Screaming Frog can run language-model prompts and other API-driven checks against page content at crawl time. They bill per call on your own key. Unbound, every prompt runs against every URL. Bound to segments, each runs only against the pages it was written for: the article prompts against articles, the trust prompts against trust pages.
On the audit above that cut the run from roughly 312 calls to 111, a 64 percent reduction, with zero out-of-scope URLs touched. Same coverage, a third of the calls. Reading per template also makes under-linked templates obvious, so you can route link equity toward priority pages instead of guessing.
Track each template over time
Compare mode loads two saved crawls run with identical settings and reports the change per metric, issue, and segment. Watch one template's counts move between crawls and catch a regression in that template before it shows up as a sitewide number, or before it shows up in rankings. It needs database storage mode and a licensed version, and the second crawl has to use identical settings, or the diff measures your configuration change instead of the site. Per-segment comparison is how a crawl becomes an evidence trail of what changed rather than a snapshot.
Segment first. Validate the binding. Then every report reads per template, the analysis costs less, and fixes land on the template that produced the problem rather than the page where you happened to notice it.
Top comments (0)