DEV Community

Kessy Stalford for SEOSage

Posted on

Stop Fixing SEO Issues in the Order Your Crawler Lists Them

Disclosure: I work with SEOSage, one of the tools mentioned near the end of this post. Everything before that section is tool-agnostic, and the workflow below works with a free crawler, a spreadsheet, and Google Search Console. I've tried to write this so it's still useful if you delete every mention of SEOSage — because it should be.

Every SEO crawler hands you the same thing: a long, intimidating list of issues, sorted by how the tool happens to organise them. Missing titles. Duplicate H1s. Redirect chains. Images without alt text. Pages with thin content.

And almost everyone — myself included, for longer than I'd like to admit — starts at the top of that list and works down.

That's the problem. Your crawler's list order is an accident of how the software groups issues. It has nothing to do with money, traffic, or what a visitor experiences today. A broken page nobody has visited in six months sits in the same list, with the same visual weight, as the landing page that earns you leads every single day and currently returns a 404 to half the people tapping your ad.

This post is about reordering that list by the one thing that actually matters: whether anyone is trying to use the page.

The real cost of list-order fixing

When you fix issues in crawler order, three things tend to happen:

  1. You burn hours on invisible pages. Old tag archives, forgotten campaign landers, staging leftovers that somehow got indexed. Technically broken, practically irrelevant.
  2. The urgent stuff waits. The money page with the redirect loop, the pricing page dropping out of the index, the checkout-adjacent page with the rendering problem. They were on page three of your export.
  3. Clients and stakeholders can't see the point. "We fixed 1,400 issues" is a vanity metric. "We fixed the 11 pages people actually land on" is a result.

Traffic-first prioritisation flips the sequence: crawl everything, find out which broken pages people are actually reaching (or trying to reach), fix those first, then sweep the rest in descending order of real-world impact.

Here's the full workflow.

Step 1: Crawl the whole site, not a sample

Start with a complete crawl. If your site is small, this is trivial. If it's large, crawl in sections — templates, folders, subdomains — but make sure the union of your sections covers everything, including the dusty corners. You can't prioritise what you never crawled.

Capture, at minimum, for every URL:

  • Status code (200, 301, 404, 500, etc.)
  • Indexability (indexable, noindex, blocked by robots, canonicalised elsewhere)
  • Title and H1 presence/duplication
  • Redirect chains and loops
  • Inlinks (how many internal links point to the page)

One honest limitation: a crawler only sees what it can reach. Orphan pages with zero inlinks won't show up unless you feed the crawler extra URL lists. Pull URLs from your XML sitemap, your CMS export, or last year's crawl and add them as a supplementary list. The pages nobody links to and nobody visits are exactly the ones you may later decide to delete — but decide that on purpose, not by accident.

Step 2: Get your traffic data into the same table

Now export two things:

  • Google Search Console → Pages report (clicks and impressions per landing page, last 90 days is a good window)
  • Google Analytics (GA4) → Landing page report (sessions/engagement per page), if you have it

Join them to your crawl data on the URL. In a spreadsheet that's a VLOOKUP/XLOOKUP on a normalised URL (strip trailing slashes and tracking parameters first, or your match rate will be miserable).

You now have, for every page: what's wrong with it and whether anyone cares.

If you have no analytics access — a prospect's site, a brand-new domain — Search Console alone is a perfectly good traffic proxy. No traffic data at all? Fall back to inlinks as a rough importance signal: heavily internally-linked pages tend to be the ones the business cares about. But treat that as a distant second choice.

Step 3: Sort by "broken × visited"

Create a simple priority view:

Priority Condition Why
P0 — fix today Error (4xx/5xx) or not indexable, and has sessions/clicks in the last 90 days Real people are hitting a dead end right now
P1 — this week Redirect chain, wrong canonical, or duplicate title on a page with traffic Visited pages leaking equity or confusing Google
P2 — batch it Issues on pages with impressions but few/no clicks Google shows them; users don't pick them. Fix after P1
P3 — backlog/delete Issues on pages with zero traffic and zero impressions Fix cheaply, noindex, or remove. Don't gold-plate these

This table is the entire method. Everything else is execution detail.

The first time you do this, the P0 list is usually shockingly short — five to fifteen URLs on most sites — and shockingly important. I have seen a single redirect-looping service page, quietly eating paid and organic landings for weeks, surface instantly the moment traffic was overlaid on the crawl. In list-order world, it was issue #1,204.

Step 4: Fix in priority order, then re-crawl

Work P0 down. For each fix, note what changed and when. Then re-crawl and diff: confirm the status code changed, the canonical points where you intended, the chain collapsed to a single hop.

Re-crawling is the step people skip, and it's the only one that proves anything. "We changed it in the CMS" is not verification. Your crawler showing a clean 200 on the previously-broken money page is.

Then move to P1. P3 pages get a deliberate decision — fix, noindex, redirect to something relevant, or delete — rather than another pass of cosmetic repairs.

Doing this with whatever tool you already have

You do not need new software for this workflow. Honestly, a spreadsheet and forty minutes will get most sites 80% of the value.

Approach How the traffic overlay happens Watch out for
Screaming Frog (free or paid) Crawl, then connect the GA4/GSC APIs (Configuration → API Access) or export CSVs and join in Sheets/Excel Free tier caps at 500 URLs; URL normalisation before joining
Sitebulb Crawl with GA/GSC data sources enabled in project settings, then filter reports by traffic Schedule crawls so the overlay doesn't go stale
SEOSage (disclosure: my employer) GA connection pulls sessions onto each crawled page; sort/filter the page list by traffic so broken-but-visited rows float up Desktop app: crawl runs on your machine, so big crawls need the laptop plugged in and awake
Pure spreadsheet Crawl export + GSC Pages export + XLOOKUP Slowest, but zero cost and you learn your data intimately

Pick the row you already own. The method is identical; only the join mechanics differ.

What this changes in practice

Your reports get shorter and better. Instead of "2,300 issues found," you report "9 visited pages were broken; all 9 fixed and verified; next batch is 40 redirect chains on pages with impressions." Stakeholders understand that sentence.

Your crawl budget conversations get easier. Pages with zero traffic, zero impressions, and zero inlinks are candidates for removal, which shrinks the site Google has to crawl and stops those URLs diluting your reports forever.

You stop being afraid of the list. A 10,000-issue crawl export is paralysing. A P0 list of 11 URLs is a Tuesday morning.

A quick start for this week

  1. Run one full crawl of your main site. Save the export; don't touch anything yet.
  2. Export 90 days of GSC landing page clicks/impressions.
  3. Join them. Normalise URLs first.
  4. Filter: status ≠ 200 (or non-indexable) and clicks > 0. That's your P0.
  5. Fix those, re-crawl, confirm.

If the P0 list comes back empty: genuinely good news, and now you know it instead of guessing.


Disclosure restated: I work with SEOSage. This post describes a workflow, not a pitch — it runs fine in Screaming Frog, Sitebulb, or a spreadsheet. SEOSage's angle on it is that GA traffic sits next to each crawled page so the sort in Step 3 is built in rather than a manual join, and crawls run locally on your Mac or Windows machine without a page cap. If the manual join doesn't bother you, you lose nothing by ignoring that entirely.

Found a broken-but-visited page this way that surprised you? I'd genuinely like to hear what it was in the comments.

Top comments (0)