Google Was Spending 52% of Its Crawl on Dead URLs. Here's Our Audit and Fix.

Our new pages were indexing slowly, and everything obvious checked out: 64/64 pages returning 200, self-referencing canonicals, a valid sitemap, no broken internal links. So we stopped guessing and opened the one report most site owners never look at: Google Search Console → Settings → Crawl stats.

It took about five minutes to find the problem.

The numbers

Over the last 90 days, Google made 546 crawl requests to our site — roughly six a day. The average response time was 370 ms, which is fine. Speed was not the bottleneck.

The response breakdown was:

• Not found (404): 52%
• OK (200): 44%
• Not modified (304): 2%
• Moved permanently (301): 2%

And by purpose: 93% refresh (re-checking pages Google already knows) against 7% discovery (finding new ones). More than half of a very small crawl budget was being spent on URLs that no longer exist, while new pages waited for discovery that barely happened.

What the dead URLs actually were

The crawl-stats drilldown lists example URLs. Ours fell into three groups, all leftovers from a period when the site was compromised, before we rebuilt and locked it down:

1. Parasite endpoints — /api/stock/check.php, /api/cart/get-simple.php and friends, from a fake e-commerce layer that once lived on the domain. These alone accounted for dozens of requests.
2. WordPress leftovers — /feed/, /comments/feed/, an old /wp-content/uploads/ image path. Our site is hand-built static HTML; there is no WordPress anywhere in the stack. Google was still testing those paths months later.
3. Injected page names — random 12345678.shtml files that existed briefly during the compromise and never again.

Every one of them correctly returns 404. The failure wasn't the responses — it was that Google kept spending requests on them.

What we deliberately did NOT do

• No redirecting the junk to the homepage. Mass redirects of unrelated dead URLs are treated as soft 404s and can create new problems. A dead parasite URL deserves a clean death, not a redirect.
• No .htaccess surgery. One suspicious entry, /index.html, turned out to already 301 to the homepage under our existing rules — Google's examples were simply older than the fix. We verified before touching anything.

The fix

One file: robots.txt. We added disallow rules for exactly the junk patterns and nothing else:

Disallow: /api/
Disallow: /wp-content/
Disallow: /feed/
Disallow: /comments/
Disallow: /*.shtml$

Backed up first, uploaded, verified live, and confirmed the homepage, service pages and sitemap still fetch normally. Google will stop re-crawling the junk patterns, which frees the refresh share for real pages. We expect the discovery share to climb over the coming weeks, and we'll re-check the same report to prove it either way.

The lesson

When indexing stalls, measure crawl allocation before blaming speed or content. A fast site (370 ms) can still starve if half of Google's visits land on dead URLs. The Crawl stats report — specifically "By response" and its example URLs — is a five-minute check that tells you what Google is actually doing with your site, instead of what you assume it's doing.

It's also one of the main reasons why your website isn't showing on Google even when the site itself is technically clean: the queue in front of your real pages is full of ghosts.

— RapidBizz team, building and documenting our own SEO programme in the open.