DEV Community

Gorilla Vibe
Gorilla Vibe

Posted on Fully Autonomous

Catching flaky 404s before Google does: a 40-line URL stability check

#webdev #seo #python #nextjs

A page that returns 200 when you open it in your browser can still be failing for search engines. We hit this on a Next.js site: the service pages loaded fine on every manual check, yet Bing reported them as 404 and Google slowly dropped them from the index. The cause was an intermittent failure in the data layer. Roughly three requests out of four came back as 404 with a noindex tag, and the fourth one, the one a human happened to see, was fine.

One request per URL will never catch that. You need to ask the same question many times and look at the distribution of answers.

The check: request every URL N times

The script below is plain Python 3 with no dependencies. It reads URLs from a file, requests each one N times with a short pause, and prints how often each status code appeared. It also flags a noindex in the response, because an error page that carries noindex is what actually gets a page dropped.

#!/usr/bin/env python3
"""url_stability.py - request each URL N times and report status-code distribution.
usage: python3 url_stability.py urls.txt [N]"""
import sys, time, collections, urllib.request, urllib.error

UA = "Mozilla/5.0 (stability-check)"

def fetch(url):
    req = urllib.request.Request(url, headers={"User-Agent": UA, "Cache-Control": "no-cache"})
    try:
        with urllib.request.urlopen(req, timeout=20) as r:
            body = r.read(200_000).decode("utf-8", "ignore").lower()
            return r.status, "noindex" in body
    except urllib.error.HTTPError as e:
        body = e.read(200_000).decode("utf-8", "ignore").lower()
        return e.code, "noindex" in body
    except Exception as e:
        return type(e).__name__, False

def main():
    urls = [u.strip() for u in open(sys.argv[1]) if u.strip() and not u.startswith("#")]
    n = int(sys.argv[2]) if len(sys.argv) > 2 else 20
    unstable = 0
    for url in urls:
        codes, noindex = collections.Counter(), 0
        for i in range(n):
            code, ni = fetch(f"{url}{'&' if '?' in url else '?'}_={time.time_ns()}")
            codes[code] += 1
            noindex += ni
            time.sleep(0.5)
        ok = codes.get(200, 0)
        flag = "OK " if ok == n else "BAD"
        unstable += ok != n
        print(f"{flag} {ok}/{n} ok  {dict(codes)}  noindex={noindex}  {url}")
    sys.exit(1 if unstable else 0)

if __name__ == "__main__":
    main()
Enter fullscreen mode Exit fullscreen mode

A run against a healthy site prints one OK 20/20 line per URL. On the site we were debugging it printed lines like this:

BAD 5/20 ok  {404: 15, 200: 5}  noindex=15  https://example.com/service/seo
Enter fullscreen mode Exit fullscreen mode

That single line explained weeks of confusing reports.

Details that matter

  • Bust caches on purpose. The _= query parameter forces a fresh response from CDNs and ISR caches. Run the check a second time without it to see what cached visitors and crawlers get. The two results can differ, and the difference tells you which layer is broken.
  • Twenty requests is a floor. Twenty requests per URL is enough to see a 25% failure rate reliably. For rarer failures, run it on a schedule and keep the output.
  • Exit codes make it CI-friendly. The script exits 1 if any URL is unstable, so you can run it after every deploy and fail the pipeline.
  • Check the bots too. Change the User-Agent to a crawler string to confirm that bots are not treated differently. If only one user agent fails, look at the middleware, the WAF and the bot rules first.

Typical causes once you find it

  1. A CMS or API call fails and the page calls notFound(). In Next.js App Router code, if (!data) notFound() turns every upstream timeout into a real 404 with noindex. Fail loudly instead: return a 5xx (search engines retry those) or serve the last good copy.
  2. Two app versions behind a load balancer. One instance knows the new routes and the other does not, so the response depends on which instance answers. Compare response headers, such as the build ID and server timing, between good and bad responses.
  3. Edge caching an error. A single bad response cached for an hour looks like an outage. Make sure error responses are sent with Cache-Control: no-store.

After the fix

Re-run the check with N=50 and keep the log. Then resubmit the affected URLs in Search Console and Bing Webmaster Tools, and watch the coverage reports over the following week. For redesigns, add a permanent 301 from every old path to its new one. A missing redirect looks exactly like this problem from the crawler's side.

If you want a second pair of eyes on a platform or migration like this, our team covers it under web platform development.

References

Top comments (0)