Soft 404s: your 'page not found' returns 200, and your link checker thinks it's fine

The link checker reported zero broken links. Every URL returned 200 OK. Then someone actually clicked through the old pricing links and found a friendly "Oops, we couldn't find that page" on every single one.

The site wasn't free of broken links. It was lying about them.

TL;DR

• A soft 404 is a "not found" page served with a 200 status (or a redirect to the homepage) instead of a real 404/410.
• Status-code-only link checkers can't see them. Search engines flag them as errors anyway.
• Detect them with a random-URL probe, content fingerprinting and redirect-to-home checks.
• Single-page apps are the worst offenders. The server returns index.html for everything.
• My opinion: redirecting every dead URL to the homepage is not a fix. It's a soft 404 with extra steps.

The three flavours

1. Error page with 200: the server renders a not-found template but forgets to set the status.
2. Redirect to homepage: every missing URL 301s to /. Users land somewhere unrelated, search engines treat it as a soft 404.
3. Empty shell: an SPA serves the app shell with 200, then client-side routing shows "not found".

All three pass a naive check like status < 400.

Test 1: request a URL that can't exist

The fastest diagnostic for a whole site:

curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' \
  "https://example.com/this-page-does-not-exist-$(date +%s)"

Anything other than 404 or 410 means the site is soft-404ing on at least some paths. Try it under a few prefixes, such as /blog/, /products/ and /docs/, because different sections are often served by different code.

Test 2: fingerprint the not-found page

Grab the body of a known-missing page, then compare every "successful" URL against it:

import hashlib, re, requests, sys

def norm(html):
    # strip the requested path and anything volatile before hashing
    html = re.sub(r"<script.*?</script>", "", html, flags=re.S)
    html = re.sub(r"\s+", " ", html)
    return html

def fp(url):
    r = requests.get(url, timeout=15, allow_redirects=True)
    return r.status_code, r.url, hashlib.sha1(norm(r.text).encode()).hexdigest(), r.text

base = sys.argv[1].rstrip("/")
_, _, missing_fp, _ = fp(base + "/zz-definitely-missing-zz")

for line in sys.stdin:
    url = line.strip()
    if not url:
        continue
    code, final, h, body = fp(url)
    reasons = []
    if code == 200 and h == missing_fp:
        reasons.append("same body as not-found page")
    if code == 200 and re.search(r"(page not found|404|doesn.t exist|couldn.t find)", body[:5000], re.I):
        reasons.append("not-found wording")
    if final.rstrip("/") == base and url.rstrip("/") != base:
        reasons.append("redirected to homepage")
    if reasons:
        print(f"SOFT 404? {url} -> {final} [{', '.join(reasons)}]")

cat urls.txt | python soft404.py https://example.com

The keyword check gives some false positives (a blog post about 404 errors will match), so treat output as a review list, not an automatic verdict.

Fix: set the real status code

Most fixes are one line once you find the right place.

Express

app.use((req, res) => res.status(404).render('not-found'));

nginx with a custom error page

error_page 404 /404.html;
location = /404.html { internal; }

Watch out for error_page 404 =200 /404.html;. That =200 is a soft 404 generator.

Next.js: return notFound: true from data fetching (or call notFound() in the App Router) instead of rendering an empty state.

Fix: SPAs without server-side routing

If your host serves index.html for every path, the server can't know which routes exist. Options, in order of preference:

1. Server-side render or pre-render known routes, and return 404 for the rest.
2. Give the host an explicit list of valid routes and a real 404 fallback.
3. At minimum, add <meta name="robots" content="noindex"> on the client-side not-found view, so crawlers don't index thousands of empty shells.

Fix: stop redirecting dead pages to the homepage

When a page is gone for good, the honest options are:

• 301 to a genuinely equivalent page if one exists (the new version, the parent category).
• 410 Gone if the content is intentionally removed.
• 404 with a helpful page: search box, links to popular sections.

The homepage is almost never equivalent. If you're cleaning up a backlog of these, Too many broken links? How to decide which to fix first helps you prioritise, and How to fix "Not found (404)" errors in Google Search Console covers the report you'll see once the real status codes go out.

Make your checker smarter

Status-code checks miss soft 404s, so a decent link-checking setup adds at least one of: a not-found fingerprint, a keyword scan, or redirect-to-home detection. Watch for loops too, since some catch-all rules bounce missing pages around before landing on /. The previous post on redirect loops has the tracing commands. linkbruno.com and similar crawlers are most useful when the site returns honest status codes in the first place, which is the real fix here.

My take

The status code is the contract between your server and every machine that reads it: browsers, crawlers, monitoring, link checkers. A friendly not-found page is great for humans. Sending it with 200 breaks the contract for everything else.

Over to you

Do you return a real 404 from your SPA, or live with the app shell returning 200? I'm curious which frameworks people have found make this easy and which fight you on it.