linkbruno.com

Posted on Nov 2 Edited on Nov 3

LinkedIn returns 999, Cloudflare returns 403: why your link checker reports links that aren't broken

#automation #python #testing #webdev

The link report had 212 broken links. The team spent an afternoon going through them. 181 opened fine in a browser. After that, nobody looked at the report again, including the week it caught a real broken checkout link.

A link checker with too many false positives fails the same way a noisy alarm does: people stop listening.

TL;DR

Many "broken" results are bot protection, rate limits or methods the server doesn't support, not dead links.
Classify results into broken, blocked and flaky instead of a single "error" bucket.
Retry with GET instead of HEAD, use a realistic User-Agent, back off on 429, and re-check failures later.
Never auto-hide a status code completely. Downgrade it to "needs a human".
My opinion: a checker that says "I couldn't verify this" is more useful than one that guesses.

The usual suspects

Status	Typical cause	Really broken?
999	LinkedIn blocking non-browser clients	almost never
403	WAF, Cloudflare bot check, hotlink protection	often not
429	you're checking too fast	no
405 / 501	server doesn't support HEAD	no
400	missing headers some servers require	sometimes
timeout	slow server, geo-blocking, tarpitting	sometimes
404 / 410	page is gone	yes
DNS failure	domain expired or typo	yes

The last two rows are the ones you care about. Most of the noise is everything above them.

Fix 1: fall back from HEAD to GET

HEAD is cheap, so checkers use it first. Plenty of servers and frameworks return 405, 404 or even 500 for HEAD while GET works fine.

import requests

UA = "Mozilla/5.0 (compatible; link-audit/1.0; +https://example.com/bot)"

def check(url):
    s = requests.Session()
    s.headers.update({"User-Agent": UA, "Accept": "text/html,*/*;q=0.8"})
    try:
        r = s.head(url, allow_redirects=True, timeout=10)
        if r.status_code in (403, 404, 405, 501) or r.status_code >= 500:
            r = s.get(url, allow_redirects=True, timeout=15, stream=True)
            r.close()
        return r.status_code, None
    except requests.RequestException as e:
        return None, type(e).__name__

Retrying 404 with GET looks odd, but some servers really do return 404 for HEAD on dynamic routes.

Fix 2: classify, don't just count

BLOCKED = {401, 403, 999}
RATE = {429}

def classify(code, err):
    if err in ("ConnectionError",) and code is None:
        return "broken?"     # DNS or refused, re-check later
    if err:
        return "flaky"       # timeout, TLS hiccup
    if code in (404, 410):
        return "broken"
    if code in BLOCKED:
        return "blocked"
    if code in RATE or code >= 500:
        return "flaky"
    return "ok"

Now the report has three sections. "Broken" goes to whoever owns the content. "Blocked" gets a manual spot check, or an allowlist entry once a human confirms the domain works. "Flaky" gets re-checked tomorrow, and only escalates if it fails again.

Fix 3: respect 429 and slow down per domain

If you check 300 links to the same documentation site in ten seconds, you'll get rate-limited and report all of them. Throttle per host and honour Retry-After:

import time
from collections import defaultdict
from urllib.parse import urlsplit

last_hit = defaultdict(float)
MIN_GAP = 1.0  # seconds between requests to the same host

def polite(url):
    host = urlsplit(url).netloc
    wait = last_hit[host] + MIN_GAP - time.time()
    if wait > 0:
        time.sleep(wait)
    last_hit[host] = time.time()

Being polite also keeps your checker's IP off block lists, which reduces 403s on the next run.

Fix 4: re-check before reporting

Real broken links stay broken. Flaky ones don't. Re-checking every failure once, a few hours later, removes a large chunk of noise for almost no cost. Only report a link as broken if it failed on two runs at least a few hours apart.

Fix 5: be honest about social sites

LinkedIn, some social networks and some large retailers block nearly all automated requests. You won't get a reliable answer from a script. Options: allowlist the domain and check those links manually each quarter, or accept "unverified" as a valid state in the report.

Where the real failures hide

While you're tuning out noise, don't tune out the quiet failures: pages that return 200 but are really "not found" templates. Those were covered in the soft 404 post, and redirect problems in the redirect chains post. Browser-based checks have their own false-positive patterns. How to find broken links with Selenium and Python without false alarms goes through those, and How to find broken outbound links and replace them using the Wayback Machine covers what to do with the links that really are dead.

A hosted crawler like linkbruno.com is useful mostly because someone else has already done this classification tuning. Whatever you use, check that its report separates "dead" from "couldn't verify".

My take

The worst outcome of a link check isn't a missed broken link. It's a report so noisy that the next real broken link gets ignored. Tune for trust first: fewer, more certain results beat a long list nobody reads.

Over to you

Which domains are on your link-checker allowlist, and how did they get there? I suspect everyone's list starts with LinkedIn, but I'd like to know what else is on yours.

DEV Community