linkbruno.com

Posted on Nov 7 Edited on Nov 7

429, 403, 999: a link checker's FAQ for sites that don't want to talk to you

#http #webdev #python #testing

Run a link checker on any content-heavy site and you'll get a familiar class of false alarms: links that work perfectly in your browser but come back as 429, 403, or some odd number like 999.

These aren't broken links. They're sites telling your checker to go away, or to slow down. Treating them as broken wastes your time; ignoring them entirely hides real breakage. Here's an FAQ on handling them properly.

What does 429 mean?

429 Too Many Requests: you're being rate limited. You sent too many requests to that host in a short period. The site is fine; your checker is being impatient.

It's especially common when a page links to many URLs on the same host (dozens of GitHub files, many pages of the same docs site) and your checker requests them in parallel.

How should a checker handle 429?

Slow down and retry, and respect the Retry-After header if the server sends one. It can be a number of seconds or an HTTP date. Here's a small, standard-library Python version:

import time, urllib.request, urllib.error
from email.utils import parsedate_to_datetime

UA = "my-link-checker/1.0 (+https://example.com/link-checker)"

def retry_after_seconds(value, default=5.0):
    if not value:
        return default
    try:
        return float(value)                       # "120"
    except ValueError:
        try:                                      # "Wed, 21 Oct 2026 07:28:00 GMT"
            return max(0.0, parsedate_to_datetime(value).timestamp() - time.time())
        except (TypeError, ValueError):
            return default

def check(url, max_tries=4, max_wait=60):
    for attempt in range(1, max_tries + 1):
        req = urllib.request.Request(url, headers={"User-Agent": UA})
        try:
            with urllib.request.urlopen(req, timeout=15) as r:
                return r.status, attempt
        except urllib.error.HTTPError as e:
            if e.code in (429, 503) and attempt < max_tries:
                wait = min(retry_after_seconds(e.headers.get("Retry-After")), max_wait)
                print(f"  {e.code} on try {attempt}, waiting {wait:.0f}s")
                time.sleep(wait)
                continue
            return e.code, attempt
    return None, max_tries

if __name__ == "__main__":
    import sys
    status, tries = check(sys.argv[1])
    print(f"{sys.argv[1]} -> {status} after {tries} tries")

Against a local test server that returns 429 twice (with Retry-After: 1) and then 200:

  429 on try 1, waiting 1s
  429 on try 2, waiting 1s
http://127.0.0.1:8798/page -> 200 after 3 tries

Note the max_wait cap. Some servers ask you to wait an hour; a checker shouldn't silently hang that long. Cap it, and if the cap is hit, report the link as "rate limited, not checked" rather than "broken".

How do I avoid 429 in the first place?

Retrying is the fallback. Prevention is better:

Limit concurrency per host, not just globally. Ten parallel requests spread across ten sites is fine; ten at once to one site is rude.
Add a small delay between requests to the same host.
Deduplicate URLs before checking. The same link on 200 pages only needs one request per run.
Cache results between runs for a while. A link that was fine an hour ago probably still is.

A simple per-host limiter in Python can be a dictionary of "last request time" per hostname, sleeping until at least N milliseconds have passed.

What about 403 Forbidden?

A 403 on a link that works in your browser usually means the site is blocking requests that look automated. Common triggers:

no User-Agent header, or a default library one like python-urllib/3.x;
missing headers a browser would send, such as Accept;
bot protection or a web application firewall that challenges non-browser clients.

Sending an honest, descriptive User-Agent (as in the code above) sometimes helps. Pretending to be a browser to bypass protections is a grey area we'd avoid: the site has made a choice, and your checker should respect it. Instead, classify the result as "blocked" and handle it separately.

And 999?

999 isn't a standard HTTP status code. It's been widely reported from some large social networks when they detect automated access. The link may be completely fine; your checker just can't verify it.

Treat non-standard codes like 999 as "unverifiable" rather than broken.

Should I use HEAD or GET?

Start with HEAD to save bandwidth, but fall back to GET when HEAD returns errors like 405 or 403, since some servers handle HEAD badly or block it. (We've covered HEAD vs GET in depth before; the short version is: never mark a link broken on a HEAD failure alone.)

How should results be reported?

The most useful thing a checker can do is separate outcomes into clear buckets instead of one "broken" list:

Bucket	Typical codes	Action
Broken	404, 410, DNS failure, connection refused	Fix or remove the link
Server error	500–599 (after retries)	Recheck later; fix if persistent
Rate limited	429 (after retries)	Recheck later, slower
Blocked / unverifiable	403, 999, bot challenges	Check manually or allowlist
OK	200–399	Nothing to do

People trust a report where the "broken" list is actually broken. Mixing in rate-limited and blocked links trains everyone to ignore the report.

Can I just allowlist the problem domains?

Yes, and it's often the pragmatic answer for big, stable sites that always block checkers. Many checkers support an exclude or allowlist setting. The trade-off: if a specific URL on that domain really does break, you won't know. A middle ground is to check allowlisted domains less often, or manually, once a quarter.

Is it OK to run a checker against other people's sites at all?

Checking that the links on your pages work is normal web hygiene, and a single polite request per URL is negligible traffic. The key word is polite: identify your checker in the User-Agent, keep concurrency low per host, respect rate limits, and don't hammer anyone.

Where a service helps

All of this logic (per-host limits, retries, classification, caching) is the unglamorous core of a reliable link checker. If you'd rather not build it, a service such as linkbruno.com handles it and reports broken links separately from ones it couldn't verify. If you do build your own, the buckets above are the thing to copy.

DEV Community