The Location header is relative more often than your link checker thinks

Quick answer

httpbin.org/redirect/2 sends you through two hops. Both Location headers are relative:

http://httpbin.org/redirect/2      -> 302  Location: /relative-redirect/1
http://httpbin.org/relative-redirect/1 -> 302  Location: /get
http://httpbin.org/get             -> 200

If your redirect-chaser treats Location as an absolute URL, hop one is already broken. If it resolves relative URLs against the original URL instead of the current hop, you'll survive this chain and silently break on a cross-host one.

Location is allowed to be almost anything 🔁

RFC 7231 permits Location to be a full URI, an absolute path, or a relative path. In practice you will meet all three, often within a single chain, often on the same host.

The rule is one line, and the important part is which URL you resolve against:

from urllib.parse import urljoin

current_url = urljoin(current_url, location)   # current hop, not the original

Resolving against the original URL works fine right up until a chain goes example.com → www.example.com → /en/home, at which point you build example.com/en/home and get a 404 for a page that is perfectly healthy. Your report says the link is broken. It isn't. You are.

That's the worst possible outcome for a link checker, because a false "broken" costs someone an afternoon chasing a URL that was never wrong.

A dead hostname is a result, not an exception 🧱

The second design decision matters more than the redirect logic.

Point a checker at 500 URLs from a real spreadsheet and some of them will not be URLs in any useful sense. Typo'd hostnames. Domains that lapsed. An internal host that only resolves on the VPN. Here's what the HTTP layer does with one:

curl: (6) Could not resolve host: no-such-host-xyzzy-9182.example

That's an exception, not a status code. There is no response object. Nothing to record a status from.

The tempting implementation lets it propagate and fails the run. The customer gets a red X, no dataset, and no idea which of their 500 URLs caused it. They paid for the run either way.

A URL that fails to resolve is one of the answers you were asking for. It gets a row:

DNS_ERROR_CODES = frozenset({
    CurlECode.COULDNT_RESOLVE_HOST,
    CurlECode.COULDNT_RESOLVE_PROXY,
})
TIMEOUT_CODES = frozenset({CurlECode.OPERATION_TIMEDOUT})
SSL_ERROR_CODES = frozenset({
    CurlECode.SSL_CONNECT_ERROR, CurlECode.PEER_FAILED_VERIFICATION, ...
})

def classify(code):
    if code in DNS_ERROR_CODES: return "dns_error"
    if code in TIMEOUT_CODES:   return "timeout"
    if code in SSL_ERROR_CODES: return "ssl_error"
    return "connection_error"          # never raises

Curl code 6 becomes error_class: "dns_error" on a row shaped exactly like every other row. The classifier is a pure function over the real curl_cffi.const.CurlECode enum — we read the enum rather than guessing the names — and its fallback branch means an unrecognised code degrades to connection_error instead of blowing up the batch.

Two fences, not one:

async def check_url(session, cfg, url) -> ResultRow:
    """Chase redirects for one URL; never raise."""
    try:
        return await _chase(session, cfg, url, start)
    except Exception:
        logger.exception("Unexpected failure checking %s", url)
        return _build_row(url, url, [], None, "connection_error", _elapsed_ms(start))

The inner one classifies known failures. The outer one catches the failure we didn't think of, and there is always a failure we didn't think of. check_url returns a ResultRow or it returns a ResultRow. There is no third outcome.

This is the single most common defect we fix across our own fleet: a recoverable per-item error crashing the whole run. It's easy to write, it passes every test where the fixtures are healthy, and it detonates on the first real customer list.

Prove all four outcomes in the cloud, not on your laptop ☁️

A checker has four outcome classes, and "the tests pass" only means you exercised the ones you thought of. Before this went live we ran all four against real hosts on the platform:

1. a clean 200
2. a two-hop redirect resolving to httpbin.org/get, with both 302 hops recorded in order
3. a 404
4. an unresolvable host, landing as a normal row with an error_class — not crashing the run

Point four is the one worth paying attention to. If it had crashed, the run would have gone red and we'd have caught it. The failure mode we were actually guarding against is subtler: the run succeeds, reports three tidy rows, and the fourth URL is simply absent. A green run with a silently missing row looks identical to a healthy one on every dashboard, and the customer is billed the same.

We have shipped that bug before. It is why "the cloud run succeeded" is not the gate — "the cloud run produced the rows we expected, including the ugly ones" is the gate.

What the Actor gives you

The Bulk URL Status & Broken Link Checker takes any list of URLs — a sitemap export, a link-audit spreadsheet, an affiliate feed, a migration checklist — and returns one row each:

• final status code and the complete redirect chain, every 301/302/303/307/308 hop in order
• response time in milliseconds
• security headers (HSTS, CSP, X-Frame-Options) for SEO and infra audits
• error_class on failures: dns_error, timeout, ssl_error, too_many_redirects, connection_error

The page body is never downloaded. This is a status and health check, not a crawler — nothing is followed beyond the URLs you supply, and no content is stored. Redirect loops are capped at 20 hops and reported as too_many_redirects rather than spinning.

The honest limitations 🚧

• We report what the server returns to us. A site that geo-blocks or serves different status codes per region will look different from your office.
• No JavaScript execution — a client-side redirect via window.location is invisible to any HTTP-level checker, including this one.
• HEAD isn't universally honoured; some hosts answer 405 to HEAD and 200 to GET, so hops may cost a GET.

Pricing

$0.20 per run plus $0.002 per URL checked — about $2.20 per 1,000 URLs. Broken or not, a recorded failure is still a delivered result and is billed as one; a run that checks nothing costs the start fee and nothing else.

→ Bulk URL Status & Broken Link Checker on Apify

---

Built by Devil Scrapes. We handle the relative Location headers, the hosts that don't resolve, and the redirect loop that would otherwise spin until your run times out.