HEAD vs GET for link checking: HEAD is faster, and it lies more than you'd think

#http #webdev #python #testing

If you've ever written a link checker, you've had this thought: I don't need the page body, I just need the status code. I'll send HEAD.

It's a reasonable instinct. HEAD is defined to return the same headers as GET without the body, so it should be cheaper and give the same answer. In practice, a noticeable share of servers answer HEAD differently from GET, and a link checker that trusts HEAD ends up with false alarms, missed breakages, or both.

Here's a direct comparison and the strategy I'd actually ship.

The theory

RFC 9110 says a server should send the same header fields for HEAD as it would for GET. Note the "should". Plenty of real-world stacks treat HEAD as an afterthought:

Frameworks that route by method and never registered a HEAD handler.
CDNs and WAFs with separate rules for "unusual" methods.
App servers that run the full GET handler and throw the body away. Correct, but no cheaper.
Hand-written handlers that return 405 Method Not Allowed for anything that isn't GET or POST.

Side by side

	HEAD	GET
Bandwidth	Headers only	Full body (unless you stop reading early)
Speed on big pages/files	Fast	Slow unless streamed
Status accuracy	Varies by server	What browsers actually get
Common false positives	405, 403, 404 on valid URLs	Rate limits on heavy crawls
Common false negatives	200 for pages that 404 on GET (rare but real)	Soft 404s (both methods)
Detects content problems	No	Yes, if you inspect the body
Bot-blocking	Often treated as more suspicious	Less so with a sane User-Agent

The key row is status accuracy. A link checker exists to report what a human clicking the link would see, and humans send GET.

Seeing the difference yourself

Here's a small script that requests each URL with both methods and flags disagreements. Run it against your own outbound links and you'll probably find a few.

# head_vs_get.py  usage: python head_vs_get.py urls.txt
import sys
import requests

UA = {"User-Agent": "Mozilla/5.0 (link-audit; +https://example.com/bot)"}

def status(method, url):
    try:
        r = requests.request(method, url, headers=UA, timeout=10,
                             allow_redirects=True, stream=True)
        r.close()  # don't download the body for GET
        return r.status_code
    except requests.RequestException as e:
        return type(e).__name__

mismatches = 0
for url in (l.strip() for l in open(sys.argv[1]) if l.strip()):
    h, g = status("HEAD", url), status("GET", url)
    flag = "" if h == g else "  <-- MISMATCH"
    mismatches += bool(flag)
    print(f"HEAD {h!s:>5}  GET {g!s:>5}  {url}{flag}")
print(f"\n{mismatches} mismatches")

stream=True plus close() is the trick: GET with streaming fetches the headers and stops, so you get GET-level accuracy at close to HEAD-level cost. The body bytes already in flight still arrive, but you don't wait for a 40 MB PDF to finish.

The mismatches you'll typically see:

HEAD 405 GET 200: the server doesn't implement HEAD.
HEAD 403 GET 200: a WAF that treats HEAD as bot traffic.
HEAD 404 GET 200: a route only registered for GET (some single-page-app hosts do this).
HEAD 200 GET 404: rare, usually a CDN answering HEAD from cache while the origin has removed the page.

The hybrid strategy

Here's what I'd use in a real checker:

Send HEAD first for speed.
If HEAD returns anything other than 2xx or 3xx, retry with streamed GET. Report the GET result.
For HTML pages you care about, use GET anyway and inspect the body for soft-404 patterns ("page not found", empty templates).
For large binary files (PDFs, zips, videos), HEAD is usually reliable, because static file servers implement it properly, and it saves real bandwidth.

def check(url):
    try:
        r = requests.head(url, headers=UA, timeout=10, allow_redirects=True)
        if r.status_code < 400:
            return r.status_code, "HEAD"
    except requests.RequestException:
        pass
    r = requests.get(url, headers=UA, timeout=15, allow_redirects=True, stream=True)
    r.close()
    return r.status_code, "GET-fallback"

The fallback costs one extra request, and only on links that look broken, which should be a small share. In exchange, you drop most of the false alarms that make people stop trusting link reports.

When HEAD-only is fine

Checking your own static assets on infrastructure you control and know implements HEAD.
Large download links where body cost matters.
Quick smoke tests where a few false positives are acceptable.

When you should go GET-first

External links to sites you don't control. Their servers are the wild west.
Anything behind Cloudflare, Akamai or similar bot protection.
Internal pages where you also want to catch soft 404s, empty pages, or missing content.

One more gotcha: redirects

Some servers redirect HEAD and GET differently, for example sending HEAD to a login page while GET goes to the content. If your checker reports the final URL, log it for both methods during debugging. It's a two-line change and it saves a lot of guessing later.

If you'd rather not maintain the retry logic, user agents and edge cases yourself, a hosted checker like linkbruno.com handles the crawl. The method choice still matters to understand, because it explains most of the "this link works for me!" disputes you'll have with a report.

DEV Community