A 200 Status Code Is Not Proof of Life: A Link-Triage Case Study

Two days before term started, I pasted 62 links into the course reading list and told myself I'd check them "later." Later arrived during office hours, when a first-year student emailed me that the Week 3 reading went to a parking page. Then a second student said the same link worked fine for her. Same URL. Two different answers. That is the moment I stopped trusting status codes.

So here is the single question this case study answers: can I tell a dead link from a walled link without opening a browser, and where exactly does the HTTP status stop being useful? I'll build the smallest tool that answers it, run it against fixtures I control, show you the fixture that broke my first rule, and then explain why the leftovers are the only part worth spending model tokens on.

Prerequisites, honestly stated

CPython 3.12, standard library only, nothing to pip install. The fixture server binds to 127.0.0.1:8765, so no network access is required for the deterministic part. The optional classification step needs an HTTP endpoint — I used MonkeyCode's free model access for that, and I run the weekly job on their free server option.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Two claims come from the operator and not from my own measurement: free model access, and a free server option. The 10-million-token allowance is what the free tier advertises today. Check the current terms yourself before you build a workflow on it — quotas move, and mine isn't a guarantee for yours.

Step 1: Build ground truth, not a scraper

My first attempt pointed the script at the real reading list, which was a mistake. Real URLs change while you're debugging them. You can't tell whether your classifier is wrong or the web is. So I wrote eight fixtures where I know the answer.

# fixtures.py — deterministic ground truth on 127.0.0.1:8765
import http.server, socketserver, time

PAGES = {
    "/ok":        (200, "text/html", "<h1>Week 3: Gradient Descent</h1><p>Read sections 1-3.</p>"),
    "/gone":      (410, "text/html", "<h1>Gone</h1>"),
    "/moved":     (301, "text/html", ""),                      # no Location header
    "/soft404":   (200, "text/html", "<h1>Sorry, this page could not be found.</h1>"),
    "/cookiewall":(200, "text/html", "<h1>We value your privacy</h1><p>Accept cookies.</p>"),
    "/shell":     (200, "text/html", "<div id=\"root\"></div>"),   # JS app, no server HTML
    "/paper":     (200, "application/pdf", "%PDF-1.4 fake bytes"),
    "/slow":      (200, "text/html", "too late"),
}

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        if self.path == "/slow":
            time.sleep(12)
        status, ctype, body = PAGES.get(self.path, (404, "text/html", "<h1>Nope</h1>"))
        data = body.encode()
        self.send_response(status)
        self.send_header("Content-Type", ctype)
        self.send_header("Content-Length", str(len(data)))
        self.end_headers()
        self.wfile.write(data)
    def log_message(self, *args):
        pass

if __name__ == "__main__":
    socketserver.TCPServer(("127.0.0.1", 8765), Handler).serve_forever()

Run it in one terminal with python fixtures.py. That file is the whole trick. Everything after this is just a program that has to agree with it.

Step 2: Probe, and record more than the status

urllib gives you the status, the final URL after redirects, the content type, and the bytes. Grab all four. A single integer throws away the evidence you'll need later.

# linktriage.py
import json, re, sys, time, urllib.request, urllib.error

UA = "linktriage/0.1 (course reading list audit)"

NOT_FOUND = ("page not found", "could not be found", "no longer available", "has been removed")
CONSENT = ("accept cookies", "cookie settings", "value your privacy")

def extract_text(raw, ctype):
    if not ctype.startswith("text/html"):
        return ""
    html = raw.decode("utf-8", "ignore")
    text = re.sub(r"<[^>]+>", " ", html)
    return re.sub(r"\s+", " ", text).strip()[:600]

def probe(url, timeout=5, limit=200_000):
    req = urllib.request.Request(url, headers={"User-Agent": UA})
    try:
        with urllib.request.urlopen(req, timeout=timeout) as r:
            raw = r.read(limit)
            ctype = r.headers.get("Content-Type", "")
            return {"url": url, "status": r.status, "final": r.geturl(),
                    "ctype": ctype, "bytes": len(raw),
                    "text": extract_text(raw, ctype), "error": None}
    except urllib.error.HTTPError as e:
        ctype = e.headers.get("Content-Type", "") if e.headers else ""
        return {"url": url, "status": e.code, "final": url, "ctype": ctype,
                "bytes": 0, "text": "", "error": "http-error"}
    except Exception as e:
        return {"url": url, "status": None, "final": url, "ctype": "",
                "bytes": 0, "text": "", "error": type(e).__name__}

Notice /moved. I gave it a 301 with no Location header, because that happens in the wild — someone moves a page and forgets the destination. urllib raises on it, status comes back as 301, and if you only check status >= 400 you'll call a broken redirect healthy. That was the first rule I had to break.

Step 3: A classifier you can defend

def classify(p):
    if p["status"] is None:
        return "unreachable"
    if 300 <= p["status"] < 400:
        return "broken-redirect"
    if p["status"] in (401, 403):
        return "walled"
    if p["status"] == 429:
        return "throttled"
    if p["status"] >= 400:
        return "dead"
    low = p["text"].lower()
    if any(s in low for s in CONSENT):
        return "walled"
    if any(s in low for s in NOT_FOUND):
        return "soft-dead"
    if p["ctype"].startswith(("application/pdf", "image/")):
        return "opaque"
    if len(p["text"]) < 120:
        return "ambiguous"
    return "alive"

def main(path, delay=1.0):
    urls = [l.strip() for l in open(path) if l.strip() and not l.startswith("#")]
    rows = []
    for u in urls:
        p = probe(u)
        p["label"] = classify(p)
        rows.append(p)
        time.sleep(delay)
    print(json.dumps(rows, indent=2))

if __name__ == "__main__":
    main(sys.argv[1])

With a links.txt of the eight fixtures, python linktriage.py links.txt gives me /ok: alive, /gone: dead, /moved: broken-redirect, /soft404: soft-dead, /cookiewall: walled, /paper: opaque, /slow: unreachable, and /shell: ambiguous after roughly twelve seconds of waiting on the timeout.

Seven out of eight, on the first try. Not bad. Now look at the one that isn't.

Where the rules quietly lie

/shell is the real world. Plenty of course pages now return a 200 with an empty <div id="root"> and paint the content in JavaScript. My heuristic sees almost no text, calls it ambiguous, and is right by accident — it has no idea why the page is thin. Is it a JavaScript shell, a soft 404 that says "Oops!" without the words "not found," or a consent wall that names its cookie banner differently? The phrase list can't tell, and no amount of adding strings fixes that, because the space of human apologies is infinite.

There's a second landmine I want to flag: many sites render a search box in the footer that says "Search the 404 page." Your substring match fires on the nav, not the content. That's how /soft404 and a perfectly healthy page end up in the same bucket. Always look at what the classifier matched before you trust the count.

So I stopped widening the rule set. I narrowed the input instead: only ambiguous rows go to a model.

Step 4: Spend tokens on the leftovers only

Out of 62 real links, the fetch pass put seven in the ambiguous bucket. Each row is a few hundred characters of extracted text, so the classification request is small. I send a constrained prompt and require one word back:

Classify this page excerpt as exactly one of:
  dead, walled, alive, unknown
Reply with the label only, no punctuation, no explanation.

EXCERPT:
{{text}}

Parsing is then a set membership check, not a regex adventure. Anything outside the four labels — a chatty model, a refusal, an empty body — becomes unknown and goes to a human. That last rule matters more than the prompt. A classifier that can't say "I don't know" will happily label a login page "dead" and get your reading list deleted.

This is the honest case for free model access: my usage pattern is spiky and tiny. Seven short calls on a Monday morning, a few hundred tokens each. Even at ten times the volume I'd be nowhere near a paid tier's floor. If your workload is thousands of full pages a day, free access is the wrong tool and you should be measuring latency and rate limits instead of reading this section.

Step 5: Put the weekly job on the free server

I don't want to remember this. The point of the tool is that I forget about it, so it lives on MonkeyCode's free server option and runs as a cron job.

crontab -e
# Mondays 06:00, write a report the rest of the week can quote
0 6 * * 1 cd ~/linktriage && python linktriage.py links.txt > report-$(date +\%F).json 2> err.log

One consequence I didn't expect: the free server made me write the output down. A JSON file per week means I can diff last week against this week and see when a link died, not just that it's dead. A link that flipped on October 20th is a much better email to a professor than "something is broken."

What I'd tell you before you copy this

It has no browser. Any page that needs JavaScript to say what it is will look thin to me, and my ambiguous bucket absorbs that — it does not solve it. It also can't judge whether a live page is correct, only that bytes came back. Be polite: one request per second per host, respect robots.txt, and don't run this against sites whose terms forbid it. And if your list is ten links you check twice a year, skip all of this and open them in tabs — the tooling costs more than the problem.

What you should take away is smaller than the code. Status codes describe transport, not meaning, and the distance between those two things is exactly where your heuristics will fail. Fixtures taught me that faster than any production incident, because a fixture can't change out from under me mid-debug.

Your turn, and I'd rather have a diagnosis than a like. Point the script at http://127.0.0.1:8765/shell and predict what your classifier returns before you run it — then tell me which fixture in your own set came back wrong. If you want to try the model step without a card, the free tier is where I'd start, but bring your own fixtures first. The fixtures are the part that teaches you something.