DEV Community

Ai-Q Labs
Ai-Q Labs

Posted on

I used the Hacker News API to check 5,000 old links. 34 domains are now unregistered.

Hacker News never rechecks a link.

A story submitted in 2012 keeps its points, its comment thread and its URL forever. The site will happily send you to that URL in 2026 whether or not anything is still there.

I wanted a base rate for that, specifically for HN rather than for the web in general, so I wrote a checker and pointed it at the archive. The numbers below are what it produced.

What I measured

Everything here comes from two official APIs. The tool reads hacker-news.firebaseio.com/v0/ for items and hn.algolia.com/api/v1/ for search. It never fetches news.ycombinator.com itself.

For each story URL it does a HEAD, falls back to GET, and follows redirects. For anything that fails it then does two more things:

  1. asks the Wayback Machine for the capture closest to the submission date — not the newest one, which is often a parking page
  2. resolves the registrable domain against the authoritative RDAP registry, found through the IANA bootstrap file

Base rates, by population:

Population Outbound links dead
Stories 2007–2011, above 300 points 24.5%
Stories 2012–2014, above 300 points 16.5%
Today's front page 0.0%

Those are three different samples, not one trend line, and the front-page 0% is the useful control: whatever the tool is measuring, it is not firing on healthy links.

In one end-to-end run over 60 stories from 2012–2013 above 300 points: 16 of 57 links dead (28.1%), and 13 of those 16 had a usable archived copy from within a few days of the submission date (81.3%). 33 of the 57 came back clean.

The two things I got wrong first

This is the part I would have wanted to read before writing my own version.

1. Between 10% and 16% of URLs answer 401, 403 or 429

My first pass counted those as dead. That roughly doubled the rot rate, and about half of the result was false.

These are bot blocks. The server is alive and declining to talk to a program. Whether a human clicking through from HN reaches a page is simply not established by that response, so it cannot be reported either way.

They now get their own bucket at the lowest severity, and I do not retry them with a browser user-agent. A refusal is an answer. Working around it would turn a measurement into an argument.

2. Some dead links answer 200 OK

Between 1% and 3.5% of old links now 301 to the site's front page and return 200. The path is dropped; the article is gone; every naive checker calls it healthy. On today's front page this is 0%, which is what you would expect — it is a symptom of years of site migrations.

This is the failure mode I actually built the thing for. It is invisible from HN, invisible in a status-code check, and it is silently eating old references.

A third one worth mentioning because it is a non-finding: http:// accounts for over 80% of links in old stories, and most of them upgrade to HTTPS on their own. Flagging bare http:// would have put a warning on the majority of perfectly healthy links, which is the fastest way to make a report worthless.

The part I did not expect

Across the 5,000 stories the sweep covered, there were 2,127 registrable domains. 133 no longer resolve. 34 of those are not registered at all right now.

Some of them carried real threads:

Domain Story Points Year
ndy.gd PayPal CEO writes personal response 619 2012
danielhope.org 9 year old's DIY Cardboard Arcade gets Flash mobbed 603 2012
stopsoap.com Stop SOAP 598 2012
paulasmuth.com Dear Github, please fix the Watch-Button 477 2012

I rechecked all four against the authoritative registries immediately before publishing this — Verisign for .com, PIR for .org, CentralNic for .gd — and all four return 404. From the same servers, shopify.com, python.org and google.gd return 200, so this is not a registry handing out blanket negatives.

Anyone can register those four addresses and decide what a decade of accumulated inbound links now points at. Hacker News will keep vouching for them.

Limitations

I would rather list these than have them found.

  • Algolia's counts are not always exact. exhaustiveNbHits stays false on large result sets, and nbHits then reports the size of the whole index — 44M items — instead of your query. It looks like a real number. The tool splits queries by year to stay inside the regime where the count is trustworthy, and both branches are pinned by live tests.
  • RDAP does not cover every TLD. The IANA bootstrap file lists about 1,200. .io, .co and .jp are not among them. For those the tool returns unknown rather than guessing "available" — the one mistake in this category that actually costs somebody money.
  • Restricted TLDs are never called takeable. change.gov from 2008 returns 404 from RDAP, but .gov requires US government sponsorship, so "anyone can register this" would be a confident lie at the highest severity. .gov, .mil, .edu, .int and .arpa are separated out and worded differently.
  • Every URL is checked once. A 5xx may just be a bad minute.
  • Ask HN and Show HN text posts have no outbound link. They come back as text_post with no findings rather than being counted as healthy, which would quietly inflate the clean rate.

The tool

It runs on Apify and is free: https://apify.com/aiqlabs/hacker-news-link-rot

Running it needs a free Apify account. The README, the input schema and a sample of the output are readable without one.

Give it a slice of HN — a year range, a points threshold, a list of story ids, or today's front page — and it returns, per story, whether the link works, what kind of dead it is if not, and the archived copy from around the time people were discussing it.

If you look at one thing, make it the false-positive handling. That is the part I am least confident in, and it is the part that decides whether a report like this is worth reading.

Top comments (0)