tags: node, webdev, showdev, testing
I had a list of 115 domains and one question: which of these are still alive?
The naive version is ten lines. Fetch each one, check the status code, done. I wrote that, ran it, and it got 12 of the 115 wrong — in both directions. Here is what it missed, because every one of these is a different failure wearing the same disguise.
1. A 403 is almost never a dead site
Cloudflare's challenge page returns HTTP 403 to anything that doesn't look like a browser. The body is about 23 words and the title is Just a moment....
My first pass called every one of those dead. That threw away eight perfectly good domains, including Sooper Articles, The Free Library, Diigo, HubPages, Typepad and the SitePoint and Joomla forums. All alive. All reachable by a human. All reported as gone.
if (res.status === 403 || /just a moment|attention required/i.test(title)) {
return { verdict: "LIVE", note: "bot wall, host is up" };
}
The trade-off is real: a genuinely dead host that happens to 403 now reads as alive. For my use case that is the cheaper mistake by a wide margin. Deleting a working entry costs me something real. Keeping a dead one costs five minutes when someone tries it.
2. Parked domains return a cheerful 200
A domain that lapsed and got picked up by a parking service is up. It resolves, it serves HTML, it returns 200. It is also completely useless.
My first attempt matched on parking-service brand names — godaddy, sedo, bodis, namecheap. That is the obvious approach and it is wrong, because plenty of live sites load a script, a font or an analytics beacon from one of those hosts.
It confidently reported that Zotero was for sale.
The fix is to match only phrases that appear on a genuine sale page and nowhere else:
const FOR_SALE = /\b(this domain (name )?is for sale|buy this domain|make an offer on this domain)\b/i;
Narrow and boring beats clever here. The other parking pattern worth catching is the JS bounce, where the page is a stub that redirects to /lander on load.
3. Byte length is not a proxy for content
I tried "is the response under N bytes" as a shortcut for "this page is empty". It fails immediately on anything client-rendered — an SPA ships a big HTML file with a <div id="root"> and nothing else.
Strip the scripts and the markup first, then count what a human would actually read:
const wordCount = (html) =>
html.replace(/<script[\s\S]*?<\/script>/gi, "")
.replace(/<style[\s\S]*?<\/style>/gi, "")
.replace(/<[^>]+>/g, " ")
.replace(/\s+/g, " ").trim()
.split(" ").filter(Boolean).length;
Under about 30 words and there is no page there, whatever the byte count says.
4. A connection error is several different problems
fetch throwing tells you almost nothing on its own. These all land in the same catch block and mean very different things:
| Code | What it actually means |
|---|---|
ENOTFOUND |
The host does not resolve. Genuinely gone. |
ECONNREFUSED |
DNS resolves, nothing is listening. |
CERT_HAS_EXPIRED |
The site works fine in a browser after a click-through. |
ERR_TLS_CERT_ALTNAME_INVALID |
Certificate does not cover this hostname. Often a live site behind a misconfigured proxy. |
AbortError |
Slow, not dead. Worth a retry before you write it off. |
Lumping the last three in with the first is how you delete working entries.
What I ended up with
Four verdicts instead of a boolean:
- LIVE — real page, or a bot wall, which still means a human can reach it
- DEAD — does not resolve, refused, expired cert, 4xx or 5xx
- PARKED — for sale, or bouncing to a parking lander
- EMPTY — 200 with essentially nothing rendered
$ node check.mjs sites.txt
LIVE sooperarticles.com bot wall, could not read the page but the host is up
LIVE zotero.org
DEAD some-old-directory.com host does not resolve
3 checked LIVE 2 DEAD 1
No dependencies, Node 18+, MIT. It's here if it's useful: github.com/damani-pixel/citation-check
The broader lesson, and the reason I wrote this up rather than just fixing it quietly: my first version was confidently wrong. It printed a clean report with no errors and no warnings, and a third of the failures in it were mine. If you are writing anything that classifies the outside world, the interesting work is not the happy path — it is enumerating the ways the outside world lies to you.
Top comments (1)
The identity problem in #1 gets worse if you ever point a checker at a URL rather than a domain root. I hit this yesterday with DEV's own API:
GET /api/articles/<id>returned{"error":"not found","status":404}for a draft of mine that exists and opens fine in my dashboard, and sending the owner api-key did not change it, so the 404 described what that endpoint exposes to that caller rather than the resource. The cheap discriminator is to fetch the same URL twice, once with credentials and once without, and treat any disagreement as a fact about the representation instead of liveness.