Curated link lists rot. Everyone knows this in the abstract. What I wanted to know was the shape of the rot: how much of it a normal link checker catches, and how much of it looks perfectly healthy from the outside.
So I took one well-known OSINT tool directory — 12,680 outbound links, 9,889 unique URLs across 7,069 hosts — and checked every single one. Not a sample. Every row.
The headline number is not the dead links. It is the ones that return HTTP 200.
39 links now serve gambling or SEO spam
A status-code checker gives all of these a clean bill of health. Seventeen of the thirty-nine serve the spam content on the originally listed URL, with no redirect at all — the domain simply changed hands and the new owner put a casino on it. Your checker asks "is this 200?", gets "yes", and moves on.
The ones that read worst out of context:
- A row titled "Russian Disinformation Dashboard" now lands on a 1xBet affiliate.
- A row titled "Bellingcat's Online Investigation Toolkit" — the link went to a document-hosting service that shut down, and the path was repurposed. It now serves an Indonesian slot site.
- Two voter-records entries under Public Records / Legal. One serves a live casino page, the other redirects to a betting-app download.
- Three separate links to counter-extremism research. The entire domain now serves "Best Online Casinos Australia 2026" on every path.
Two more are not spam, but no longer are what the row says. One tool listed as a face-search service now redirects to an unrelated mobile game. Another returns 200 with the title "Redirecting..." over a body that is pure adblock-detection JavaScript — the search engine is gone, the monetised shell remains. That one appears four times across two sections, so one fix cleans four rows.
The detection method that works: fetch the page and compare the <title> and final effective domain against what the row claims. Status codes tell you nothing here.
A string-handling bug was hiding 99 working tools
This was the cheapest finding in the whole audit and needed no research at all.
135 rows had a description concatenated into the URL field — two spreadsheet columns merged at some point during collection. They look like this:
http://archive.is / archive.fo
http://blacklistalert.org blacklist check tool
http://checkshorturl.com expand url
http://thatsthem.com / reverse-ip-lookup
Every one of the 135 is broken as rendered. I probed the base URL before the first whitespace on all of them: 99 return HTTP 200. Ninety-nine live, working tools were unreachable purely because of a data-entry artifact.
Truncating each URL at the first whitespace fixes all 99 in one pass. If you maintain a list in a CSV, this is worth grepping for right now.
What I deliberately refused to call dead
This is the part that matters more than any individual finding, and it is where link audits usually destroy their own credibility.
1,096 links returned HTTP 403. I called none of them dead. A 403 during a bulk sweep is a bot defence, not a death. The list included inteltechniques.com — Michael Bazzell's site, one of the most important live resources in the OSINT world — which answers a clean 200 on an individual request. A naive checker would have flagged it as broken.
36 returned 429. That is rate limiting caused by my own request volume. It is a property of my sweep, not of the target.
86 .onion addresses, never probed. Tor hidden services do not resolve from the clearnet at all, so a failed probe proves exactly nothing. Separately worth noting: the 14 clearnet Tor-gateway hosts in the list — onion.pet, onion.to, onion.ws and friends — are genuinely NXDOMAIN now, so those rows are broken even for readers who do not run Tor.
Three .mil hosts and three geo-filtered government portals (China, Russia, Ukraine). These filter by geography and client fingerprint. Unreachable from my vantage point is not the same as gone.
443 additional hosts that resolve but refuse connections on every scheme. Probably mostly dead. I am not asserting it. "Resolves yet refuses" can equally be a firewall, a geo-block, or an origin that is down this afternoon.
I would rather hand a maintainer 513 findings they can trust than 956 they cannot. One wrong claim in a report like this and nothing else in it gets read.
The two-pass rule
A single-pass curl sweep produces a pile of false positives. The second pass is not optional.
Pass 1 — cheap, parallel, over everything:
curl -sS -o /dev/null -w '%{http_code} %{url_effective}\n' \
-L --max-time 20 -A "$BROWSER_UA" "$URL"
Pass 2 — only for what looked dead, and properly:
- try both schemes,
http://andhttps://, and both with and withoutwww - send a full browser header set, not just a User-Agent (
Accept,Accept-Language,Sec-Fetch-*,--compressed) — many hosts reject a bare curl and serve a real browser fine - resolve the domain:
dig +short <domain>— no A/AAAA record across multiple public resolvers is the only really solid death - follow redirects and look at the final URL; a redirect to a parking page is a different finding from a 404 and deserves its own bucket
On this dataset the second pass removed 149 false deaths that the first pass had flagged. On a smaller list I audited earlier — 428 outbound links — it removed 5 false positives out of 29. Skip it and roughly a sixth of your report is wrong.
Have a skeptic re-check your own findings
After the audit was written, I ran every single death claim through a second independent pass whose only job was to disprove them: re-resolve against three public resolvers plus two DoH providers, five curl probes per URL, and a confirmation from a different physical network.
It killed two claims — two domains I had bucketed as dead were alive. It also caught something subtler: I had written that a particular dead domain "has no nameserver delegation at all", and whois shows three delegated nameservers. The delegation exists; the in-bailiwick nameservers simply do not answer, which is why every resolver returns SERVFAIL. The link is dead either way, but the stated evidence was falsifiable in one command, and that is exactly the kind of thing a maintainer checks first.
If you publish findings about someone else's work, build the refutation step into the process. Your own second pass is cheaper than their loss of trust.
Other categories worth knowing about
117 rows where HTTPS fails but HTTP works. These hosts serve nothing on port 443 and respond normally on port 80. Chrome and Firefox both do HTTPS-first now, so a reader clicking through sees a connection failure and concludes the tool is dead. It is not — the row just needs its scheme changed. Jeffrey Friedl's EXIF viewer is in this bucket.
37 rows with TLS certificates that fail verification — but split them into two tiers, because they do not behave the same way. Expired (verify code 10) and self-signed (18) throw a full browser interstitial. Incomplete-chain (20/21) usually loads anyway, because Chrome fetches the missing intermediate via AIA. Reporting both tiers as "broken TLS" overstates the problem.
498 rows returning 404 while the parent site is perfectly healthy. Deep-link rot. The organisation is fine, the page moved. These are cheap to repair by repointing at the site root, and lumping them in with genuine deaths is what makes naive audits look catastrophic.
One row shipping http://127.0.0.1:5001 as a public link. Worth grepping your own data for private-range addresses before any release.
Dead tools that have living successors
The most useful column in an audit like this is not "dead" — it is "dead, and here is what replaced it". A few from this run, each verified 200 at the time of writing:
| Dead | Living successor |
|---|---|
botometer.iuni.iu.edu |
botometer.osome.iu.edu — same project, same team, retired hostname |
gigablast.com (engine dissolved, 2023) |
mojeek.com and marginalia-search.com — the surviving independent crawlers |
spyse.com (shut down 2022) |
netlas.io — where the userbase actually migrated |
findmyfbid.com |
lookup-id.com — same function |
browsershots.org |
browsershots.at — the official continuation |
darksearch.io |
ahmia.fi — maintained clearnet-accessible Tor index |
| ASKfm (shut down Dec 2024) | revived in 2026 at askfm.app
|
That last one is a reminder to check before deleting a row. A tool being gone in 2024 does not mean it is gone now.
If you maintain a list
The audit is mechanical and you can run it yourself with the two-pass recipe above. If you would rather not, I will run it over your list and send you the CSV — findings only, no strings, and privately rather than published. I did exactly that with the directory in this post, which is why it is not named here: the point is the pattern, not the pillory. Most of those 39 rows were completely fine on the day they were added.
I build Face2social, a face search engine over public social media profiles — which is how I ended up reading a lot of OSINT tool directories in the first place. It indexes public social profile pictures only and does not crawl the open web, so for whole-web face search PimEyes and FaceCheck.ID cover far more ground. US only.
Top comments (1)
The 135-rows-merged-into-the-URL-field finding is the most useful part of this. 99 of them returned 200 behind the first whitespace, so a checker that resolves the whole rendered href reports them as broken and nobody looks again. Truncating at the first space is a five-line fix that hides for months because the list still looks maintained.
Comparing the fetched title against what the row claims is the right signal, and the 'Redirecting...' adblock-detection shell is exactly what a status code can't see. Did you keep the redirect chain itself (final effective URL after every hop) in the comparison, or only the last title? A row that 301s to a different registration domain but still renders the original heading would pass a title-only check.