DEV Community: David Anderson

I checked 12,680 links in an OSINT tool directory. 39 of them now serve casino spam.

David Anderson — Mon, 14 Sep 2026 14:09:34 +0000

Curated link lists rot. Everyone knows this in the abstract. What I wanted to know was the shape of the rot: how much of it a normal link checker catches, and how much of it looks perfectly healthy from the outside.

So I took one well-known OSINT tool directory — 12,680 outbound links, 9,889 unique URLs across 7,069 hosts — and checked every single one. Not a sample. Every row.

The headline number is not the dead links. It is the ones that return HTTP 200.

39 links now serve gambling or SEO spam

A status-code checker gives all of these a clean bill of health. Seventeen of the thirty-nine serve the spam content on the originally listed URL, with no redirect at all — the domain simply changed hands and the new owner put a casino on it. Your checker asks "is this 200?", gets "yes", and moves on.

The ones that read worst out of context:

A row titled "Russian Disinformation Dashboard" now lands on a 1xBet affiliate.
A row titled "Bellingcat's Online Investigation Toolkit" — the link went to a document-hosting service that shut down, and the path was repurposed. It now serves an Indonesian slot site.
Two voter-records entries under Public Records / Legal. One serves a live casino page, the other redirects to a betting-app download.
Three separate links to counter-extremism research. The entire domain now serves "Best Online Casinos Australia 2026" on every path.

Two more are not spam, but no longer are what the row says. One tool listed as a face-search service now redirects to an unrelated mobile game. Another returns 200 with the title "Redirecting..." over a body that is pure adblock-detection JavaScript — the search engine is gone, the monetised shell remains. That one appears four times across two sections, so one fix cleans four rows.

The detection method that works: fetch the page and compare the <title> and final effective domain against what the row claims. Status codes tell you nothing here.

A string-handling bug was hiding 99 working tools

This was the cheapest finding in the whole audit and needed no research at all.

135 rows had a description concatenated into the URL field — two spreadsheet columns merged at some point during collection. They look like this:

http://archive.is / archive.fo
http://blacklistalert.org blacklist check tool
http://checkshorturl.com expand url
http://thatsthem.com / reverse-ip-lookup

Every one of the 135 is broken as rendered. I probed the base URL before the first whitespace on all of them: 99 return HTTP 200. Ninety-nine live, working tools were unreachable purely because of a data-entry artifact.

Truncating each URL at the first whitespace fixes all 99 in one pass. If you maintain a list in a CSV, this is worth grepping for right now.

What I deliberately refused to call dead

This is the part that matters more than any individual finding, and it is where link audits usually destroy their own credibility.

1,096 links returned HTTP 403. I called none of them dead. A 403 during a bulk sweep is a bot defence, not a death. The list included inteltechniques.com — Michael Bazzell's site, one of the most important live resources in the OSINT world — which answers a clean 200 on an individual request. A naive checker would have flagged it as broken.

36 returned 429. That is rate limiting caused by my own request volume. It is a property of my sweep, not of the target.

86 .onion addresses, never probed. Tor hidden services do not resolve from the clearnet at all, so a failed probe proves exactly nothing. Separately worth noting: the 14 clearnet Tor-gateway hosts in the list — onion.pet, onion.to, onion.ws and friends — are genuinely NXDOMAIN now, so those rows are broken even for readers who do not run Tor.

Three .mil hosts and three geo-filtered government portals (China, Russia, Ukraine). These filter by geography and client fingerprint. Unreachable from my vantage point is not the same as gone.

443 additional hosts that resolve but refuse connections on every scheme. Probably mostly dead. I am not asserting it. "Resolves yet refuses" can equally be a firewall, a geo-block, or an origin that is down this afternoon.

I would rather hand a maintainer 513 findings they can trust than 956 they cannot. One wrong claim in a report like this and nothing else in it gets read.

The two-pass rule

A single-pass curl sweep produces a pile of false positives. The second pass is not optional.

Pass 1 — cheap, parallel, over everything:

curl -sS -o /dev/null -w '%{http_code} %{url_effective}\n' \
  -L --max-time 20 -A "$BROWSER_UA" "$URL"

Pass 2 — only for what looked dead, and properly:

try both schemes, http:// and https://, and both with and without www
send a full browser header set, not just a User-Agent (Accept, Accept-Language, Sec-Fetch-*, --compressed) — many hosts reject a bare curl and serve a real browser fine
resolve the domain: dig +short <domain> — no A/AAAA record across multiple public resolvers is the only really solid death
follow redirects and look at the final URL; a redirect to a parking page is a different finding from a 404 and deserves its own bucket

On this dataset the second pass removed 149 false deaths that the first pass had flagged. On a smaller list I audited earlier — 428 outbound links — it removed 5 false positives out of 29. Skip it and roughly a sixth of your report is wrong.

Have a skeptic re-check your own findings

After the audit was written, I ran every single death claim through a second independent pass whose only job was to disprove them: re-resolve against three public resolvers plus two DoH providers, five curl probes per URL, and a confirmation from a different physical network.

It killed two claims — two domains I had bucketed as dead were alive. It also caught something subtler: I had written that a particular dead domain "has no nameserver delegation at all", and whois shows three delegated nameservers. The delegation exists; the in-bailiwick nameservers simply do not answer, which is why every resolver returns SERVFAIL. The link is dead either way, but the stated evidence was falsifiable in one command, and that is exactly the kind of thing a maintainer checks first.

If you publish findings about someone else's work, build the refutation step into the process. Your own second pass is cheaper than their loss of trust.

Other categories worth knowing about

117 rows where HTTPS fails but HTTP works. These hosts serve nothing on port 443 and respond normally on port 80. Chrome and Firefox both do HTTPS-first now, so a reader clicking through sees a connection failure and concludes the tool is dead. It is not — the row just needs its scheme changed. Jeffrey Friedl's EXIF viewer is in this bucket.

37 rows with TLS certificates that fail verification — but split them into two tiers, because they do not behave the same way. Expired (verify code 10) and self-signed (18) throw a full browser interstitial. Incomplete-chain (20/21) usually loads anyway, because Chrome fetches the missing intermediate via AIA. Reporting both tiers as "broken TLS" overstates the problem.

498 rows returning 404 while the parent site is perfectly healthy. Deep-link rot. The organisation is fine, the page moved. These are cheap to repair by repointing at the site root, and lumping them in with genuine deaths is what makes naive audits look catastrophic.

One row shipping http://127.0.0.1:5001 as a public link. Worth grepping your own data for private-range addresses before any release.

Dead tools that have living successors

The most useful column in an audit like this is not "dead" — it is "dead, and here is what replaced it". A few from this run, each verified 200 at the time of writing:

Dead	Living successor
`botometer.iuni.iu.edu`	`botometer.osome.iu.edu` — same project, same team, retired hostname
`gigablast.com` (engine dissolved, 2023)	`mojeek.com` and `marginalia-search.com` — the surviving independent crawlers
`spyse.com` (shut down 2022)	`netlas.io` — where the userbase actually migrated
`findmyfbid.com`	`lookup-id.com` — same function
`browsershots.org`	`browsershots.at` — the official continuation
`darksearch.io`	`ahmia.fi` — maintained clearnet-accessible Tor index
ASKfm (shut down Dec 2024)	revived in 2026 at `askfm.app`

That last one is a reminder to check before deleting a row. A tool being gone in 2024 does not mean it is gone now.

If you maintain a list

The audit is mechanical and you can run it yourself with the two-pass recipe above. If you would rather not, I will run it over your list and send you the CSV — findings only, no strings, and privately rather than published. I did exactly that with the directory in this post, which is why it is not named here: the point is the pattern, not the pillory. Most of those 39 rows were completely fine on the day they were added.

I build Face2social, a face search engine over public social media profiles — which is how I ended up reading a lot of OSINT tool directories in the first place. It indexes public social profile pictures only and does not crawl the open web, so for whole-web face search PimEyes and FaceCheck.ID cover far more ground. US only.

How we search a face across 1 billion photos in under a second (with FAISS)

David Anderson — Thu, 27 Aug 2026 19:11:03 +0000

Disclosure: I'm the founder of Face2social, a face-recognition search engine. This post is about the vector-search engineering behind it — no product pitch, just the stuff that kept me up at night (including the part where I tell you what we're bad at). It's also the write-up I wish I'd found when I started.

TL;DR

A face becomes a vector. Similarity search becomes nearest-neighbor search in high-dimensional space.
Brute force dies somewhere around a few million vectors. We're at ~1 billion.
FAISS is what makes "compare against a billion" feel like "compare against a thousand."
The real trick isn't one magic index — it's the trade-off triangle: speed ↔ memory ↔ recall. You don't get all three.
And the biggest trade-off isn't even in the index. It's what you choose to put in it — more on that, honestly, near the end.

If you've ever wondered how "search by image" actually works at scale, this one's for you. 👇

The problem, honestly stated

Someone uploads a photo. We need to answer one question fast: does this face appear anywhere in a very large set of public profile pictures?

The naive version is easy to describe and impossible to ship:

Turn the uploaded face into a vector. Turn every known face into a vector. Compare the new vector against all of them. Return the closest matches.

That last step is the whole ballgame. Compare against a thousand faces? Trivial. A million? Now you're sweating. A billion? A single brute-force scan would have a user staring at a spinner long enough to close the tab, make coffee, and reconsider their life choices.

So the interesting engineering isn't "how do we compare two faces." It's "how do we not compare against all billion and still find the right one."

Step 1: a face is just a point in space

Before any search happens, every face — the uploaded one and every one we already know about — goes through the same pipeline:

detect the face → align it → turn it into a fixed-length vector (an "embedding").

The important intuition: two photos of the same person land close together in this space, even across different angles, lighting, crops, or filters. Two different people land far apart. "How similar are these faces?" collapses into "how close are these two points?" — a plain distance measurement.

That's the quiet superpower of embeddings: they turn a messy perceptual problem ("do these look like the same person?") into clean geometry ("what's the distance between these vectors?"). Everything downstream is just... finding nearby points. A lot of nearby points. Very fast.

(I'm deliberately staying vendor-neutral on the embedding model itself — pick your favorite; the search problem below is the same regardless.)

Step 2: why brute force falls off a cliff

Exact nearest-neighbor search means measuring the distance from your query to every stored vector, then sorting. It's O(n) per query, and each comparison is real work in a high-dimensional space.

Here's the mental model for why it doesn't scale:

A thousand vectors: instant. Do it in a loop. Nobody cares.
A million: still fine on a good box, especially batched.
A hundred million: now every query is chewing through serious memory bandwidth, and your p99 latency is a horror story.
A billion: the vectors alone don't comfortably fit in RAM on one machine, and a full scan per query is simply off the table for anything interactive.

The wall isn't CPU cleverness. It's physics — memory and bandwidth. You cannot touch a billion things per request and stay under a second. So you stop being exact.

Step 3: approximate is the whole point

This is the mindset shift that makes billion-scale search possible:

You don't need the mathematically perfect nearest neighbor. You need a very good neighbor, almost always, fast.

That's Approximate Nearest Neighbor (ANN) search, and it's where FAISS (Facebook AI Similarity Search) earns its keep. FAISS is a library built for exactly one job: nearest-neighbor search over massive vector sets, with a menu of indexes that each pick a different spot on the trade-off triangle.

Two ideas do most of the heavy lifting. Neither is exotic once you see the intuition:

1. Don't search everything — search a neighborhood.
Instead of scanning all billion vectors, you pre-cluster them into buckets during an offline training step. At query time you figure out which few buckets your query probably lives in, and you only scan those. You've quietly turned "compare against a billion" into "compare against a few million" — the same answer, a tiny fraction of the work. The knob for how many buckets you probe is your dial between speed and recall.

2. Don't store fat vectors — store compressed sketches.
Full-precision embeddings are heavy, and a billion of them is a lot of RAM. So you compress each vector into a compact code that's cheap to store and cheap to compare — trading a little precision for a massive memory win. This is what makes "a billion vectors" go from rent-a-data-center to actually-runs. You lose a hair of accuracy; you gain the ability to exist.

Stack those two — search a neighborhood, over compressed sketches — and a billion-vector search starts to feel routine. I'm intentionally not naming the exact FAISS index recipe we run in production (that part's ours to keep), but everything above is the standard playbook, and FAISS gives you all the building blocks out of the box.

Step 4: the trade-off triangle nobody escapes

Here's the thing I most wish someone had tattooed on my monitor early on. Every knob you turn is a negotiation between three things, and you only get to optimize two:

⚡ Speed — how fast a single query comes back.
🧠 Memory — how much RAM the index eats (this is your infra bill).
🎯 Recall — how often the true match is actually in your results.

Want blazing speed and tiny memory? Recall suffers — you'll miss real matches. Want near-perfect recall and speed? Get your wallet out for RAM. Want cheap and accurate? It'll be slow.

There is no free lunch. The entire job is choosing which corner to sacrifice for your use case — and for a face-search product, quietly missing a real match is the worst failure mode, so we bias toward recall and pay for it elsewhere. Your product's priorities will land you somewhere different, and that's the point: the "right" index is the one that fits your failure modes, not a leaderboard.

Step 5: one machine ends, sharding begins

Even compressed, a billion vectors plus the overhead to serve them will eventually blow past a single machine. So you shard: split the index into pieces across several machines, run the query against all shards in parallel, then merge the top results.

Conceptually it's the classic scatter–gather:

query ──► [ shard 1 ]─┐
      ├──► [ shard 2 ]─┤
      ├──► [ shard 3 ]─┼──► merge top-K ──► results
      └──► [ shard N ]─┘

Your effective latency becomes the slowest shard plus a cheap merge — not the sum. Add shards to grow capacity, keep each shard small enough to stay fast, and you've got a search tier that scales sideways instead of falling over.

Step 6: the trade-off that actually defines the product

Everything above is about the index. But the decision that shapes your product most isn't how you search — it's what you put in the corpus. And this is where I should be straight with you, because it's our biggest limitation.

We index social media profile pictures. That's it. Instagram, Facebook, TikTok, X.

The upside is real: a narrow, dense, well-understood corpus means the index is tuned for one kind of photo, matches come back fast, and when someone asks "does this face have a social profile?" we're very good at answering it.

The downside is just as real, and I'd rather say it out loud than have a user discover it: we're blind to everything else. News articles, personal blogs, forums, company team pages, image boards, the open web in general — we don't see any of it. A face that's all over the web but has no social presence will come back empty from us, and that's not a bug we're about to fix; it's a direct consequence of the corpus we chose. Tools that crawl the open web will beat us on exactly those queries. They'll also index a much noisier space and inherit different problems, but that's their trade-off, not a defense of ours.

So: if your question is "is this person on social media, and where?" — that's our lane and we're strong in it. If your question is "where does this face appear anywhere online?" — we are the wrong tool, and you should use something that crawls the whole web.

I'm spelling this out because the corpus decision gets glossed over in most write-ups about vector search, and it's the one your users actually feel. A perfectly tuned index over the wrong data is still the wrong answer, delivered quickly. Recall against your index means nothing if the thing you're looking for was never in it. Define your corpus honestly, then tell people where its edges are.

What I'd tell past me

The embedding model gets the glory; the index does the work. A great model that you can't search at scale is a demo, not a product.
"Approximate" is not a compromise you apologize for — it's the enabling idea. Chasing exact nearest-neighbor at a billion scale is how you burn a quarter and ship nothing.
Pick your sacrifice on purpose. Speed, memory, recall — decide before you tune which one you're willing to bleed, based on what "wrong" costs your users.
Your corpus is a product decision, not a data decision. Narrow and deep beats broad and shallow — as long as you're upfront about the edges.
FAISS is genuinely the boring-in-a-good-way choice. Battle-tested, flexible, and it lets you move the trade-off dials instead of reinventing them.

We run this in production at Face2social (US-only, for the privacy-law reasons you'd expect) to match an uploaded photo against public social profiles — mostly for catfish and romance-scam checks, and for people who want to know whether their own photos are being used on fake accounts. But honestly, swap "faces" for "product images," "audio fingerprints," or "document embeddings" and the entire playbook above is identical. Vector search at scale is one of those skills that quietly transfers everywhere.

If you're building anything that searches by similarity, learn FAISS, internalize the trade-off triangle, and be honest about your corpus. Future-you will thank present-you. 🙏

What are you using for large-scale vector search — FAISS, or something else? And how did you scope your corpus? Curious what trade-offs you landed on. Drop it in the comments.