DEV Community

Ken-Mutisya
Ken-Mutisya

Posted on

"Stop Scraping WHOIS. The Registries Serve JSON Now"

#api #node #webdev #showdev

Every few months someone asks how to scrape WHOIS at scale, and the answers are always the same: port 43 sockets, regex soup for 200 registrar formats, and rate limits that ban you by lunchtime. Almost nobody mentions that the registries replaced the whole thing years ago with a JSON protocol called RDAP, and it is open, keyless, and standardized.

One GET, structured JSON 

GET https://rdap.verisign.com/com/v1/domain/stripe.com
Accept: application/rdap+json
Enter fullscreen mode Exit fullscreen mode

You get events (registration, expiration, last changed), status locks, nameservers, DNSSEC delegation, and the registrar as a structured entity. No text parsing. The dates are ISO 8601. The response for a .com domain comes straight from Verisign, the registry itself, not a reseller or a cached mirror.

{
  "events": [
    { "eventAction": "registration", "eventDate": "1995-09-12T04:00:00Z" },
    { "eventAction": "expiration", "eventDate": "2027-09-11T04:00:00Z" }
  ],
  "status": ["client delete prohibited", "client transfer prohibited"],
  "nameservers": [{ "ldhName": "NS1.STRIPE.COM" }]
}
Enter fullscreen mode Exit fullscreen mode

Routing: the IANA bootstrap file

Different TLDs live at different registries, and IANA publishes the routing table as plain JSON:

GET https://data.iana.org/rdap/dns.json
Enter fullscreen mode Exit fullscreen mode

It maps about 1,200 TLDs to their RDAP base URLs. Fetch it once, build a Map of TLD to endpoint, and query each domain at its own registry. Concurrency 8 with a 15 second timeout checks 2,000 domains in a couple of minutes, and nobody rate walls you, because you are spreading standard queries across the services built to answer them.

const boot = await (await fetch('https://data.iana.org/rdap/dns.json')).json();
const map = new Map();
for (const [tlds, urls] of boot.services) {
  const base = urls.find(u => u.startsWith('https://'));
  for (const tld of tlds) map.set(tld, base);
}
Enter fullscreen mode Exit fullscreen mode

The 404 is a feature

An RDAP 404 from the authoritative registry means the domain is not registered. That turns a registration data checker into a bulk availability checker for free. But only trust the 404 when it comes from the TLD's own registry. Aggregators like rdap.org return 404 both for "domain not found" and "TLD not supported", which will happily tell you github.io is available.

The gaps are ccTLDs

ICANN requires RDAP for every gTLD, so .com, .net, .org and the hundreds of new TLDs have full coverage. Country TLDs are voluntary: .ai, .tv, .cc and .uk are in the bootstrap, while .io and .sh are missing from it but quietly served by the registry operator at rdap.identitydigital.services/rdap/. Others (.de, .co, .me) still have nothing public. Handle the miss case explicitly instead of guessing.

Parsing the registrar

The registrar hides inside a jCard, which is vCard reimagined as nested JSON arrays. The name is the fn entry, and the IANA registrar ID is in publicIds:

const reg = data.entities?.find(e => e.roles?.includes('registrar'));
const name = reg?.vcardArray?.[1]?.find(i => i[0] === 'fn')?.[3];
Enter fullscreen mode Exit fullscreen mode

One postscript on privacy: registrant name and email are redacted nearly everywhere since GDPR, in RDAP and WHOIS alike. What registries still publish for every domain is the part that matters for most workflows anyway: dates, registrar, locks, nameservers. Domain age alone sorts a lead list by business credibility, flags a phishing lookalike registered last Tuesday, and prices an expired domain hunt.

I packaged this whole flow (bootstrap routing, ccTLD supplements, jCard parsing, availability detection) into a pay per row actor: Domain WHOIS & Age Checker. Paste domains, get rows. It is free to try until July 24. But if you would rather build it yourself, everything above is the entire recipe.

Top comments (0)