If you've ever tried to run WHOIS for a few hundred domains, you know the pain: every registry formats its text differently, port 43 servers throttle you, and half the fields are redacted anyway. The good news is that you don't need WHOIS any more. Generic TLD registries have been required to run RDAP (Registration Data Access Protocol) since 2019, and ICANN let them switch off the old WHOIS service in January 2025. RDAP is plain HTTPS returning JSON, with the same structure everywhere.
This post builds one script that takes a messy list of domains and returns, for each one: registrar, creation date, expiry date, domain age, days until expiry, EPP status codes, nameservers, and whether the domain is unregistered. It's useful for lead qualification (older domains are usually real businesses), brand and typo-squat monitoring, expiring-domain lists, and checking your own portfolio.
How RDAP lookup works
- Find the right server. IANA publishes a bootstrap file, data.iana.org/rdap/dns.json, that maps each TLD to its registry's RDAP base URL. Download it once per run.
-
Ask for the domain.
GET <base>/domain/example.comreturns JSON. A 404 means the name isn't registered, which gives you an availability check for free. -
Read the fields. Dates are in
events(registration,expiration,last changed), the registrar is the entity with theregistrarrole (its name is in a jCardfnfield), status codes are instatus, and nameservers are innameservers[].ldhName.
The script
# Usage: python3 whois_rdap_bulk.py domains.txt
# Bulk WHOIS replacement: looks up every domain over RDAP (JSON) in parallel and writes whois_report.csv
# with registrar, created/expires dates, domain age, days until expiry, EPP status, nameservers, and availability.
import csv, sys, json, time, datetime as dt, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor
UA = {'User-Agent': 'rdap-bulk-check/1.0 (+python)', 'Accept': 'application/rdap+json'}
MAX_WAIT = 10 # some registries answer 429 with Retry-After of several hours: give up instead of hanging
NOW = dt.datetime.now(dt.timezone.utc)
# Registries that run RDAP but are missing from IANA's bootstrap file.
EXTRA = {'io': 'https://rdap.identitydigital.services/rdap/', 'co': 'https://rdap.registry.co/co/',
'us': 'https://rdap.nic.us/', 'me': 'https://rdap.identitydigital.services/rdap/'}
def get_json(url, tries=3):
for i in range(tries):
try:
with urllib.request.urlopen(urllib.request.Request(url, headers=UA), timeout=15) as r:
return 200, json.load(r)
except urllib.error.HTTPError as e:
wait = int(e.headers.get('Retry-After') or 2 * (i + 1)) if str(e.headers.get('Retry-After') or '1').isdigit() else 5
if e.code in (429, 500, 502, 503) and i < tries - 1 and wait <= MAX_WAIT:
time.sleep(wait); continue
return e.code, None
except Exception:
if i == tries - 1: return 0, None
time.sleep(1)
def load_bootstrap():
_, data = get_json('https://data.iana.org/rdap/dns.json')
servers = {tld: urls[0] for tlds, urls in data['services'] for tld in tlds}
servers.update({k: v for k, v in EXTRA.items() if k not in servers})
return servers
def normalize(line):
d = line.strip().lower()
for p in ('https://', 'http://'): d = d.removeprefix(p)
d = d.split('/')[0].split('@')[-1].split(':')[0].removeprefix('www.').rstrip('.')
return d if '.' in d else None
def parse_date(s):
return dt.datetime.fromisoformat(s.replace('Z', '+00:00')) if s else None
def lookup(domain, servers):
row = {'domain': domain}
base = servers.get(domain.rsplit('.', 1)[-1])
if not base:
return {**row, 'result': 'no-rdap'}
code, j = get_json(base.rstrip('/') + '/domain/' + domain)
if code == 404:
return {**row, 'result': 'available'}
if code != 200 or not j:
return {**row, 'result': 'rate-limited' if code == 429 else f'error {code}'}
ev = {e.get('eventAction'): e.get('eventDate') for e in j.get('events', [])}
created, expires = parse_date(ev.get('registration')), parse_date(ev.get('expiration'))
registrar = next((e for e in j.get('entities', []) if 'registrar' in e.get('roles', [])), {})
name = next((v[3] for v in (registrar.get('vcardArray') or [None, []])[1] if v[0] == 'fn'), '')
return {**row, 'result': 'registered', 'registrar': name,
'created': created.date() if created else '', 'expires': expires.date() if expires else '',
'age_years': round((NOW - created).days / 365.25, 1) if created else '',
'days_left': (expires - NOW).days if expires else '',
'status': ' '.join(j.get('status', [])),
'nameservers': ' '.join(sorted(n.get('ldhName', '').lower() for n in j.get('nameservers', [])))}
if __name__ == '__main__':
domains = list(dict.fromkeys(filter(None, map(normalize, open(sys.argv[1])))))
servers = load_bootstrap()
t = time.time()
with ThreadPoolExecutor(8) as pool:
rows = list(pool.map(lambda d: lookup(d, servers), domains))
cols = ['domain', 'result', 'registrar', 'created', 'expires', 'age_years', 'days_left', 'status', 'nameservers']
with open('whois_report.csv', 'w', newline='') as f:
w = csv.DictWriter(f, cols); w.writeheader(); w.writerows(rows)
for r in rows:
print(f"{r['result']:<11} {r['domain']:<26} {str(r.get('registrar',''))[:28]:<28} "
f"{str(r.get('created','')):<11} {str(r.get('expires','')):<11} {r.get('days_left','')}")
print(f'{len(rows)} domains in {time.time()-t:.1f} s -> whois_report.csv')
Put one domain per line in domains.txt. The normalize() step means you can paste URLs, www. hosts, email addresses or mailto: links and they still work, and duplicates are dropped.
What a real run returns
I ran it on 13 inputs (a mix of bare domains, a URL, a mailto: address, a duplicate, and one name I made up):
registered github.com MarkMonitor Inc. 2007-10-09 2028-10-09 716
registered python.org Gandi SAS 1995-03-27 2033-03-28 2346
registered wikipedia.org MarkMonitor Inc. 2001-01-13 2027-01-13 80
registered amazon.com MarkMonitor Inc. 1994-11-01 2027-10-31 371
registered bbc.co.uk British Broadcasting Corpora 1994-12-13 2034-12-13 2971
registered apify.com Amazon Registrar, Inc. 2009-06-02 2035-06-02 3142
registered crawlee.dev GoDaddy.com, LLC 2022-07-08 2028-07-08 622
registered google.io MarkMonitor Inc. 2002-10-01 2027-09-30 340
no-rdap notion.so
registered vercel.com Amazon Registrar, Inc. 1999-10-04 2029-10-04 1076
registered openai.com MarkMonitor Inc. 2007-01-19 2029-01-19 818
available datawright-unregistered-test-8812.com
registered stackoverflow.com CSC Corporate Domains, Inc. 2003-12-26 2027-02-02 100
13 domains in 0.5 s -> whois_report.csv
The CSV also has age_years, the EPP status codes (for example client transfer prohibited, which means the domain is locked against transfers) and the full nameserver list, which tells you who hosts the DNS. github.com, for instance, splits its nameservers between NS1 and Amazon Route 53.
A few things stand out. Big brands cluster at MarkMonitor and CSC, which are corporate registrars, so a registrar column is a decent signal for "is this a large company". wikipedia.org and stackoverflow.com were both within about three months of expiry when I ran this; that's normal for domains on auto-renew, but it's exactly the window to watch if you manage domains for clients. And the made-up name came back as a clean 404, so the same script doubles as a bulk availability checker.
Gotchas I hit
-
A registry rate-limited me for almost five hours. After a handful of test runs, Nominet (the
.ukregistry) answered with HTTP 429 andRetry-After: 16949. My first version politely slept onRetry-After, so the whole script just hung. Cap the wait (the script uses 10 seconds) and report the row asrate-limitedinstead. Also keep concurrency per registry low; the thread pool here is fine for a few hundred domains, but for thousands, group by RDAP host and throttle each one. -
The IANA bootstrap file is missing some big ccTLDs.
.io,.co,.meand.usall run RDAP but aren't in the file, so the script adds them by hand inEXTRA. Others, like.so,.jp,.euand.es, have no public RDAP at all, so they come back asno-rdap. For those you're back to port-43 WHOIS or the registry's website. -
Some ccTLDs leave fields out.
.dereturns status and nameservers but no dates, and.chhas no expiry date. Don't treat a blank date as an error. -
Send a real User-Agent. Some RDAP servers reject Python's default
urllibagent, so the script sets one explicitly. -
The domain normalizer is naive. It keeps
shop.example.comas-is instead of reducing it toexample.com, and the registry will return 404 for a subdomain, which looks like "available". For real input, reduce each host to its registrable domain with the Public Suffix List (thetldextractpackage does this). - Registrant contacts are gone. Since GDPR, almost every registry redacts the owner's name and email in RDAP, same as in WHOIS. You get the registrar and its abuse contact, not the person.
Running it on a schedule or at scale
The script is enough for a weekly cron job over your own domains. If you'd rather not handle the rate limits, the extra registry servers and the subdomain logic yourself, I also published this as a cloud tool: WHOIS Domain Lookup (RDAP) on Apify (disclosure: I built it). It reduces every input to its registrable domain with the Public Suffix List, throttles per registry, adds the registrar's IANA ID and abuse contact, DNSSEC, a DNS-provider guess from the nameservers, and filters like "only expiring soon" or "only available". It costs $1 per 1,000 domains looked up; failed lookups and TLDs without RDAP are free.
Either way, the old WHOIS scraping era is over. RDAP gives you the same data as clean JSON, and a few dozen lines of Python are enough to use it.
Top comments (2)
Keeping rate-limited rows distinct from “available” is an important choice here. One regression case I would add is several domains on the same registry receiving 429 together: a shared per-host cooldown would stop the worker pool from retrying them independently, while unrelated registries could still finish. The CSV should preserve that uncertainty rather than turn an incomplete lookup into a business decision.
The Retry-After cap is the detail I would steal first. I have run into the same trap with a batch job that honored a long Retry-After literally and sat idle while the rest of the queue waited behind it. Marking the row as rate-limited, moving on, and letting a later pass retry it keeps one slow registry from holding the whole run hostage.
I also like that you keep rate-limited rows separate from available ones, because a 429 is not an answer about the domain at all, it is an answer about your own pace. One question: do you cache the IANA bootstrap file between runs, or re-download it every time, and how do you handle a registry that changes its RDAP base URL mid-week?