DEV Community

Andy Zeng
Andy Zeng

Posted on

How to bulk-check domain expiry dates and availability with RDAP

#python #webdev #devops #tutorial

Disclosure: I build and sell the paid Apify Actor mentioned near the end. Everything before that section uses free, public services. This article was drafted with AI assistance; the code and examples were tested.

Sooner or later, most people who own more than a handful of domains need the same answers: Which of our domains expire soon? Which of these name ideas are actually free? Is that competitor's domain about to lapse? The traditional answer was "run whois in a loop and parse the output", which breaks constantly because every registry formats WHOIS text differently.

There's a better way. RDAP (Registration Data Access Protocol) is the ICANN-mandated successor to WHOIS, and it returns structured JSON. This tutorial shows how to:

  1. Find the right RDAP server for any TLD.
  2. Look up creation date, expiry date and status for a list of domains.
  3. Tell "available" apart from "not in the registry, but you still can't have it".
  4. Add DNS checks (website? email? SPF/DMARC?) for context.

Step 1: Find the RDAP server for each TLD

IANA publishes a bootstrap file that maps every TLD to its RDAP base URL:

curl -s https://data.iana.org/rdap/dns.json | python3 -m json.tool | head -40
Enter fullscreen mode Exit fullscreen mode

For .com it points to https://rdap.verisign.com/com/v1/. A lookup is just a GET:

curl -s https://rdap.verisign.com/com/v1/domain/stackoverflow.com | python3 -m json.tool
Enter fullscreen mode Exit fullscreen mode

The parts you care about:

  • events: an array with eventAction values such as registration, expiration and last changed
  • status: EPP statuses like client transfer prohibited or pending delete
  • entities: the registrar (with its IANA ID) and sometimes registrant info, which is often redacted
  • nameservers and secureDNS

For "thin" registries like .com, the registry response has the core dates. The registrar's own RDAP server (linked in the links array) sometimes has more detail, such as abuse contacts.

A 404 from the registry RDAP server means the registry has no record of that domain. That's the start of an availability check, not the end of one (see Step 3).

Step 2: A bulk lookup script

This tested script (needs pip install httpx) loads the bootstrap file once, looks up each domain, computes days until expiry, and prints CSV:

import csv, sys, time
from datetime import datetime, timezone
import httpx

UA = {"User-Agent": "domain-expiry-script (contact: you@example.com)"}
client = httpx.Client(headers=UA, timeout=20, follow_redirects=True)

bootstrap = client.get("https://data.iana.org/rdap/dns.json").json()
RDAP = {tld: svc[1][0] for svc in bootstrap["services"] for tld in svc[0]}

def lookup(domain):
    tld = domain.rsplit(".", 1)[-1]
    base = RDAP.get(tld)
    if not base:
        return {"domain": domain, "availability": "unknown", "note": "no RDAP for TLD"}
    r = client.get(f"{base.rstrip('/')}/domain/{domain}")
    if r.status_code == 404:
        return {"domain": domain, "availability": "available?"}
    if r.status_code == 429:
        time.sleep(5)
        return lookup(domain)
    r.raise_for_status()
    data = r.json()
    events = {e["eventAction"]: e["eventDate"] for e in data.get("events", [])}
    expiry = events.get("expiration")
    days_left = None
    if expiry:
        exp = datetime.fromisoformat(expiry.replace("Z", "+00:00"))
        days_left = (exp - datetime.now(timezone.utc)).days
    statuses = data.get("status", [])
    return {
        "domain": domain,
        "availability": "pending-delete" if "pending delete" in statuses else "registered",
        "created": events.get("registration"),
        "expires": expiry,
        "days_until_expiry": days_left,
        "status": "; ".join(statuses),
    }

rows = [lookup(d.strip().lower()) for d in sys.argv[1:]]
w = csv.DictWriter(sys.stdout, fieldnames=["domain", "availability", "created", "expires",
                                           "days_until_expiry", "status", "note"])
w.writeheader(); w.writerows(rows)
Enter fullscreen mode Exit fullscreen mode 
python rdap_check.py stackoverflow.com example.org my-new-startup-idea-zzq.com apify.com > domains.csv
Enter fullscreen mode Exit fullscreen mode

Output from a run I did today (dates come straight from the registries):

domain,availability,created,expires,days_until_expiry,status,note
stackoverflow.com,registered,2003-12-26T19:18:07Z,2027-02-02T11:59:59Z,98,client transfer prohibited,
example.org,registered,1995-08-31T04:00:00Z,2027-08-30T04:00:00Z,307,server delete prohibited; server transfer prohibited; server update prohibited,
my-new-startup-idea-zzq.com,available?,,,,,
apify.com,registered,2009-06-02T17:14:10Z,2035-06-02T17:14:10Z,3141,client transfer prohibited,
Enter fullscreen mode Exit fullscreen mode

Things to add before you run it on thousands of domains:

  • Rate limits are per server. Registries throttle aggressively and answer with HTTP 429. Group domains by RDAP server, keep a small concurrency per server, and back off on 429. The naive retry in the script is fine for a short list, not for a portfolio.
  • Input cleanup. Real lists contain URLs, emails and subdomains. Extract the registrable domain with the Public Suffix List (for example the tldextract package), so shop.example.co.uk becomes example.co.uk, not co.uk.
  • Not every TLD has RDAP. Some ccTLDs still only offer port-43 WHOIS. For those you need a WHOIS fallback with per-registry parsing, or at least a DNS check (Step 4) to infer whether the name is delegated.

Step 3: What "available" really means

An RDAP 404 means the registry has no record. You still might not be able to register the name:

  • Premium and reserved names can be missing from RDAP and still be unbuyable, or priced far above normal.
  • Pending delete domains (status pending delete) are about to drop. They're registered right now, but they'll become available within days, which is why domain investors watch for them.
  • Redemption period / auto-renew grace: for many gTLDs, the registry auto-renews at expiry and the registrar decides later whether to delete. So an expiry date that just jumped forward a year doesn't prove the owner actually renewed.

Always confirm at a registrar before you plan around a name.

Step 4: Add DNS for context

Registration data tells you who holds a domain. DNS tells you whether it's used. DNS-over-HTTPS makes this easy without installing anything:

# Is it delegated at all? (NXDOMAIN = Status 3)
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=example.com&type=NS'

# Does it receive email?
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=example.com&type=MX'

# Email authentication policy
curl -s -H 'accept: application/dns-json' 'https://cloudflare-dns.com/dns-query?name=_dmarc.example.com&type=TXT'
Enter fullscreen mode Exit fullscreen mode

Useful combinations:

  • Registered, no NS/A records: parked or abandoned, sometimes a candidate for a purchase offer.
  • Registered, has MX but no DMARC: a spoofing risk if it's one of your domains.
  • Your own domain expiring in under 30 days with auto-renew off: fix it today.

Step 5: Turn it into a monitor

For a portfolio you own, run the check weekly and alert on:

  • days_until_expiry below a threshold (30 is a common default)
  • a status change, for example losing client transfer prohibited (transfer lock), which is worth knowing about immediately
  • nameserver changes you didn't make

A cron job that writes the CSV and diffs it against last week's copy is enough for most teams.

Step 6: Automating it (optional)

If you don't want to maintain the bootstrap handling, per-server rate limits, WHOIS fallback and DNS checks yourself, I packaged this as an Apify Actor: Bulk WHOIS & Domain Availability Checker. It's paid, so here's exactly what it does:

  • takes domains, URLs or email addresses, and extracts and de-duplicates the registrable domain
  • queries RDAP via the IANA bootstrap and follows the registrar's RDAP server for extra detail, with a port-43 WHOIS fallback for ccTLDs without RDAP
  • returns availability (registered / available / pending-delete / unknown), registrar, created/expiry dates, domainAgeYears, daysUntilExpiry, an expiringSoon flag, EPP statuses, nameservers and DNSSEC
  • optionally adds NS/A/MX, SPF and DMARC, plus hasWebsite / hasEmail flags
  • writes a SUMMARY record and an AVAILABLE CSV. Redacted registrant data stays empty and nothing is guessed.

Pricing from the README: $2 per 1,000 domains checked, plus $0.005 per run, with DNS included. Its "available" has the same caveat as above: no registry record and no DNS delegation, so premium or reserved names can still be unbuyable.

If you're vetting an expired domain before buying it, pair this with a look at its history. I covered that in How to find and fix lost pages after a website migration using the Wayback Machine CDX API.

Checklist

  • [ ] Use RDAP (JSON) via the IANA bootstrap instead of parsing WHOIS text
  • [ ] Clean input to registrable domains with the Public Suffix List
  • [ ] Respect per-server rate limits, and back off on 429
  • [ ] Treat "available" as "confirm at a registrar"
  • [ ] Add DNS (NS/MX/DMARC) for context
  • [ ] Schedule it and alert on expiry, status and nameserver changes

Top comments (0)