DEV Community: Saad | Data APIs

WHOIS is dead: check domain expiry dates in bulk with Python and RDAP (free, no API key)

Saad | Data APIs — Sun, 11 Oct 2026 09:42:36 +0000

Since 28 January 2025, ICANN no longer requires gTLD registries and registrars to run the old port-43 WHOIS service. RDAP (Registration Data Access Protocol, RFC 9082/9083) is now the official source of domain registration data. If you still parse WHOIS text with regexes, here's the replacement: free JSON over HTTPS, no API key, no scraping.

This post shows how to:

Look up a domain's expiration date with RDAP in 6 lines of Python
Find the right RDAP server for any TLD (the IANA bootstrap file)
Check hundreds of domains in bulk and get a CSV sorted by days until expiry
Handle the edge cases: not-found domains, ccTLDs without RDAP, rate limits

Everything uses the Python standard library only.

WHOIS vs RDAP in one table

	WHOIS (port 43)	RDAP
Format	Free text, different per registrar	JSON (RFC 9083)
Transport	Raw TCP	HTTPS
Parsing	Regex per registry	`json.load()`
Server discovery	Hard-coded lists	IANA bootstrap file
Status for gTLDs	No longer required (since Jan 2025)	Mandatory

RDAP domain lookup in Python (expiration date)

For .com and .net, Verisign's RDAP server answers directly:

import json, urllib.request

req = urllib.request.Request("https://rdap.verisign.com/com/v1/domain/github.com",
                             headers={"Accept": "application/rdap+json"})
data = json.load(urllib.request.urlopen(req, timeout=20))
for e in data["events"]:
    print(e["eventAction"], e["eventDate"])

Output (real run, today):

registration 2007-10-09T18:20:50Z
expiration 2028-10-09T18:20:50Z
last changed 2026-09-07T09:22:52Z
last update of RDAP database 2026-10-11T09:41:35Z

The dates live in the events array. status holds the EPP codes (client transfer prohibited...), nameservers the NS records, and the registrar sits in entities with the role registrar.

How to find the RDAP server for any TLD

There is no single RDAP server. Each registry runs its own, and IANA publishes the map at https://data.iana.org/rdap/dns.json:

{"services": [ [["com", "net"], ["https://rdap.verisign.com/com/v1/"]],
               [["org"], ["https://rdap.publicinterestregistry.org/rdap/"]], ... ]}

RFC 9224 says to match the longest label suffix (so co.uk wins over uk if both exist). Cache this file: it changes rarely, and registries don't like clients that download it before every query.

Bulk domain expiry checker (full script)

Save as rdap_expiry.py, put one domain per line in domains.txt, run python rdap_expiry.py domains.txt > report.csv:

"""Bulk domain expiry checker using RDAP (the official WHOIS replacement).
Standard library only. Usage: python rdap_expiry.py domains.txt > report.csv"""
import csv, json, sys, time, urllib.request, urllib.error
from concurrent.futures import ThreadPoolExecutor
from datetime import datetime, timezone

BOOTSTRAP_URL = "https://data.iana.org/rdap/dns.json"
UA = {"User-Agent": "rdap-expiry-checker/1.0", "Accept": "application/rdap+json"}

def get_json(url, timeout=20, retries=3):
    for attempt in range(retries + 1):
        req = urllib.request.Request(url, headers=UA)
        try:
            with urllib.request.urlopen(req, timeout=timeout) as r:
                return json.load(r)
        except urllib.error.HTTPError as e:
            if e.code == 404:
                return None                      # registry says: no such domain
            if e.code in (429, 500, 502, 503) and attempt < retries:
                wait = e.headers.get("Retry-After")
                time.sleep(int(wait) if wait and wait.isdigit() else 2 ** attempt)
                continue
            raise

def load_bootstrap():
    """Map every TLD to its registry's RDAP base URL (cache this; IANA updates it rarely)."""
    data = get_json(BOOTSTRAP_URL)
    servers = {}
    for tlds, urls in data["services"]:
        for tld in tlds:
            servers[tld.lower()] = urls[0].rstrip("/") + "/"
    return servers

def rdap_base(domain, servers):
    """Label-wise longest match, as RFC 9224 says (e.g. 'co.uk' before 'uk')."""
    labels = domain.split(".")
    for i in range(1, len(labels)):
        suffix = ".".join(labels[i:])
        if suffix in servers:
            return servers[suffix]
    return None

def parse_date(s):
    return datetime.fromisoformat(s.replace("Z", "+00:00")) if s else None

def registrar_name(data):
    for ent in data.get("entities", []):
        if "registrar" in ent.get("roles", []):
            for item in ent.get("vcardArray", [None, []])[1]:
                if item[0] == "fn":
                    return item[3]
    return None

def lookup(domain, servers):
    domain = domain.strip().lower().rstrip(".")
    row = {"domain": domain, "status": "", "registered": "", "expires": "",
           "days_left": "", "registrar": "", "error": ""}
    base = rdap_base(domain, servers)
    if not base:
        row["error"] = "TLD has no RDAP server in the IANA bootstrap (use port-43 WHOIS)"
        return row
    try:
        data = get_json(base + "domain/" + domain)
    except Exception as e:
        row["error"] = f"{type(e).__name__}: {e}"
        return row
    if data is None:
        row["status"] = "not found (likely available)"
        return row
    events = {e.get("eventAction"): e.get("eventDate") for e in data.get("events", [])}
    exp = parse_date(events.get("expiration"))
    row.update(status=", ".join(data.get("status", [])),
               registered=events.get("registration", ""),
               expires=events.get("expiration", ""),
               days_left=(exp - datetime.now(timezone.utc)).days if exp else "",
               registrar=registrar_name(data) or "")
    return row

if __name__ == "__main__":
    src = open(sys.argv[1]) if len(sys.argv) > 1 else sys.stdin
    domains = sorted({d.strip() for d in src if d.strip()})
    servers = load_bootstrap()
    with ThreadPoolExecutor(max_workers=8) as pool:
        rows = list(pool.map(lambda d: lookup(d, servers), domains))
    rows.sort(key=lambda r: r["days_left"] if isinstance(r["days_left"], int) else 10**9)
    w = csv.DictWriter(sys.stdout, fieldnames=list(rows[0]))
    w.writeheader(); w.writerows(rows)

Real output for a mixed list (.com, .org, .co.uk, .ai, .to, .de, and one made-up name), in under a second:

domain,status,registered,expires,days_left,registrar,error
wikipedia.org,"client delete prohibited, ...",2001-01-13T00:12:14.754Z,2027-01-13T00:12:14Z,93,MarkMonitor Inc.,
dev.to,active,2016-04-05T02:27:16.000Z,2027-04-05T02:27:16.000Z,175,Tonic,
github.com,"client delete prohibited, ...",2007-10-09T18:20:50Z,2028-10-09T18:20:50Z,729,MarkMonitor Inc.,
openai.ai,"client transfer prohibited, inactive",2017-12-16T02:53:06Z,2029-11-03T02:53:06Z,1118,Zenaida.cate.ai,
python.org,client transfer prohibited,1995-03-27T05:00:00.545Z,2033-03-28T05:00:00Z,2359,Gandi SAS,
bbc.co.uk,"server delete prohibited, ...",1994-12-13T03:49:48Z,2034-12-13T03:49:48Z,2984,British Broadcasting Corporation,
apify.com,client transfer prohibited,2009-06-02T17:14:10Z,2035-06-02T17:14:10Z,3156,"Amazon Registrar, Inc.",
example.de,,,,,,TLD has no RDAP server in the IANA bootstrap (use port-43 WHOIS)
this-domain-should-not-exist-8812.com,not found (likely available),,,,,

Rows are sorted so the domains closest to expiry come first. Put it in a cron job and alert when days_left < 30.

Gotchas I hit

404 means "not registered" (usually). The registry returns HTTP 404 for names it doesn't hold. That's a strong hint the domain is available, not a guarantee (reserved and premium names exist).
Some ccTLDs have no RDAP yet. .de is the classic example: it's not in the bootstrap file, so you still need port-43 WHOIS or the registry's own service for those.
Thin registries. For .com/.net, Verisign only stores the basics. The registrant organization (when it's not redacted) is on the registrar's RDAP server, linked from the response's links with rel: "related". One more request per domain if you need it.
Rate limits. Registries throttle aggressive clients with 429. Keep concurrency per server low (the script uses 8 threads total) and honour Retry-After.
Personal data is redacted. Since GDPR most registrant contact fields are hidden. Don't try to work around it; for monitoring you only need dates, status, registrar and nameservers anyway.
Mixed date formats. Some registries add milliseconds (.754Z). datetime.fromisoformat in Python 3.11+ handles both once you swap Z for +00:00.

When the script isn't enough

The script above is fine for a few hundred domains. If you need thousands, messy input (URLs and emails, IDNs, www. prefixes), per-registry rate limiting, the registrar hop for .com, and DNS/MX/SPF/DMARC in the same row, I packaged all of that as an Apify Actor: Bulk WHOIS Lookup API (RDAP). It costs $1 per 1,000 domains and failed lookups are free. Disclosure: I built it. The free script above does the core job, so use whichever fits.

Questions about a TLD that behaves oddly? Drop it in the comments and I'll check what its RDAP server returns.

How to get every Greenhouse, Lever and Ashby job as JSON (and search 400k jobs in one call)

Saad | Data APIs — Sun, 11 Oct 2026 08:40:39 +0000

If you are building a job board, a job-alert bot or a hiring-signals tool, you quickly learn a nice secret: most tech companies don't host their own jobs. They use an applicant tracking system (ATS) such as Greenhouse, Lever, Ashby, Workable, Recruitee or SmartRecruiters, and almost all of these publish a free, public job-board API.

In this post I'll show the raw APIs first (free, no key), then the shortcut I built when I got tired of maintaining six parsers. Disclosure: I built the Apify Actors linked at the end.

1. The free public ATS APIs

ATS	Public endpoint (no auth)	Where to find the slug
Greenhouse	`https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true`	`boards.greenhouse.io/{slug}`
Lever	`https://api.lever.co/v0/postings/{slug}?mode=json`	`jobs.lever.co/{slug}`
Ashby	`https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true`	`jobs.ashbyhq.com/{slug}`
Recruitee	`https://{slug}.recruitee.com/api/offers/`	`{slug}.recruitee.com`
SmartRecruiters	`https://api.smartrecruiters.com/v1/companies/{id}/postings`	`jobs.smartrecruiters.com/{id}`

Quick Python example for one Greenhouse board:

import requests

slug = "airbnb"
r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", params={"content": "true"}, timeout=30)
for job in r.json()["jobs"]:
    print(job["title"], "|", job["location"]["name"], "|", job["absolute_url"])

That's great for one company. The problems start at scale:

Every ATS has its own JSON shape. Lever has categories.location, Ashby has compensation, Greenhouse has no salary field at all, dates come in three formats.
You need the slugs. The APIs are per company. To answer "show me remote data-engineer jobs posted this week" you first need a list of thousands of boards.
Boards die and move. Companies switch ATS, slugs change, boards go empty.

2. Normalizing several ATSs into one schema

If you already have a company list, the ATS Jobs API Actor takes career-page URLs or slugs, auto-detects the ATS and returns one row per job with the same fields: title, location, country, isRemote, salaryMin/Max/Currency, postedAt, applyUrl, descriptionText.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_TOKEN")
run = client.actor("saadhazz/ats-jobs-api").call(run_input={
    "companies": ["https://boards.greenhouse.io/airbnb", "lever:palantir", "https://jobs.ashbyhq.com/ramp"],
    "maxJobsPerCompany": 50,
})
for job in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(job["ats"], job["companySlug"], job["title"], job.get("salaryMin"))

3. Searching all companies at once by title, country and remote

The bigger problem was the slug list. So I built an index: every day it re-reads about 13,000 live company boards on Greenhouse, Lever, Ashby, Workable, Recruitee, Personio and SmartRecruiters (around 400,000 open jobs). The Job Search API: Career Site Jobs Actor searches that index and then re-checks the matching boards live, so closed jobs are dropped.

curl -X POST "https://api.apify.com/v2/acts/saadhazz~career-site-job-search/run-sync-get-dataset-items?token=$APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"keywords": ["data engineer"], "remoteOnly": true, "postedWithinDays": 3, "maxJobs": 100}'

One returned row:

{
  "title": "Software Engineering Manager (Mission Ops)",
  "companyName": "Sift",
  "ats": "ashby",
  "location": "Marina Del Rey, CA",
  "country": "US",
  "isRemote": false,
  "salaryMin": 200000, "salaryMax": 265000, "salaryCurrency": "USD", "salaryPeriod": "YEAR",
  "postedAt": "2026-10-08T20:22:47Z",
  "applyUrl": "https://jobs.ashbyhq.com/siftstack/33454cb1-.../application",
  "dataSource": "live"
}

Turning it into a daily job alert

Set "onlyNewSinceLastRun": true and an alertId, then create a daily Schedule in Apify. Each run returns only jobs it hasn't returned before, which you can push to Slack, a Google Sheet or your own database via webhook.

4. What it costs

Raw ATS APIs: free, plus your time to build and maintain the parsers and the board list.
ATS Jobs API: $1 per 1,000 jobs.
Career-site job search: $2 per 1,000 jobs. A new Apify account's free monthly credit covers a few thousand jobs to try it.

5. Notes on doing this responsibly

These are the APIs ATS vendors publish so jobs get syndicated, but keep the original jobUrl/applyUrl when you republish.
Don't collect recruiter or candidate personal data. Job postings only.
Cache, and don't hammer one board. One request per board per day is plenty.

If you build something with this, I'd love to hear about it in the comments. And if you know an ATS with a public API that I should add, tell me.