DEV Community: Jackson Kramer

Bulk Domain Expiry & WHOIS Lookup in Python with RDAP (and the .io Trap)

Jackson Kramer — Sat, 31 Oct 2026 17:18:15 +0000

WHOIS is on its way out. The registries now publish the same data through RDAP, a JSON API that's free, needs no key, and returns registrar, creation and expiry dates, and status in a consistent format. If you track client domains, buy expiring names, or audit a portfolio, a few lines of Python replace a paid WHOIS API.

The script

rdap.org is a public bootstrap service: request rdap.org/domain/<name> and it redirects you to the right registry. Standard library only:

"""Bulk domain lookup with RDAP (the official successor to WHOIS): registrar, created, expires, status.
Usage: python3 domain_expiry.py apify.com python.org some-free-name-123.com"""
import json
import socket
import sys
import urllib.error
import urllib.request
from concurrent.futures import ThreadPoolExecutor


def rdap(domain):
    url = f"https://rdap.org/domain/{domain}"  # redirects to the right registry's RDAP server
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers={"Accept": "application/rdap+json", "User-Agent": "domain-expiry-script"}), timeout=20) as r:
            data = json.load(r)
    except urllib.error.HTTPError as e:
        if e.code != 404:
            return {"domain": domain, "status": f"HTTP {e.code}"}
        # 404 means "no RDAP record", which is NOT proof it's free: some registries (.io, .co…) have no RDAP at all
        try:
            socket.getaddrinfo(domain, None)
            return {"domain": domain, "status": "registered (no RDAP)"}
        except socket.gaierror:
            return {"domain": domain, "status": "maybe free"}
    events = {e["eventAction"]: e["eventDate"][:10] for e in data.get("events", [])}
    registrar = next((v[3] for ent in data.get("entities", []) if "registrar" in ent.get("roles", [])
                      for v in ent.get("vcardArray", [None, []])[1] if v[0] == "fn"), "")
    return {"domain": domain, "status": "registered", "registrar": registrar,
            "created": events.get("registration", ""), "expires": events.get("expiration", "")}


with ThreadPoolExecutor(5) as pool:
    for row in pool.map(rdap, sys.argv[1:]):
        print(f"{row['domain']:28} {row['status']:20} {row.get('expires', ''):11} {row.get('registrar', '')}")

Real run (October 2026):

$ python3 domain_expiry.py apify.com python.org openalex.org github.io zzqxv-free-name-12345.com
apify.com                    registered           2035-06-02  Amazon Registrar, Inc.
python.org                   registered           2033-03-28  Gandi SAS
openalex.org                 registered           2028-03-14  GoDaddy.com, LLC
github.io                    registered (no RDAP)
zzqxv-free-name-12345.com    maybe free

Three traps I hit writing this

The .io trap. My first version printed github.io → available?, which is obviously wrong. Some registries (.io, .co, and others) don't run RDAP at all, so a 404 means "no RDAP record", not "free". The fix: on 404, check DNS. If the name resolves, it's registered.
403s from .org. The .org registry rejected requests with no User-Agent. Always send one.
"Maybe free" really means maybe. No RDAP record and no DNS still isn't proof: names can be reserved, premium, or registered with no DNS. Confirm at a registrar before you get excited.

Also: RDAP servers rate-limit. Keep concurrency low (5 here) for big lists.

Thousands of domains, scheduled

For portfolio checks or weekly expiry monitoring, I put a fuller version on Apify: WHOIS Domain Lookup. It covers RDAP across 1,200+ endings, DNS records (A, MX, NS, TXT), deduplication, and graceful handling of registries without RDAP. It costs $2 per 1,000 domains, and endings without RDAP are DNS-only and free. It never returns registrant contact details.

Either way, the script above is yours.

More free scripts and tools from Kramer Labs.

Turn Any Docs Site Into Clean Markdown for RAG (Python, No Dependencies)

Jackson Kramer — Sat, 31 Oct 2026 17:17:17 +0000

If you're building a RAG chatbot or giving an AI agent "the docs", the first problem is boring: web pages are 80% navigation, footers, cookie banners, and script tags. Feed that raw HTML to an embedding model and your chunks fill up with "Skip to main content" and menu links.

What you want is the main content as Markdown: headings kept (they make great chunk boundaries), code blocks fenced, lists intact, and everything else gone.

The script

Standard library only. It keeps <main> (or role="main"), drops nav/header/footer/aside/script and ARIA navigation landmarks, and converts headings, paragraphs, lists, inline code, and <pre> blocks:

"""Turn a docs page into clean Markdown for RAG, with the standard library only.
Usage: python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md"""
import re
import sys
import urllib.request
from html.parser import HTMLParser

SKIP = {"script", "style", "nav", "header", "footer", "aside", "form", "noscript", "svg"}
HEADINGS = {"h1": "# ", "h2": "## ", "h3": "### ", "h4": "#### "}


class ToMarkdown(HTMLParser):
    def __init__(self):
        super().__init__()
        self.out, self.skip, self.pre = [], 0, False

    def handle_starttag(self, tag, attrs):
        role = dict(attrs).get("role", "")
        if tag in SKIP or role in ("navigation", "banner", "contentinfo"):
            self.skip += 1
        elif self.skip:
            return
        elif tag in HEADINGS:
            self.out.append("\n\n" + HEADINGS[tag])
        elif tag in ("p", "div", "section"):
            self.out.append("\n\n")
        elif tag == "li":
            self.out.append("\n- ")
        elif tag == "pre":
            self.pre = True
            self.out.append("\n\n```

\n")
        elif tag == "code" and not self.pre:
            self.out.append("`")

    def handle_endtag(self, tag):
        if tag in SKIP:
            self.skip = max(0, self.skip - 1)
        elif self.skip:
            return
        elif tag == "pre":
            self.pre = False
            self.out.append("\n```\n")
        elif tag == "code" and not self.pre:
            self.out.append("`")

    def handle_data(self, data):
        if not self.skip:
            self.out.append(data if self.pre else re.sub(r"\s+", " ", data))


url = sys.argv[1]
req = urllib.request.Request(url, headers={"User-Agent": "docs-to-markdown"})
html = urllib.request.urlopen(req, timeout=30).read().decode("utf-8", "replace")
main = re.search(r"<main\b.*?</main>|<div[^>]+role=\"main\".*", html, re.S | re.I)  # prefer the main content
parser = ToMarkdown()
parser.feed(main.group(0) if main else html)
md = re.sub(r"\n{3,}", "\n\n", "".join(parser.out).replace("¶", "")).strip()
print(md)
print(f"\n<!-- {len(md.split())} words from {url} -->", file=sys.stderr)
``{% endraw %}{% raw %}`

Real run (October 2026):

```shell
$ python3 docs_to_markdown.py https://docs.python.org/3/library/json.html > json.md
<!-- 3810 words from https://docs.python.org/3/library/json.html -->
``{% endraw %}{% raw %}`

The output starts like this, with no menus and no breadcrumbs:

```python
# `json` — JSON encoder and decoder

Source code: Lib/json/__init__.py

JSON (JavaScript Object Notation), specified by RFC 7159 ...
``{% endraw %}{% raw %}`

## Why Markdown and not plain text?

- **Headings become chunk boundaries.** Split on `##` and each chunk is about one topic, which retrieves far better than fixed 500-token windows.
- **Code stays fenced.** Your model can tell code from prose.
- **It's compact.** Markdown is much smaller than the HTML it came from, so you spend fewer tokens per page.

## What this script doesn't do (and what bites at scale)

- **Tables and links.** It flattens them. For docs heavy on parameter tables, convert `<table>` to pipe tables and keep `[text](url)` links.
- **Crawling.** One page is easy. A whole docs site needs the sitemap, link-following that stays in scope (only `/docs/v2/`, not the blog), and deduplication.
- **robots.txt.** Be a good citizen and check it before crawling. Also, Python's `urllib.robotparser` mishandles `*` wildcards and rewrites `Disallow: /?` as `Disallow: /`, so test it against real files.
- **JavaScript-rendered docs.** Plain HTTP sees an empty shell. Those need a headless browser.

## The no-code version

I packaged a fuller version (tables, links, nested lists, sitemap plus in-scope crawling, robots.txt, about 2 seconds for 8 pages) as [Website to Markdown on Apify](https://apify.com/kramer_labs/website-to-markdown). Give it a start URL like `https://docs.python.org/3/library/` and get one Markdown record per page, ready for LangChain, LlamaIndex, or your vector store. It costs $2 per 1,000 pages, and failed pages and pages robots.txt disallows are free.

Either way, the script above is yours to copy.

*More free scripts and tools from [Kramer Labs](https://kramer-labs.netlify.app/).*

A Free Google Scholar Alternative API: Search 250M Papers With OpenAlex in Python

Jackson Kramer — Sat, 31 Oct 2026 05:26:58 +0000

Google Scholar has no API, and its terms don't allow scraping. If you need research papers in a script (for a literature review, a reading list, or a RAG pipeline), use OpenAlex instead. It's a free, open catalog of over 250 million scholarly works (the successor to Microsoft Academic), and its data is CC0, so you can reuse it however you like.

The script

Search titles and abstracts, keep papers from 2015 on, and sort by citations:

"""Search scholarly papers with the free OpenAlex API. Usage: python3 paper_search.py '"growth mindset"'"""
import json
import sys
import urllib.parse
import urllib.request

query = sys.argv[1]
params = {
    "filter": f"title_and_abstract.search:{query},publication_year:2015-",
    "sort": "cited_by_count:desc",
    "per-page": 10,
    # "api_key": "YOUR_KEY",  # free at openalex.org/settings/api ($1/day of usage)
}
url = "https://api.openalex.org/works?" + urllib.parse.urlencode(params)
data = json.load(urllib.request.urlopen(url, timeout=30))

print(f"{data['meta']['count']:,} papers match {query}. Most cited since 2015:\n")
for w in data["results"]:
    venue = ((w.get("primary_location") or {}).get("source") or {}).get("display_name", "")
    pdf = (w.get("best_oa_location") or {}).get("pdf_url") or ""
    print(f"{w['cited_by_count']:>6}  {w['publication_year']}  {w['title']}")
    print(f"        {venue}  {pdf}")

Real output (October 2026):

$ python3 paper_search.py '"growth mindset"'
9,092 papers match "growth mindset". Most cited since 2015:

  1694  2019  PISA 2018 Results (Volume I)
        Programme for international student assessment/Internationale Schulleistungsstudie
  1511  2019  A national experiment reveals where a growth mindset improves achievement
        Nature  https://www.nature.com/articles/s41586-019-1466-y.pdf
  1259  2019  Mindsets: A View From Two Eras
        Perspectives on Psychological Science
  1054  2016  Motivation to learn: an overview of contemporary theories
        Medical Education  https://onlinelibrary.wiley.com/doi/pdfdirect/10.1111/medu.13074

Quotes around the phrase matter. Without them, OpenAlex matches the words anywhere, and "most cited" fills up with unrelated famous papers.

Useful details

Abstracts come as an "inverted index" ({"word": [positions]}). Rebuild the text by sorting positions: " ".join(w for _, w in sorted((i, w) for w, idx in inv.items() for i in idx)).
Free PDFs: best_oa_location.pdf_url links to a legal open-access copy when one exists. Add is_oa:true to the filter to keep only those.
Filters chain with commas: type:review, cited_by_count:>100, publication_year:2020-2024.
DOI lookups: filter=doi:10.1038/s41586-019-1466-y|10.1177/1529100612453266 fetches up to 50 at once.
Limits: without a key, you share a small free daily budget with everyone on your IP. A free key (openalex.org/settings/api) gives you $1/day of usage, where a search costs $0.001 for up to 100 results. That's plenty for most projects.

No-code version

If you'd rather get a spreadsheet, I wrapped this as Research Paper Search on Apify. Type queries or paste DOIs, set filters (year, citations, type, open access), and export CSV or Excel with authors, institutions, journal, citations, the full abstract, topic, and PDF link. It costs $2 per 1,000 papers, and no key is needed.

The script is yours either way. Happy reading.

How to Get Every Open Job From a Company's Careers Page (Greenhouse, Lever & Ashby APIs)

Jackson Kramer — Sat, 31 Oct 2026 05:22:47 +0000

Most tech companies don't build their own careers page. They use an applicant tracking system (ATS), and the three big ones (Greenhouse, Lever, and Ashby) each publish a free, public JSON API for every company's job board. They exist so companies can show their jobs on their own sites, so there's no scraping, no login, and no API key.

If you know the company's board name (usually just the company name), one request gets you every open job.

ATS	Public endpoint
Greenhouse	`https://boards-api.greenhouse.io/v1/boards/{company}/jobs`
Lever	`https://api.lever.co/v0/postings/{company}?mode=json`
Ashby	`https://api.ashbyhq.com/posting-api/job-board/{company}`

How do you find the name? Open the company's careers page and click any job. The link will contain boards.greenhouse.io/airbnb, jobs.lever.co/spotify, or jobs.ashbyhq.com/ramp.

The script (standard library only)

It tries all three platforms, keeps the one with the most jobs, and saves a CSV.

"""Every open job at a company, from its public job-board API. Usage: python3 company_jobs.py airbnb"""
import csv
import json
import sys
import urllib.error
import urllib.request

BOARDS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever": "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{}",
}


def get(url):
    try:
        with urllib.request.urlopen(urllib.request.Request(url, headers={"User-Agent": "jobs-script"}), timeout=30) as r:
            return json.load(r)
    except urllib.error.HTTPError:
        return None  # no board with that name


def rows(ats, data):
    if ats == "greenhouse":
        return [(j["title"], j["location"]["name"], j["absolute_url"]) for j in data["jobs"]]
    if ats == "lever":
        return [(j["text"], j["categories"].get("location", ""), j["hostedUrl"]) for j in data]
    return [(j["title"], j.get("location", ""), j["jobUrl"]) for j in data["jobs"]]


def jobs_for(company):
    # A company name can exist on more than one platform (often an old, nearly empty board), so keep the biggest
    found = [(ats, rows(ats, data)) for ats, url in BOARDS.items() if (data := get(url.format(company)))]
    return max(found, key=lambda f: len(f[1]), default=(None, []))


company = sys.argv[1]
ats, jobs = jobs_for(company)
if not ats:
    sys.exit(f"No Greenhouse, Lever or Ashby board found for {company}")
with open(f"{company}-jobs.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow(["title", "location", "url"])
    w.writerows(jobs)
print(f"{company} ({ats}): {len(jobs)} open jobs saved to {company}-jobs.csv")

Real runs (October 2026):

$ python3 company_jobs.py airbnb
airbnb (greenhouse): 201 open jobs saved to airbnb-jobs.csv
$ python3 company_jobs.py spotify
spotify (lever): 89 open jobs saved to spotify-jobs.csv

The gotcha: stale boards

My first version returned the first platform that answered. For Ramp, that was a Greenhouse board with exactly 1 job. Ramp's real board is on Ashby, with 170. Companies switch ATS and leave the old board up, so check every platform and keep the biggest. That's what the max() line does.

Other things to know:

Descriptions are in the full responses: Greenhouse needs ?content=true, Lever has descriptionPlain, and Ashby has descriptionPlain.
Salary shows up on Ashby (?includeCompensation=true) and Lever (salaryRange) when the company publishes it.
Lever EU companies live at api.eu.lever.co.
Be polite: cache results, and don't hammer the endpoints. Job boards change a few times a day at most.

Tracking new jobs (no code)

The script is enough for a one-off export. If you want many companies, more platforms, and only the jobs posted since yesterday, I packaged it as a Company Jobs Scraper on Apify. It covers Greenhouse, Lever, Ashby, Workable, SmartRecruiters, and Recruitee in one format, with title, location, remote, and date filters. It has a "only new jobs since the last run" mode for schedules, and costs $1.40 per 1,000 jobs with no subscription. People use it for job alerts, recruiting research, and hiring signals ("who's hiring ML engineers this week?").

Either way, the script above is yours to copy.

How to Find Broken Links on Any Website (Free Python Script + No-Code Option)

Jackson Kramer — Sat, 31 Oct 2026 05:21:49 +0000

Broken links quietly hurt a site. Readers hit dead ends, and search engines waste crawl time on pages that don't exist. Most broken-link checkers are browser extensions that check one page, or SEO suites that cost $100+ a month.

Here's a free Python script that checks every link on a page, using only the standard library, no installs. After that, a no-code option for checking a whole site.

The script

It does three things:

Downloads the page and pulls out every <a href>.
Checks each link with a HEAD request, falling back to GET because some servers refuse HEAD.
Prints anything that returns 4xx/5xx or doesn't connect at all.

"""Find broken links on one web page. Usage: python3 find_broken_links.py https://example.com"""
import sys
import urllib.error
import urllib.parse
import urllib.request
from concurrent.futures import ThreadPoolExecutor
from html.parser import HTMLParser

UA = {"User-Agent": "Mozilla/5.0 (link checker)"}


class Links(HTMLParser):
    def __init__(self):
        super().__init__()
        self.hrefs = []

    def handle_starttag(self, tag, attrs):
        href = dict(attrs).get("href")
        if tag == "a" and href:
            self.hrefs.append(href)


def status(url):
    for method in ("HEAD", "GET"):  # some servers refuse HEAD
        try:
            req = urllib.request.Request(url, method=method, headers=UA)
            with urllib.request.urlopen(req, timeout=15) as r:
                return r.status
        except urllib.error.HTTPError as e:
            if method == "GET" or e.code not in (403, 405, 501):
                return e.code
        except Exception as e:
            return f"error: {getattr(e, 'reason', e)}"


page = sys.argv[1]
html = urllib.request.urlopen(urllib.request.Request(page, headers=UA), timeout=15).read().decode("utf-8", "replace")
parser = Links()
parser.feed(html)
links = {urllib.parse.urldefrag(urllib.parse.urljoin(page, h))[0] for h in parser.hrefs}
links = sorted(u for u in links if u.startswith("http"))

with ThreadPoolExecutor(10) as pool:
    results = list(zip(links, pool.map(status, links)))

broken = [(u, s) for u, s in results if not isinstance(s, int) or s >= 400]
print(f"Checked {len(links)} links on {page}: {len(broken)} broken")
for u, s in broken:
    print(f"  {s}  {u}")

Run it:

python3 find_broken_links.py https://www.python.org/about/apps/

Real output (October 2026):

Checked 119 links on https://www.python.org/about/apps/: 3 broken
  error: [Errno 61] Connection refused  http://mcsp.wartburg.edu/zelle/python/
  error: [Errno 61] Connection refused  http://pragprog.com/book/gwpy2/practical-programming
  404  http://wiki.python.org/moin/TkInter

Even python.org has a few. Old links rot: domains expire, wikis move, and http:// links stop answering.

Things that will trip you up

False alarms from social sites. LinkedIn, X, and Instagram often answer automated requests with 999, 403, or 520 even when the page works fine in a browser. Treat those as "check by hand", not "broken".
Redirects. urllib follows them silently. A link that redirects to a 404 shows as 404, which is what you want. A long redirect chain is still worth fixing.
JavaScript sites. If a site builds its pages with JavaScript, the raw HTML has no links in it. You'd need a headless browser.
Be polite. Ten threads is fine for one page. For a whole site, limit requests per server and respect robots.txt (urllib.robotparser exists, but it mishandles * wildcards and some ? rules, so test it).

Checking a whole site (no code)

The script checks one page. For a full site you need the list of pages (from the sitemap), deduplication across pages, "which page is this broken link on?", and the false-alarm handling above.

I packaged all of that as a Broken Link Checker on Apify. Type a domain and it reads the sitemap (or follows internal links), checks every link, marks bot-blocking sites as restricted instead of broken, and tells you which page each broken link is on. It costs $0.80 per 1,000 links checked, with no subscription, and you can schedule it weekly. On python.org, 30 pages and 2,000 links took about a minute and found 4 broken links. It also respects each site's robots.txt, so it skips what sites ask crawlers not to touch, and doesn't charge for those.

If you also want titles, meta descriptions, and H1s checked, the same idea powers a Website SEO Audit that scores each page and lists fixes.

Either way, the script above is yours. Copy it, change it, run it on a schedule.