DEV Community: Datawright Labs

Find broken links on any website with Python, grouped by the page you need to fix (2026)

Datawright Labs — Wed, 21 Oct 2026 02:16:10 +0000

A broken-link report is only useful if it tells you where to go to fix things. "This URL returns 404" is half the answer. "These three pages link to it" is the part you can act on. Most quick scripts check one page at a time and lose that mapping, or they check the same footer link 50 times.

This post builds a small crawler that does it properly: crawl the site, collect every outgoing URL once, check each one once, and write a CSV that maps every broken URL back to the pages that contain it. Then it looks at a real run, because the interesting part of link checking is the false positives.

The approach

Breadth-first crawl of same-site pages. Start at the homepage, follow <a href> links that stay on the same host, stop at a page limit.
Collect every outgoing URL with its source pages. Links, images, scripts and stylesheets all count. A missing image or script is as broken as a dead link. Strip #fragments so /docs#install and /docs#usage are one check.
Check each unique URL once, in parallel. Use GET with stream=True instead of HEAD. Plenty of servers answer HEAD with 405 or 404 even when the page is fine, and some just hang on it. A streamed GET reads only the headers and closes the connection.
Retry once before calling something broken. Transient timeouts are common.

The script

# Usage: python3 broken_links.py https://example.com [max_pages]
# Crawls same-site pages (BFS), checks every link and image once, writes broken.csv grouped by source page.
import csv, sys, time, collections
from concurrent.futures import ThreadPoolExecutor
from urllib.parse import urljoin, urldefrag, urlparse
import requests
from bs4 import BeautifulSoup

start = sys.argv[1]
max_pages = int(sys.argv[2]) if len(sys.argv) > 2 else 50
host = urlparse(start).netloc
s = requests.Session()
s.headers["User-Agent"] = "Mozilla/5.0 (compatible; link-check-tutorial/1.0)"

def check(url):
    # GET with stream=True reads headers only; many servers answer HEAD wrongly (405/404) or hang.
    for attempt in range(2):
        try:
            r = s.get(url, stream=True, timeout=15, allow_redirects=True)
            r.close()
            return r.status_code, r.url
        except requests.RequestException as e:
            err = type(e).__name__
            time.sleep(2)
    return err, url

sources = collections.defaultdict(set)   # link -> pages that contain it
queue, seen_pages, i = [start], {start}, 0
t = time.time()
while i < len(queue) and i < max_pages:
    page = queue[i]; i += 1
    try:
        r = s.get(page, timeout=20)
    except requests.RequestException:
        continue
    if "text/html" not in r.headers.get("content-type", ""):
        continue
    soup = BeautifulSoup(r.text, "html.parser")
    for tag, attr in (("a", "href"), ("img", "src"), ("link", "href"), ("script", "src")):
        for el in soup.find_all(tag):
            raw = el.get(attr)
            if not raw or raw.startswith(("mailto:", "tel:", "javascript:", "data:", "#")):
                continue
            link = urldefrag(urljoin(r.url, raw))[0]
            if not link.startswith("http"):
                continue
            sources[link].add(page)
            if tag == "a" and urlparse(link).netloc == host and link not in seen_pages:
                seen_pages.add(link); queue.append(link)

links = sorted(sources)
with ThreadPoolExecutor(10) as pool:
    results = dict(zip(links, pool.map(check, links)))

broken = [(u, st) for u, (st, _) in results.items() if not (isinstance(st, int) and st < 400)]
with open("broken.csv", "w", newline="") as f:
    w = csv.writer(f); w.writerow(["source_page", "broken_url", "status"])
    for u, st in sorted(broken):
        for p in sorted(sources[u]):
            w.writerow([p, u, st])
print(f"{i} pages crawled, {len(links)} unique URLs checked, {len(broken)} broken in {time.time() - t:.0f} s -> broken.csv")
for u, st in sorted(broken, key=lambda x: str(x[1]))[:15]:
    print(f"{str(st):<16} {u}  (on {len(sources[u])} page(s))")

Install the two dependencies with pip install requests beautifulsoup4 and run python3 broken_links.py https://your-site.com 200.

Results on crawlee.dev

I ran python3 broken_links.py https://crawlee.dev 50 on October 20, 2026:

50 pages crawled, 3,008 unique URLs checked in 245 seconds, 10 checks in parallel.
35 URLs flagged. That number is misleading, which is the main lesson of this post.

Here is what the 35 actually were:

Status	Count	Really broken?
403	14	No. Stack Overflow, npm, Crunchbase, Upwork, OpenAI and others block scripted requests. The Stack Overflow tag link alone appears on 37 pages.
999	2	No. LinkedIn's "go away, bot" status.
Connect timeout	7	No. All were plain `http://` links, and my test machine's network blocks outbound port 80. Worth knowing if you run this from a locked-down server or CI runner.
404	8	7 yes, 1 no (see below).
SSL error	4	Yes.

So 11 of the 35 were genuinely broken, and every one of them is a real typo or a bad link:

https://www.crawlee.dev/python/docs/examples/playwright-crawler-with-fingeprint-generator on a blog post, with "fingeprint" missing an "r".
Three links like https://github.com//github.com/apify/crawlee/pull/1890/..., where a full URL got pasted into a field that already prefixed https://github.com/.
https://www.en.wikipedia.org/wiki/Robots.txt and https://www.developer.mozilla.org/...: an extra www. in front of hosts that don't have one, so the TLS certificate doesn't match.
A github.com/renovate%5Bbot%5D profile link (bot accounts have no profile page) and two SwaggerHub API docs that no longer exist.

The one 404 that wasn't broken was https://5JC94MPMLY-dsn.algolia.net, a <link rel="preconnect"> to the search provider. Its root path 404s, but nobody ever navigates there.

Things that will bite you

Treat 401, 403, 429 and 999 as "blocked", not "broken". Report them in a separate column, or re-check them from a real browser. If you count them as broken, your report is mostly noise and people stop reading it.
Skip preconnect and dns-prefetch links, or at least don't treat them as pages. They point at hosts, not documents.
Check from a normal network. Corporate proxies and CI runners often block plain HTTP or whole domains, which shows up as timeouts.
Be polite on your own site. This script checks external URLs 10 at a time but has no per-host limit. On a big site most links point at your own host, so add a per-host concurrency cap and respect robots.txt before you crawl thousands of pages.
Watch the redirect chains. allow_redirects=True hides a 301 to a 404. The final status is what matters for users, but r.history tells you which links are worth updating to skip the redirect.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this as a hosted tool on Apify: Broken Link Checker. It adds robots.txt handling, a per-host concurrency limit, a spending cap per run, and source-page grouping for every broken link, image and script. It costs $0.50 per 1,000 URLs checked, so Apify's free monthly credit covers small and medium sites, and you can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.

Google autocomplete for keyword research in Python: A-Z, question and YouTube suggestions in bulk (2026)

Datawright Labs — Tue, 20 Oct 2026 18:20:02 +0000

Autocomplete is the closest thing to free search demand data you can get. Every suggestion Google, YouTube or Bing shows is something real people type often enough for the engine to offer it, and the endpoints that power those dropdowns return plain JSON with no API key. Tools like AnswerThePublic and keyword.io are mostly this idea plus a nice UI.

This post covers the three endpoints, the expansion trick that turns 10 suggestions into hundreds, and a small script that merges everything into one ranked CSV.

The endpoints

Google (and YouTube and Google Shopping, same host):

https://suggestqueries.google.com/complete/search?client=chrome&hl=en&gl=us&q=standing+desk

client=chrome returns JSON shaped like ["query", ["suggestion 1", ...], [descriptions], [], {"google:suggesttype": ["QUERY", "NAVIGATION", ...], "google:suggestrelevance": [...]}]. The type list lets you drop navigational results (URLs) and keep real queries.
ds=yt switches to YouTube suggestions, ds=sh to Google Shopping. YouTube ideas skew toward "how to", reviews and comparisons, which is exactly what you want for video or tutorial content.
hl is the language and gl the country. Suggestions differ a lot by market, so set both.
Decode the body as UTF-8 yourself. With client=chrome the response can come back without a charset, and non-English suggestions get mangled if you trust the default.

Bing:

https://api.bing.com/osjson.aspx?query=standing+desk&mkt=en-US

This is the OpenSearch suggestion format: ["query", ["suggestion 1", ...]]. Bing's list is noticeably different from Google's, so it's worth a second source.

Turning 10 suggestions into 1,000

Each request only returns up to about 10 suggestions. The standard trick is to query variations of the seed and merge the results:

A-Z: standing desk a, standing desk b, ... standing desk z. Each letter surfaces a different slice of long-tail phrases.
Question prefixes: how standing desk, is standing desk, why standing desk. These give you the "is a standing desk worth it" style queries that make good FAQ sections and blog posts.
Modifier suffixes: vs, for, without, near me surface comparisons, audiences and local intent.

Then dedupe (lowercase and collapse whitespace) and keep track of which engines suggested each idea. A phrase that Google, YouTube and Bing all suggest is a stronger signal than one that shows up once.

The script

Only dependency: pip install requests.

import csv, json, string, sys, time
import requests

UA = {"User-Agent": "Mozilla/5.0 (autocomplete-tutorial/1.0)"}
PREFIXES = ["how", "what", "why", "can", "is", "best"]  # question-style ideas
SUFFIXES = ["vs", "for", "without", "near me"]

def google(q, hl="en", gl="us", ds=None):
    p = {"client": "chrome", "ie": "utf-8", "oe": "utf-8", "hl": hl, "gl": gl, "q": q}
    if ds:
        p["ds"] = ds  # "yt" = YouTube, "sh" = Google Shopping
    r = requests.get("https://suggestqueries.google.com/complete/search", params=p, headers=UA, timeout=20)
    r.raise_for_status()
    data = json.loads(r.content.decode("utf-8"))
    types = (data[4] if len(data) > 4 else {}).get("google:suggesttype", [])
    return [s for i, s in enumerate(data[1]) if i >= len(types) or types[i] == "QUERY"]

def bing(q, mkt="en-US"):
    r = requests.get("https://api.bing.com/osjson.aspx", params={"query": q, "mkt": mkt}, headers=UA, timeout=20)
    r.raise_for_status()
    return r.json()[1]

ENGINES = {"google": google, "youtube": lambda q: google(q, ds="yt"), "bing": bing}

def expand(seed):
    yield seed
    for c in string.ascii_lowercase:
        yield f"{seed} {c}"
    for w in PREFIXES:
        yield f"{w} {seed}"
    for w in SUFFIXES:
        yield f"{seed} {w}"

def ideas(seed, engines=("google", "youtube", "bing"), pause=0.3):
    seen = {}
    for engine in engines:
        for q in expand(seed):
            try:
                results = ENGINES[engine](q)
            except requests.RequestException as e:
                print(f"  {engine} {q!r}: {e}", file=sys.stderr)
                time.sleep(5)
                continue
            for pos, s in enumerate(results, 1):
                k = " ".join(s.lower().split())
                row = seen.setdefault(k, {"keyword": k, "engines": set(), "best_position": pos, "hits": 0})
                row["engines"].add(engine)
                row["best_position"] = min(row["best_position"], pos)
                row["hits"] += 1
            time.sleep(pause)
    return sorted(seen.values(), key=lambda r: (-len(r["engines"]), -r["hits"], r["best_position"]))

if __name__ == "__main__":
    seed = " ".join(sys.argv[1:]) or "standing desk"
    t = time.time()
    rows = ideas(seed)
    with open("keywords.csv", "w", newline="") as f:
        w = csv.writer(f)
        w.writerow(["keyword", "engines", "hits", "best_position"])
        for r in rows:
            w.writerow([r["keyword"], "+".join(sorted(r["engines"])), r["hits"], r["best_position"]])
    print(f"{len(rows)} unique ideas for {seed!r} in {time.time() - t:.1f} s -> keywords.csv")
    for r in rows[:10]:
        print(f'{r["keyword"]:<45} {"+".join(sorted(r["engines"])):<20} hits={r["hits"]}')

Results

I ran python3 autocomplete_ideas.py standing desk on October 20, 2026. That's 37 queries per engine, 111 requests in total with a 0.3 s pause between them:

1,013 unique keyword ideas in 57 seconds.
73 of them were suggested by all three engines, for example "standing desk with drawers", "standing desk topper", "standing desk treadmill" and "standing desk benefits".
Each engine contributed a lot of its own: 286 ideas only from Google, 275 only from YouTube and 212 only from Bing. Using just one engine would have missed roughly two thirds of the list.
The question prefixes produced usable content ideas straight away: "is standing desk worth it", "is standing desk better than sitting", "is standing desk hsa eligible", "what standing desk to buy".
One Bing request failed with a transient SSL error. The script logs it, waits 5 seconds and moves on, which is why the error handling is there.

Things that will bite you at scale

Rate limits: a few hundred requests from one IP is fine. Thousands in a row from a single IP will start getting HTTP 429 or a captcha page instead of JSON (the response stops parsing as JSON, which is the easiest way to detect it). Slow down, add retries with backoff, or spread requests across IPs.
Personalization: suggestions vary with language, country and sometimes location. Pin hl, gl and mkt so runs are comparable over time.
No volume numbers: autocomplete tells you a query is popular enough to suggest, not how many searches it gets. Use the engine count, the number of expansions an idea appears in, and its position as a relative signal, then check the top candidates in a volume tool if you need numbers.
Duplicates across expansions: "standing desk with drawers" comes back from the seed, from "w" and from "with". Merge before you count, otherwise your favourite ideas look more important than they are.

Hosted version

Disclosure: I'm with Datawright Labs, and we packaged this (Google, YouTube, Google Shopping and Bing, A-Z, 0-9 and question expansions, any language and country, recursive depth, must-contain and exclude-word filters, retries and proxy rotation for big seed lists) as a hosted tool on Apify: Autocomplete Keyword Scraper. It costs $0.20 per 1,000 unique suggestions, with duplicates merged before you pay, so Apify's free monthly credit covers most small jobs. You can run it on a schedule or call it by API. The script above is the whole idea, though, so use whichever fits.

Extract every URL from a sitemap.xml in Python (indexes, .gz, robots.txt) and spot competitors' new pages

Datawright Labs — Tue, 20 Oct 2026 16:21:16 +0000

A site's sitemap is the cheapest full list of its pages you will ever get: the site publishes it for search engines, it usually carries a lastmod date per URL, and reading it takes seconds instead of crawling HTML for hours. That makes it the right starting point for SEO audits, migrations, content inventories, feeding a crawler or RAG pipeline, and keeping an eye on what competitors publish.

The catch is that "the sitemap" is rarely one file. Here's what you actually run into, and a small Python script that handles it.

What real sitemaps look like

Discovery: the reliable place to look is robots.txt, which lists sitemaps on Sitemap: lines (often several). If it has none, try the usual paths: /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml (WordPress core), /sitemap.xml.gz.
Sitemap indexes: big sites publish a <sitemapindex> that points to more sitemaps, and indexes can point to other indexes. Yoast, Rank Math, Shopify and most frameworks split by post type or by 50,000 URLs (the per-file limit in the protocol).
Gzip: .xml.gz files are common on news and e-commerce sites. Don't trust the Content-Type header; check for the gzip magic bytes 1f 8b instead.
Namespaces: every tag is namespaced ({http://www.sitemaps.org/schemas/sitemap/0.9}loc), and image, video, news and xhtml:link hreflang extensions add more. Strip the namespace and match on the local name.
No sitemap at all: some well-known sites don't publish one. In my test python.org returned nothing from robots.txt or the common paths, so you need a crawler for those.
Loops and duplicates: some indexes list themselves, and the same URL can appear in several sitemaps. Track what you've visited and dedupe.

The script

Only dependency: pip install requests. Standard-library xml.etree does the parsing.

import gzip, json, pathlib, re, sys, time
import xml.etree.ElementTree as ET
from urllib.parse import urljoin
import requests

UA = {"User-Agent": "sitemap-tutorial/1.0"}
COMMON = ["/sitemap.xml", "/sitemap_index.xml", "/wp-sitemap.xml", "/sitemap.xml.gz"]

def find_sitemaps(site):
    site = site if site.startswith("http") else "https://" + site
    found = []
    try:
        r = requests.get(urljoin(site, "/robots.txt"), headers=UA, timeout=20)
        if r.ok:
            found = [l.split(":", 1)[1].strip() for l in r.text.splitlines()
                     if l.lower().startswith("sitemap:")]
    except requests.RequestException:
        pass
    if found:
        return found
    for path in COMMON:  # fall back to the usual locations
        u = urljoin(site, path)
        try:
            r = requests.get(u, headers=UA, timeout=20)
            if r.ok and b"<" in r.content[:200]:
                return [u]
        except requests.RequestException:
            pass
    return []

def fetch_xml(url):
    r = requests.get(url, headers=UA, timeout=30)
    r.raise_for_status()
    body = r.content
    if body[:2] == b"\x1f\x8b":  # gzipped, whatever the Content-Type says
        body = gzip.decompress(body)
    return ET.fromstring(body)

def local(tag):  # strip the XML namespace: {http://...}loc -> loc
    return tag.rsplit("}", 1)[-1]

def walk(url, seen=None):
    seen = seen if seen is not None else set()
    if url in seen:  # some indexes list themselves
        return
    seen.add(url)
    root = fetch_xml(url)
    if local(root.tag) == "sitemapindex":
        for sm in root:
            loc = next((c.text.strip() for c in sm if local(c.tag) == "loc" and c.text), None)
            if loc:
                yield from walk(loc, seen)
    else:
        for u in root:
            row = {local(c.tag): (c.text or "").strip() for c in u if local(c.tag) in ("loc", "lastmod")}
            if row.get("loc"):
                yield row

def all_urls(site):
    urls = {}
    for sm in find_sitemaps(site):
        try:
            for row in walk(sm):
                urls[row["loc"]] = row.get("lastmod")
        except Exception as e:
            print(f"  skipped {sm}: {e}", file=sys.stderr)
    return urls

def new_since_last_run(site, urls, state=pathlib.Path("seen.json")):
    seen = json.loads(state.read_text()) if state.exists() else {}
    old = set(seen.get(site, []))
    seen[site] = sorted(urls)
    state.write_text(json.dumps(seen))
    return [u for u in urls if u not in old] if old else []

if __name__ == "__main__":
    for site in sys.argv[1:]:
        t = time.time()
        urls = all_urls(site)
        print(f"{site}: {len(urls)} URLs in {time.time() - t:.1f}s")

Run it with one or more sites:

python3 sitemap_urls.py crawlee.dev docs.apify.com allbirds.com stripe.com

What I got just now (October 20, 2026):

Site	URLs	Time
crawlee.dev	4,848	0.5 s
docs.apify.com	4,071	1.0 s
allbirds.com (Shopify)	2,537	2.7 s
stripe.com	7,113	7.6 s
python.org	0 (no sitemap published)	1.8 s

Gzip works the same way: walk("https://www.nytimes.com/sitemaps/new/news.xml.gz") returned 621 news URLs in 0.2 s, the newest with a lastmod only minutes old.

One thing to watch: robots.txt lives at the root, so asking for wordpress.org/news gave me 189,145 URLs for the whole of wordpress.org in 53 s. Filter on the path afterwards ([u for u in urls if "/news/" in u]) when you only care about one section.

Spotting new pages on a schedule

new_since_last_run() saves the URL set per site in seen.json and returns only the URLs it hasn't seen before. The first run records a baseline and returns nothing; after that, run it daily (cron, GitHub Actions) and you get a list of a competitor's new blog posts, product pages or docs:

urls = all_urls("competitor.com")
for u in new_since_last_run("competitor.com", urls):
    print("NEW:", u, urls[u])

To check it, I removed five URLs from the saved state and re-ran: exactly those five came back as new.

Ideas for extending it:

Filter by lastmod ("changed in the last 7 days") to find refreshed content, not just new content.
Read xhtml:link alternates to audit hreflang coverage across languages.
Feed the URL list into a bulk PageSpeed or broken-link check.
Send the new URLs to Slack or a Google Sheet.

If you'd rather not run it yourself

Disclosure: I'm with Datawright Labs, and we packaged this (plus text, RSS and Atom sitemaps, hreflang, image and Google News fields, URL and date filters, and the new-URL monitoring) as a hosted tool on Apify: Sitemap URL Extractor. It costs $0.20 per 1,000 URLs, which Apify's free monthly credit covers for most small jobs. You can run it on a schedule and get results by API, webhook or CSV. The script above is the whole idea, though, so use whichever fits.

Greenhouse, Lever, Ashby, Workable & SmartRecruiters job APIs: endpoints, fields and one normalized feed (2026)

Datawright Labs — Tue, 20 Oct 2026 13:00:00 +0000

If you're building a niche job board, a hiring-signal alert, or just tracking a list of target companies, you don't need to scrape HTML. Most tech companies post jobs through an applicant tracking system (ATS), and the big five all expose the same public JSON feed that powers the company's own careers page. No login, no API key, no headless browser.

I checked all five this morning (October 2026). Here's what each one returns, the gotchas, and a small Python script that merges them into one schema.

The five endpoints

ATS	Careers page looks like	JSON endpoint
Greenhouse	`boards.greenhouse.io/figma`	`https://boards-api.greenhouse.io/v1/boards/{slug}/jobs`
Lever	`jobs.lever.co/spotify`	`https://api.lever.co/v0/postings/{slug}?mode=json`
Ashby	`jobs.ashbyhq.com/ramp`	`https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true`
Workable	`apply.workable.com/huggingface`	`https://apply.workable.com/api/v1/widget/accounts/{slug}`
SmartRecruiters	`careers.smartrecruiters.com/Ubisoft2`	`https://api.smartrecruiters.com/v1/companies/{slug}/postings?limit=100&offset=0`

The slug is just the last part of the careers-page URL.

Test run just now: Figma 141 jobs (0.26 s), Spotify 81 (1.1 s), Ramp 167 (0.16 s), Hugging Face 6 (0.17 s), Ubisoft 329 across 4 pages. That's 724 jobs from five companies in about 3.5 seconds.

Quirks worth knowing

Greenhouse: the list endpoint has title, location.name, absolute_url, updated_at and first_published. Add ?content=true if you need the HTML description and departments in the same call (much bigger response).
Lever: returns a bare JSON array. createdAt is epoch milliseconds, not ISO. EU-hosted companies live on api.eu.lever.co; the US host answers 404 for them, so try both. Plain-text descriptions come for free (descriptionPlain).
Ashby: includeCompensation=true adds structured salary ranges when the company publishes them. You also get isRemote and workplaceType directly, which is the cleanest remote flag of the five.
Workable: the widget endpoint is light; pass ?details=true to get descriptions. Remote is a boolean called telecommuting.
SmartRecruiters: paginated with limit (max 100) and offset; stop when offset >= totalFound. The list has no description; fetch /postings/{id} per job if you need it, and be polite about rate.

One normalized feed in Python

import requests

UA = {"User-Agent": "job-feed-tutorial/1.0"}

def greenhouse(slug):
    r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", headers=UA, timeout=30)
    for j in r.json()["jobs"]:
        yield {"title": j["title"], "location": j["location"]["name"],
               "url": j["absolute_url"], "posted": j.get("first_published") or j["updated_at"]}

def lever(slug):
    r = requests.get(f"https://api.lever.co/v0/postings/{slug}?mode=json", headers=UA, timeout=30)
    if r.status_code == 404:  # EU-hosted boards live on a different host
        r = requests.get(f"https://api.eu.lever.co/v0/postings/{slug}?mode=json", headers=UA, timeout=30)
    for j in r.json():
        yield {"title": j["text"], "location": j["categories"].get("location"),
               "url": j["hostedUrl"], "posted": j["createdAt"]}  # createdAt is epoch ms

def ashby(slug):
    r = requests.get(f"https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true", headers=UA, timeout=30)
    for j in r.json()["jobs"]:
        yield {"title": j["title"], "location": j["location"],
               "url": j["jobUrl"], "posted": j["publishedAt"]}

def workable(slug):
    r = requests.get(f"https://apply.workable.com/api/v1/widget/accounts/{slug}", headers=UA, timeout=30)
    for j in r.json()["jobs"]:
        loc = ", ".join(x for x in (j.get("city"), j.get("country")) if x)
        yield {"title": j["title"], "location": loc,
               "url": j["url"], "posted": j.get("published_on")}

def smartrecruiters(slug):
    offset = 0
    while True:
        r = requests.get(f"https://api.smartrecruiters.com/v1/companies/{slug}/postings",
                         params={"limit": 100, "offset": offset}, headers=UA, timeout=30).json()
        for j in r["content"]:
            yield {"title": j["name"], "location": j["location"].get("fullLocation") or j["location"].get("city"),
                   "url": f"https://jobs.smartrecruiters.com/{slug}/{j['id']}", "posted": j["releasedDate"]}
        offset += 100
        if offset >= r["totalFound"]:
            break

PLATFORMS = {"greenhouse": greenhouse, "lever": lever, "ashby": ashby,
             "workable": workable, "smartrecruiters": smartrecruiters}

Use it like this:

companies = ["greenhouse:figma", "lever:spotify", "ashby:ramp",
             "workable:huggingface", "smartrecruiters:Ubisoft2"]
feed = []
for c in companies:
    platform, slug = c.split(":")
    feed += [dict(j, company=slug, platform=platform) for j in PLATFORMS[platform](slug)]
print(len(feed))  # 724 this morning

Detecting new and closed jobs

The interesting signal is usually the change: a company that opens ten data-engineering roles in a week is hiring a team. Keep the last run's job URLs and diff:

import json, pathlib

state = pathlib.Path("seen.json")
before = set(json.loads(state.read_text())) if state.exists() else set()
now = {j["url"] for j in feed}

new_jobs = [j for j in feed if j["url"] not in before]
closed = before - now
state.write_text(json.dumps(sorted(now)))
print(f"{len(new_jobs)} new, {len(closed)} closed")

Run it on a cron and push new_jobs to Slack, a sheet, or your board.

Pitfalls

Finding the slug: companies with a custom careers domain often still embed boards.greenhouse.io/embed/job_board?for=<slug> or a jobs.lever.co/<slug> link. View source and search for the ATS name.
Bad slugs fail differently: Greenhouse, Lever, Ashby and Workable return 404, but SmartRecruiters returns 200 with an empty list (totalFound: 0), so check the count too. Log per company instead of crashing the whole run.
Be a good citizen: these feeds are public, but one request per company per few hours is plenty. Set a real User-Agent.
Postings only: none of these endpoints expose recruiter personal data, and you don't need it.

If you'd rather not maintain it

I packaged the same approach (all five platforms, one schema, title/location/remote/date filters, salary where published, and new/closed detection across scheduled runs) as a hosted tool: ATS Jobs Scraper on Apify. It's pay-per-result at $1 per 1,000 jobs, and Apify's free monthly credit covers small watch-lists. You can also call it as an MCP tool from an AI agent.

But honestly, for a handful of companies the script above is all you need. Happy building!