DEV Community: William Jin

Scrape Greenhouse, Lever and Ashby job boards into one JSON feed (Python)

William Jin — Mon, 26 Oct 2026 02:21:20 +0000

Open the careers page of almost any startup or scale-up and the job list is usually served by one of three applicant tracking systems: Greenhouse, Lever or Ashby. You can tell from the URL (boards.greenhouse.io/stripe, jobs.lever.co/spotify, jobs.ashbyhq.com/ramp) or from where the "Apply" button points.

All three publish the board as public JSON, so you don't need a browser or HTML parsing. The catch is that each one uses a different shape. This post shows the three endpoints, a script that merges them into one feed, the quirks to normalize, and a hosted option you can also give to an AI agent.

The three endpoints

ATS	Endpoint	Notes
Greenhouse	`GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs`	Add `?content=true` for descriptions (much larger response)
Lever	`GET https://api.lever.co/v0/postings/{slug}?mode=json`	`createdAt` is a Unix timestamp in milliseconds
Ashby	`GET https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true`	Includes `isRemote`, `workplaceType` and pay when the company publishes it

The slug is the company's board name, usually the last part of the careers URL.

One script, three boards

import requests

def greenhouse(slug):
    r = requests.get(f"https://boards-api.greenhouse.io/v1/boards/{slug}/jobs", timeout=30)
    r.raise_for_status()
    for j in r.json()["jobs"]:
        yield {"company": slug, "title": j["title"], "location": j["location"]["name"],
               "posted": j.get("first_published"), "url": j["absolute_url"]}

def lever(slug):
    r = requests.get(f"https://api.lever.co/v0/postings/{slug}?mode=json", timeout=30)
    r.raise_for_status()
    for j in r.json():
        yield {"company": slug, "title": j["text"], "location": j["categories"].get("location"),
               "posted": j["createdAt"], "url": j["hostedUrl"]}

def ashby(slug):
    r = requests.get(f"https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true", timeout=30)
    r.raise_for_status()
    for j in r.json()["jobs"]:
        yield {"company": slug, "title": j["title"], "location": j["location"],
               "posted": j["publishedAt"], "url": j["jobUrl"]}

boards = [(greenhouse, "stripe"), (lever, "spotify"), (ashby, "ramp")]
jobs = [j for fetch, slug in boards for j in fetch(slug)]
data_jobs = [j for j in jobs if "data" in j["title"].lower()]
print(len(jobs), "jobs,", len(data_jobs), "with 'data' in the title")
for j in data_jobs[:3]:
    print(j)

When I ran it, it returned 976 jobs across the three companies, 41 of them with "data" in the title:

{'company': 'stripe', 'title': 'Account Executive, Product Sales (Data)', 'location': 'US-SF-HQ, US-NYC, US-CHI, US-SEA', 'posted': '2026-10-05T12:08:53-04:00', 'url': 'https://stripe.com/jobs/search?gh_jid=8190059'}
{'company': 'stripe', 'title': 'Data Analyst', 'location': 'Canada', 'posted': '2023-11-01T13:50:50-04:00', 'url': 'https://stripe.com/jobs/search?gh_jid=5416444'}

That's a usable feed in about 30 lines. Here's what you'll want to fix before relying on it.

The gotchas

1. Dates mean different things. Greenhouse has updated_at (changes whenever the posting is edited) and first_published (when it went live). Use first_published for "new jobs" alerts, or a job edited yesterday looks brand new. Lever's createdAt is milliseconds since the epoch (1792767640768), so divide by 1000 before converting. Ashby gives an ISO publishedAt.

2. Location is free text. Greenhouse returns whatever the company typed ("US-SF-HQ, US-NYC, US-CHI, US-SEA"), Lever has a categories.location string plus a separate workplaceType (onsite, hybrid, remote), and Ashby has location, secondaryLocations and isRemote. To filter by "remote" or "Germany" reliably, check all of those fields, not just the location string.

3. Pay is only sometimes structured. Ashby returns a compensation object when the company shows pay (for example "$211.4K – $290.6K • Offers Equity" as a summary). Greenhouse and Lever usually put pay ranges inside the description text, so you need a parser that handles $90K, €60,000, ranges and hourly rates.

4. Descriptions are heavy. Greenhouse's ?content=true turned Stripe's board into a 5.6 MB response. Fetch descriptions only when you need them.

5. Unknown slugs return 404. A typo in the slug and a company that moved to a different ATS look the same. Log which boards returned nothing, so an empty result doesn't quietly look like "no jobs".

6. Be polite. These are public endpoints, but they serve the company's real careers page. One request per board per run is plenty; back off on 429 and 5xx.

If you'd rather not maintain it

Disclosure: I packaged this as a hosted tool, Career Site Jobs Scraper on Apify, with the source on GitHub. It covers Greenhouse, Lever, Ashby, Workable and Recruitee, accepts board names or full URLs, and returns one normalized row per job, with dates, remote status and parsed salary.

The input:

{
  "companies": ["stripe", "https://jobs.lever.co/spotify", "https://jobs.ashbyhq.com/ramp"],
  "titleKeywords": ["data"],
  "maxJobsPerCompany": 3,
  "includeDescription": false
}

One row from that run (an Ashby job with published pay), trimmed:

{
  "ats": "ashby",
  "companySlug": "ramp",
  "companyName": "Ramp",
  "title": "Data Scientist",
  "department": "Data",
  "location": "New York, NY (HQ)",
  "locations": ["New York, NY (HQ)", "San Francisco, CA", "Remote (US)"],
  "remote": true,
  "workplaceType": "hybrid",
  "employmentType": "FullTime",
  "postedAt": "2026-06-03T17:53:31.348Z",
  "jobUrl": "https://jobs.ashbyhq.com/ramp/e577622f-6657-4e53-8941-b3a774b04448",
  "salaryText": "$137,120 - $297,330",
  "salaryMin": 137120,
  "salaryMax": 297330,
  "salaryCurrency": "USD",
  "salaryInterval": "year"
}

The run also writes a per-company summary, so you can see why a board returned few rows:

{"input": "stripe", "ats": "greenhouse", "status": "ok", "jobsFound": 719, "jobsMatched": 3, "jobsSaved": 3}

Other filters, applied before anything is charged: excludeTitleKeywords, locationKeywords, departmentKeywords, remoteOnly and postedSince (a date or a period like "7 days").

Call it from Python

import os
from apify_client import ApifyClient  # pip install apify-client

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/career-site-jobs-scraper").call(
    run_input={
        "companies": ["stripe", "https://jobs.lever.co/spotify", "https://jobs.ashbyhq.com/ramp"],
        "titleKeywords": ["data"],
        "postedSince": "14 days",
        "includeDescription": False,
    },
    max_total_charge_usd=0.05,
)
for job in client.dataset(run.default_dataset_id).iterate_items():
    print(job.get("companyName"), "|", job.get("title"), "|", job.get("location"))

max_total_charge_usd caps what the run can cost; when it's reached the run stops cleanly and keeps the rows it already saved. (On 1.x versions of the client, use run["defaultDatasetId"].)

Let an AI agent pull the jobs (MCP)

In Claude, Cursor, VS Code or another MCP client, add Apify's hosted MCP server with just this tool:

{
  "mcpServers": {
    "apify": { "url": "https://mcp.apify.com?tools=swillz/career-site-jobs-scraper" }
  }
}

Then ask something like:

"Find remote data or ML jobs posted in the last 7 days at stripe, ramp, notion and figma, and give me a table with company, title, location and link."

The agent fills in companies, titleKeywords, remoteOnly: true and postedSince: "7 days". Tips:

Keep answers fast. Have the agent set maxJobs to 50–200 so the run finishes within the MCP call-actor wait (about 45 seconds).
Empty results come with a reason. A company with no matching jobs gets a free status row such as "N open jobs found, 0 matched your filters" or "not_found", so the agent can fix the slug or loosen a filter instead of reporting "no jobs".
Chain it. The dataset ID from the run can go straight into a Google Sheets export or a Slack webhook.

A daily schedule with postedSince: "1 day" across a list of companies gives you a "new jobs at companies I care about" feed without maintaining any scrapers.

Wrap-up

Greenhouse, Lever and Ashby all serve job boards as public JSON; the slug is in the careers URL.
Normalize dates (first_published, millisecond createdAt), locations and remote flags before filtering.
Pay is structured on Ashby when published; elsewhere it's in the description text.
Track boards that return nothing, so a wrong slug doesn't look like "not hiring".

How to scrape Workday job postings to JSON (and get past the 2,000-job limit)

William Jin — Fri, 23 Oct 2026 20:41:46 +0000

A huge number of large employers host their careers pages on Workday. You can spot them by the address: something.wd5.myworkdayjobs.com/SiteName. If you're building a job board, tracking hiring at target companies, doing labor-market research, or just want an alert when a company posts a role you care about, you'll want those postings as structured data, not HTML.

The good news is that Workday's job search page is a single-page app backed by a JSON API, so you don't need a headless browser. The less good news is a handful of quirks, the main one being that search results stop at 2,000. This post walks through the API, a short Python scraper, the gotchas, and a hosted option you can also hand to an AI agent.

Step 1: Decode the career-site URL

Take NVIDIA's careers site:

https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite
         └─tenant─┘└host┘                └──────site──────┘

Host: nvidia.wd5.myworkdayjobs.com (the wd1, wd3, wd5... part varies by company)
Tenant: nvidia
Site: NVIDIAExternalCareerSite

Language prefixes like /en-US/ in front of the site name are optional. Some companies use wd3.myworkdaysite.com/recruiting/<tenant>/<site> instead; same idea, different shape.

Step 2: Call the jobs endpoint

The search page posts to:

POST https://{host}/wday/cxs/{tenant}/{site}/jobs
Content-Type: application/json

{"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "data engineer"}

The response has total, a page of jobPostings (each with title, locationsText, postedOn and externalPath), and facets, the filter options you see in the sidebar.

Here's a minimal scraper using requests:

import requests

def workday_jobs(tenant, host, site, search="", limit=20, max_jobs=100):
    url = f"https://{host}/wday/cxs/{tenant}/{site}/jobs"
    offset, jobs = 0, []
    while len(jobs) < max_jobs:
        r = requests.post(url, json={"appliedFacets": {}, "limit": limit,
                                     "offset": offset, "searchText": search}, timeout=30)
        r.raise_for_status()
        data = r.json()
        page = data.get("jobPostings", [])
        if not page:
            break
        for p in page:
            jobs.append({
                "title": p.get("title"),
                "location": p.get("locationsText"),
                "posted": p.get("postedOn"),
                "url": f"https://{host}/{site}{p.get('externalPath', '')}",
            })
        offset += limit
        if offset >= data.get("total", 0):
            break
    return jobs[:max_jobs]

jobs = workday_jobs("nvidia", "nvidia.wd5.myworkdayjobs.com",
                    "NVIDIAExternalCareerSite", search="data engineer", max_jobs=5)
for j in jobs:
    print(j)

When I ran it, the first rows looked like this:

{'title': 'Senior Data Engineer', 'location': '3 Locations', 'posted': 'Posted 4 Days Ago',
 'url': 'https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/US-CA-Remote/Senior-Data-Engineer_JR2027678'}
{'title': 'Senior Data Center Engineer', 'location': 'Israel, Tel Aviv', 'posted': 'Posted 30+ Days Ago',
 'url': 'https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/Israel-Tel-Aviv/Senior-Data-Center-Engineer_JR2023708'}

That's a working scraper in about 30 lines. Now for the parts that bite.

The gotchas

1. Search results stop at 2,000. Ask NVIDIA's endpoint for all jobs with an empty search and total comes back as exactly 2000, even though the company has more open roles than that. Paging past offset 2,000 gets you nothing. To get everything, split the search into smaller slices with appliedFacets (the facet IDs come from the facets array in the response; on NVIDIA's site they include jobFamilyGroup, timeType, workerSubType and locationMainGroup). Query each slice, keep splitting any slice that still reports 2,000, and de-duplicate by job ID at the end, because one job can appear in several slices.

2. limit maxes out at 20. Ask for "limit": 21 and you get an HTTP 400. Pages of 20 it is, so be patient and pace your requests.

3. The list view is a summary. locationsText says "3 Locations" rather than listing them, and postedOn is relative text ("Posted Yesterday", "Posted 30+ Days Ago"). For full locations, the description, time type and pay range, fetch each job's detail JSON at https://{host}/wday/cxs/{tenant}/{site}{externalPath}. That's one request per job, so this is where most of the time goes.

4. "30+ Days Ago" is a floor, not a date. If you convert relative dates to absolute ones, anything older than 30 days collapses into "30+". Track the first time you saw each job if you need real ages.

5. Salary is in the text. Where pay transparency laws apply, the range usually sits inside the description HTML as text ("$139,000 - $257,550"), not in a field. You'll need a parser that handles currencies, "k" suffixes and hourly vs. yearly rates.

6. Be a polite client. These are public endpoints, but they're the company's production careers site. Keep concurrency low, back off on 429 and 5xx responses, and respect robots.txt.

If you'd rather not maintain it

Facet splitting, per-job detail fetches, salary parsing, retries and pacing add up, and every company's site is a little different. Disclosure: I packaged all of it as a hosted tool, Workday Jobs Scraper on Apify, and the source is on GitHub. It splits large searches automatically to get past the 2,000 cap (in testing it collected 2,718 of 2,718 NVIDIA jobs), takes many companies in one run, and returns one clean row per job.

The input:

{
  "startUrls": [
    { "url": "https://adobe.wd5.myworkdayjobs.com/external_experienced" }
  ],
  "searchText": "software engineer",
  "maxJobs": 2
}

One row of real output from that run (description trimmed):

{
  "company": "Adobe",
  "title": "Sr. Software Engineer",
  "jobId": "R165261",
  "location": "San Jose",
  "allLocations": ["San Jose"],
  "country": "United States of America",
  "countryCode": "US",
  "remote": false,
  "timeType": "Full time",
  "postedDate": "2026-09-18",
  "postedText": "Posted 30+ Days Ago",
  "postedDaysAgo": 30,
  "postedDaysAgoIsMinimum": true,
  "jobUrl": "https://adobe.wd5.myworkdayjobs.com/external_experienced/job/San-Jose/Sr-Software-Engineer_R165261",
  "applyUrl": "https://adobe.wd5.myworkdayjobs.com/external_experienced/job/San-Jose/Sr-Software-Engineer_R165261/apply",
  "salaryText": "$139,000 - $257,550",
  "salaryMin": 139000,
  "salaryMax": 257550,
  "salaryCurrency": "USD",
  "salaryPeriod": "year",
  "descriptionText": "Our Team\n\nJoin the Adobe Journey Optimizer B2B Insights and Analytics team...",
  "ats": "workday"
}

Note postedDaysAgoIsMinimum: true. That's gotcha #4 made explicit, so you don't treat "30+" as exactly 30.

Useful filters, all applied before you're charged: titleKeywords / excludeTitleKeywords, locationKeywords, postedWithinDays, remoteOnly, and maxJobs / maxJobsPerSite. Set includeDetails: false for a fast title, location and link list without opening each posting.

Call it from Python

import os
from apify_client import ApifyClient  # pip install apify-client (v3)

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/workday-jobs-scraper").call(
    run_input={
        "startUrls": [{"url": "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"}],
        "searchText": "data engineer",
        "postedWithinDays": 14,
        "maxJobs": 3,
    },
    max_total_charge_usd=0.05,
)
for job in client.dataset(run.default_dataset_id).iterate_items():
    print(job["title"], "|", job["location"], "|", job["postedDate"])

Run against NVIDIA, that printed jobs like Senior Data Engineer | US, CA, Remote | 2026-10-19, with allLocations listing all three locations instead of "3 Locations". (On 1.x versions of the client, use run["defaultDatasetId"].)

Let an AI agent pull the jobs (MCP)

If you use Claude, Cursor, VS Code or another MCP client, you can skip the code. Add Apify's hosted MCP server with just this tool enabled:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=swillz/workday-jobs-scraper"
    }
  }
}

It signs you in to Apify with OAuth on first connect (or send Authorization: Bearer <APIFY_TOKEN>). Then ask something like:

"Find data engineering jobs posted in the last 14 days on NVIDIA's and Adobe's Workday sites, remote or in California, and give me a table with title, location, pay range and apply link."

The agent turns that into startUrls, searchText, postedWithinDays: 14 and locationKeywords: ["Remote", "CA"]. Tips for agent use:

Keep runs short. For a quick answer, have the agent set maxJobs to 50–200 and includeDetails: false. The MCP call-actor tool waits at most 45 seconds before handing back a run ID, so smaller runs come back in one turn.
Cap spend. Pass maxTotalChargeUsd in the call options.
Empty means explained. When a site returns nothing, you get a free status row such as "N jobs listed, 0 matched your filters" or "not_found" instead of a silent empty list, so the agent can fix the filter or the URL rather than guess.

A nice pattern is a daily schedule plus postedWithinDays: 1 across a list of target companies, piped into Slack or a spreadsheet: a personal "new jobs at companies I care about" feed.

Wrap-up

Workday career sites expose POST /wday/cxs/{tenant}/{site}/jobs; no browser needed.
Pages are capped at 20 results and searches at 2,000. Split by facets and de-duplicate to get everything.
The list view is a summary; full locations, descriptions and pay come from each job's detail JSON.
Treat "30+ Days Ago" as a lower bound, and parse salary out of the description text.
Go slowly, back off on errors, and respect the site's rules.

Happy (polite) scraping.

How to extract all URLs from a sitemap (and export them to CSV)

William Jin — Fri, 23 Oct 2026 20:37:46 +0000

You need a list of every page on a website. Maybe you're planning an SEO audit or a site migration. Maybe you want to check for broken links, or you're feeding pages into a crawler or an LLM pipeline. The fastest route is usually the XML sitemap: the site already publishes a list of its URLs for search engines.

The catch is that real sitemaps are rarely one tidy file. They're split into indexes, gzipped, capped at 50,000 URLs each, and sometimes hidden somewhere unexpected. This post covers how to find a sitemap and pull the URLs out by hand. Then come short Python and Node scripts that handle nested indexes and gzip, followed by the pitfalls that tend to bite.

Step 1: Find the sitemap

Try these in order:

robots.txt. Open https://example.com/robots.txt and look for lines starting with Sitemap:. This is the most reliable source, and many sites list several sitemaps here.
The usual paths. /sitemap.xml, /sitemap_index.xml, /sitemap-index.xml, /wp-sitemap.xml (WordPress 5.5+), /sitemap.xml.gz.
The CMS defaults. Shopify uses /sitemap.xml with child sitemaps like sitemap_products_1.xml. Yoast (WordPress) uses /sitemap_index.xml.
Search Console, if it's your own site, lists the sitemaps you've submitted.

A sitemap comes in one of two shapes:

<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <url>
    <loc>https://example.com/about</loc>
    <lastmod>2026-09-14</lastmod>
  </url>
</urlset>

<sitemapindex xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
  <sitemap><loc>https://example.com/sitemap-posts.xml</loc></sitemap>
  <sitemap><loc>https://example.com/sitemap-pages.xml.gz</loc></sitemap>
</sitemapindex>

If you get a sitemapindex, the URLs you want are one level deeper, or sometimes two.

Step 2: Manual methods (small sites)

For a single, small, uncompressed sitemap you don't need code at all.

Google Sheets. In a blank cell:

=IMPORTXML("https://example.com/sitemap.xml", "//*[local-name()='loc']")

The local-name() trick sidesteps the XML namespace, which otherwise makes a plain //loc return nothing. Then use File → Download → CSV. Sheets times out on big files and can't follow indexes or open .gz, so this only works for small sites.

Command line. A quick and dirty version:

curl -s https://example.com/sitemap.xml \
  | grep -oE '<loc>[^<]+</loc>' \
  | sed -E 's#</?loc>##g' > urls.csv

This is fine for a one-off. It breaks on CDATA-wrapped URLs, on entities like &, and on indexes. For a gzipped file, add | gunzip after curl (or use curl --compressed if the server only gzips the transfer).

Browser. Open the sitemap, press Ctrl+F, and copy. That works for 30 URLs. It doesn't work for 30,000.

Step 3: A script that handles indexes and gzip

Here's a small Python version using only requests and the standard library. It follows nested indexes, decompresses .gz files, de-duplicates URLs, and writes a CSV.

import csv, gzip, sys
import xml.etree.ElementTree as ET
import requests

HEADERS = {"User-Agent": "sitemap-extractor/1.0 (+https://example.com/contact)"}

def fetch_xml(url):
    r = requests.get(url, headers=HEADERS, timeout=30)
    r.raise_for_status()
    body = r.content
    if body[:2] == b"\x1f\x8b":            # gzip magic bytes, regardless of extension
        body = gzip.decompress(body)
    return ET.fromstring(body)

def local(tag):                             # strip "{namespace}"
    return tag.rsplit("}", 1)[-1]

def extract(url, seen_maps=None, depth=0):
    seen_maps = seen_maps if seen_maps is not None else set()
    if url in seen_maps or depth > 5:       # guard against loops
        return
    seen_maps.add(url)
    root = fetch_xml(url)
    kind = local(root.tag)
    for entry in root:
        fields = {local(c.tag): (c.text or "").strip() for c in entry}
        loc = fields.get("loc")
        if not loc:
            continue
        if kind == "sitemapindex":
            yield from extract(loc, seen_maps, depth + 1)
        else:
            yield {"url": loc, "lastmod": fields.get("lastmod", ""), "sitemap": url}

if __name__ == "__main__":
    start = sys.argv[1]
    seen = set()
    with open("urls.csv", "w", newline="") as f:
        w = csv.DictWriter(f, fieldnames=["url", "lastmod", "sitemap"])
        w.writeheader()
        for row in extract(start):
            if row["url"] not in seen:
                seen.add(row["url"])
                w.writerow(row)
    print(f"{len(seen)} unique URLs written to urls.csv")

Run it with python sitemap_urls.py https://example.com/sitemap_index.xml.

As a sanity check, I pointed it at https://docs.apify.com/sitemap.xml, which is a sitemap index with six child sitemaps. It wrote 4,082 unique URLs in a few seconds (the count will drift as the docs change).

The same idea in Node 18+ (it uses the built-in fetch), with fast-xml-parser as the only dependency (npm i fast-xml-parser, and save the file as .mjs so top-level await works):

import { XMLParser } from 'fast-xml-parser';
import { gunzipSync } from 'node:zlib';
import { writeFileSync } from 'node:fs';

const parser = new XMLParser({ removeNSPrefix: true });
const asArray = (x) => (x ? (Array.isArray(x) ? x : [x]) : []);

async function extract(url, out = new Map(), seen = new Set()) {
  if (seen.has(url)) return out;
  seen.add(url);
  let buf = Buffer.from(await (await fetch(url)).arrayBuffer());
  if (buf[0] === 0x1f && buf[1] === 0x8b) buf = gunzipSync(buf);
  const doc = parser.parse(buf.toString('utf8'));
  for (const s of asArray(doc.sitemapindex?.sitemap)) await extract(String(s.loc).trim(), out, seen);
  for (const u of asArray(doc.urlset?.url)) out.set(String(u.loc).trim(), u.lastmod ?? '');
  return out;
}

const urls = await extract(process.argv[2]);
const csv = ['url,lastmod', ...[...urls].map(([u, m]) => `"${u.replace(/"/g, '""')}",${m}`)].join('\n');
writeFileSync('urls.csv', csv);
console.log(`${urls.size} URLs written to urls.csv`);

Both versions check the gzip magic bytes rather than trusting the .gz extension. That matters more than you'd expect: some servers send .xml.gz files already decompressed, and some send gzipped bytes under a plain .xml name.

Nested sitemap indexes, in practice

The sitemaps.org protocol says an index should only list urlsets, not other indexes. In the wild you'll still find indexes that point to indexes, especially on big e-commerce and news sites that shard by date or category. So:

Recurse, but cap the depth (5 is plenty).
Track the sitemaps you've visited. Circular references happen, usually when an index lists itself.
Expect volume. One file can hold up to 50,000 URLs and 50 MB uncompressed, and an index can list 50,000 sitemaps. Large sites reach millions of URLs, so stream rows to disk instead of building one giant list in memory if you're going big.
Be polite. Fetch child sitemaps a few at a time, not hundreds in parallel, and back off on 429 or 503 responses.

Pitfalls

The XML namespace. root.findall("url") returns nothing because the real tag is {http://www.sitemaps.org/schemas/sitemap/0.9}url. Strip the namespace as the snippets do, or pass a namespace map.

Whitespace and CDATA. <loc> values are often padded with newlines, or wrapped in <![CDATA[ ... ]]>. Strip them. A real XML parser handles CDATA for you, and regex doesn't.

Escaped characters. & in a <loc> is a literal & in the URL. Parsers decode it, and grep doesn't.

HTML instead of XML. Some sites return a 200 with an HTML "not found" page, or a styled HTML view of the sitemap. Check that the response actually starts with <?xml or <urlset/<sitemapindex before you parse.

Bot protection. Cloudflare and similar services sometimes challenge plain HTTP clients even on /sitemap.xml. Send a descriptive User-Agent, and respect the site's robots.txt and terms. If you're blocked, don't try to force your way through.

Text and RSS sitemaps. The protocol also allows plain-text sitemaps (one URL per line) and RSS/Atom feeds. If the parser fails, look at the first bytes of the file.

Stale sitemaps. A sitemap is what the site says it has, not what's actually live. For an audit, follow up by checking status codes. And lastmod is notoriously unreliable on some CMSs, which stamp every URL with the generation time.

Duplicates and variants. The same page can appear in several sitemaps, or with and without a trailing slash, www, or tracking parameters. Decide on a normalization rule before you de-duplicate.

CSV quoting. URLs can contain commas. Use a real CSV writer (as the Python version does) or quote every field.

If you'd rather not maintain a script

The scripts above cover most sites. Once you're doing this regularly, or across many domains, the edge cases add up: discovery through robots.txt, retries and backoff, huge indexes, text sitemaps, and filtering. Disclosure: I packaged all of that as a hosted tool, Sitemap URL Extractor on Apify, and the source is on GitHub.

You give it a domain or a sitemap URL. It discovers the sitemaps, follows nested indexes and .gz files, and returns one row per URL with lastmod, changefreq, priority and the sitemap it came from. Optional extras: include/exclude patterns, a lastmod date window (2026-01-01 or 7 days), an HTTP status check for every URL (to find 404s and redirects), and an "only new URLs since the last run" mode for scheduled monitoring.

The input is plain JSON:

{
  "startUrls": [{ "url": "https://docs.apify.com" }],
  "includeUrlPatterns": ["*/academy/*"],
  "maxUrls": 1000
}

maxUrls defaults to 1,000 so an accidental run on a huge site stays small. Set it to 0 for no limit.

From the command line (no SDK)

Apify's run-sync-get-dataset-items endpoint runs the tool and streams the results back in one request. This returns CSV directly:

curl -s -X POST \
  "https://api.apify.com/v2/acts/swillz~sitemap-url-extractor/run-sync-get-dataset-items?format=csv&fields=url,lastmod,sitemapUrl&maxTotalChargeUsd=0.50" \
  -H "Authorization: Bearer $APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"startUrls":[{"url":"https://docs.apify.com"}],"includeUrlPatterns":["*/academy/*"],"maxUrls":3}'

Output (trimmed):

"url","lastmod","sitemapUrl"
"https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actor-description",,"https://docs.apify.com/sitemap_base.xml"
"https://docs.apify.com/academy/actor-marketing-playbook/actor-basics/actors-and-emojis",,"https://docs.apify.com/sitemap_base.xml"

Note the empty lastmod: that sitemap simply doesn't publish one, which is common. maxTotalChargeUsd is a hard cap on what the run can cost you.

From Python

With the official client (pip install apify-client, tested with v3):

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/sitemap-url-extractor").call(
    run_input={
        "startUrls": [{"url": "https://docs.apify.com"}],
        "includeUrlPatterns": ["*/academy/*"],
        "maxUrls": 5,
    },
    max_total_charge_usd=0.05,
)
for item in client.dataset(run.default_dataset_id).iterate_items():
    print(item["url"], item.get("lastmod"))

On older 1.x versions of the client, call() returns a dict, so use run["defaultDatasetId"] instead.

Using it from an AI agent (MCP)

If you work with Claude, Cursor, VS Code or another MCP-capable client, you don't need to write any of the code above. Apify runs a hosted MCP server that exposes Store tools to agents. To give an agent just this one tool, add the server with a tools filter:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=swillz/sitemap-url-extractor"
    }
  }
}

On first connect it opens a browser window to sign in to Apify (OAuth). You can also pass an Authorization: Bearer <APIFY_TOKEN> header instead. Then you can ask in plain language:

"List every URL under /blog/ in example.com's sitemap that changed in the last 30 days, and flag any that return 404."

The agent maps that to startUrls, includeUrlPatterns: ["*/blog/*"], lastmodFrom: "30 days" and checkUrlStatus: true. The input fields carry descriptions and examples written for exactly this, so agents tend to get the call right the first time.

A few details that help in agent workflows:

Bounded by default. The 1,000-URL default keeps an agent from accidentally pulling a million-URL site. For run limits, the MCP call-actor tool also accepts maxItems and maxTotalChargeUsd in its call options.
No silent empties. If a site has no sitemap, or your filters matched nothing, the run says why (for example "0 results: No sitemap found for https://example.com in robots.txt or at /sitemap.xml, /sitemap_index.xml, /wp-sitemap.xml and similar paths. Pass the sitemap URL directly if you know it.") instead of returning an empty list that the agent has to guess about.
Results by ID. call-actor returns the run and dataset IDs; the agent reads rows with get-dataset-items, which supports limit, offset and field selection, so it can page through large results instead of stuffing everything into context.

If you're building your own agent with LangChain, LlamaIndex or plain function calling, the curl call above is a good tool definition: one POST in, rows out.

Wrap-up

Start at robots.txt, then try the common paths.
Handle both urlset and sitemapindex, and recurse with a depth limit.
Detect gzip by magic bytes, not by file extension.
Strip namespaces and whitespace, and use a real XML parser and a real CSV writer.
Treat the sitemap as a claim, not the truth, and verify what matters.
For recurring jobs or AI agents, a hosted tool with sane limits saves you from maintaining the edge cases.

Happy crawling, and be nice to the servers you fetch from.