DEV Community

William Jin
William Jin

Posted on

How to scrape Workday job postings to JSON (and get past the 2,000-job limit)

#python #webscraping #tutorial #api

A huge number of large employers host their careers pages on Workday. You can spot them by the address: something.wd5.myworkdayjobs.com/SiteName. If you're building a job board, tracking hiring at target companies, doing labor-market research, or just want an alert when a company posts a role you care about, you'll want those postings as structured data, not HTML.

The good news is that Workday's job search page is a single-page app backed by a JSON API, so you don't need a headless browser. The less good news is a handful of quirks, the main one being that search results stop at 2,000. This post walks through the API, a short Python scraper, the gotchas, and a hosted option you can also hand to an AI agent.

Step 1: Decode the career-site URL

Take NVIDIA's careers site:

https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite
         └─tenant─┘└host┘                └──────site──────┘
Enter fullscreen mode Exit fullscreen mode
  • Host: nvidia.wd5.myworkdayjobs.com (the wd1, wd3, wd5... part varies by company)
  • Tenant: nvidia
  • Site: NVIDIAExternalCareerSite

Language prefixes like /en-US/ in front of the site name are optional. Some companies use wd3.myworkdaysite.com/recruiting/<tenant>/<site> instead; same idea, different shape.

Step 2: Call the jobs endpoint

The search page posts to:

POST https://{host}/wday/cxs/{tenant}/{site}/jobs
Content-Type: application/json

{"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "data engineer"}
Enter fullscreen mode Exit fullscreen mode

The response has total, a page of jobPostings (each with title, locationsText, postedOn and externalPath), and facets, the filter options you see in the sidebar.

Here's a minimal scraper using requests:

import requests

def workday_jobs(tenant, host, site, search="", limit=20, max_jobs=100):
    url = f"https://{host}/wday/cxs/{tenant}/{site}/jobs"
    offset, jobs = 0, []
    while len(jobs) < max_jobs:
        r = requests.post(url, json={"appliedFacets": {}, "limit": limit,
                                     "offset": offset, "searchText": search}, timeout=30)
        r.raise_for_status()
        data = r.json()
        page = data.get("jobPostings", [])
        if not page:
            break
        for p in page:
            jobs.append({
                "title": p.get("title"),
                "location": p.get("locationsText"),
                "posted": p.get("postedOn"),
                "url": f"https://{host}/{site}{p.get('externalPath', '')}",
            })
        offset += limit
        if offset >= data.get("total", 0):
            break
    return jobs[:max_jobs]

jobs = workday_jobs("nvidia", "nvidia.wd5.myworkdayjobs.com",
                    "NVIDIAExternalCareerSite", search="data engineer", max_jobs=5)
for j in jobs:
    print(j)
Enter fullscreen mode Exit fullscreen mode

When I ran it, the first rows looked like this:

{'title': 'Senior Data Engineer', 'location': '3 Locations', 'posted': 'Posted 4 Days Ago',
 'url': 'https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/US-CA-Remote/Senior-Data-Engineer_JR2027678'}
{'title': 'Senior Data Center Engineer', 'location': 'Israel, Tel Aviv', 'posted': 'Posted 30+ Days Ago',
 'url': 'https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite/job/Israel-Tel-Aviv/Senior-Data-Center-Engineer_JR2023708'}
Enter fullscreen mode Exit fullscreen mode

That's a working scraper in about 30 lines. Now for the parts that bite.

The gotchas

1. Search results stop at 2,000. Ask NVIDIA's endpoint for all jobs with an empty search and total comes back as exactly 2000, even though the company has more open roles than that. Paging past offset 2,000 gets you nothing. To get everything, split the search into smaller slices with appliedFacets (the facet IDs come from the facets array in the response; on NVIDIA's site they include jobFamilyGroup, timeType, workerSubType and locationMainGroup). Query each slice, keep splitting any slice that still reports 2,000, and de-duplicate by job ID at the end, because one job can appear in several slices.

2. limit maxes out at 20. Ask for "limit": 21 and you get an HTTP 400. Pages of 20 it is, so be patient and pace your requests.

3. The list view is a summary. locationsText says "3 Locations" rather than listing them, and postedOn is relative text ("Posted Yesterday", "Posted 30+ Days Ago"). For full locations, the description, time type and pay range, fetch each job's detail JSON at https://{host}/wday/cxs/{tenant}/{site}{externalPath}. That's one request per job, so this is where most of the time goes.

4. "30+ Days Ago" is a floor, not a date. If you convert relative dates to absolute ones, anything older than 30 days collapses into "30+". Track the first time you saw each job if you need real ages.

5. Salary is in the text. Where pay transparency laws apply, the range usually sits inside the description HTML as text ("$139,000 - $257,550"), not in a field. You'll need a parser that handles currencies, "k" suffixes and hourly vs. yearly rates.

6. Be a polite client. These are public endpoints, but they're the company's production careers site. Keep concurrency low, back off on 429 and 5xx responses, and respect robots.txt.

If you'd rather not maintain it

Facet splitting, per-job detail fetches, salary parsing, retries and pacing add up, and every company's site is a little different. Disclosure: I packaged all of it as a hosted tool, Workday Jobs Scraper on Apify, and the source is on GitHub. It splits large searches automatically to get past the 2,000 cap (in testing it collected 2,718 of 2,718 NVIDIA jobs), takes many companies in one run, and returns one clean row per job.

The input:

{
  "startUrls": [
    { "url": "https://adobe.wd5.myworkdayjobs.com/external_experienced" }
  ],
  "searchText": "software engineer",
  "maxJobs": 2
}
Enter fullscreen mode Exit fullscreen mode

One row of real output from that run (description trimmed):

{
  "company": "Adobe",
  "title": "Sr. Software Engineer",
  "jobId": "R165261",
  "location": "San Jose",
  "allLocations": ["San Jose"],
  "country": "United States of America",
  "countryCode": "US",
  "remote": false,
  "timeType": "Full time",
  "postedDate": "2026-09-18",
  "postedText": "Posted 30+ Days Ago",
  "postedDaysAgo": 30,
  "postedDaysAgoIsMinimum": true,
  "jobUrl": "https://adobe.wd5.myworkdayjobs.com/external_experienced/job/San-Jose/Sr-Software-Engineer_R165261",
  "applyUrl": "https://adobe.wd5.myworkdayjobs.com/external_experienced/job/San-Jose/Sr-Software-Engineer_R165261/apply",
  "salaryText": "$139,000 - $257,550",
  "salaryMin": 139000,
  "salaryMax": 257550,
  "salaryCurrency": "USD",
  "salaryPeriod": "year",
  "descriptionText": "Our Team\n\nJoin the Adobe Journey Optimizer B2B Insights and Analytics team...",
  "ats": "workday"
}
Enter fullscreen mode Exit fullscreen mode

Note postedDaysAgoIsMinimum: true. That's gotcha #4 made explicit, so you don't treat "30+" as exactly 30.

Useful filters, all applied before you're charged: titleKeywords / excludeTitleKeywords, locationKeywords, postedWithinDays, remoteOnly, and maxJobs / maxJobsPerSite. Set includeDetails: false for a fast title, location and link list without opening each posting.

Call it from Python 

import os
from apify_client import ApifyClient  # pip install apify-client (v3)

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("swillz/workday-jobs-scraper").call(
    run_input={
        "startUrls": [{"url": "https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite"}],
        "searchText": "data engineer",
        "postedWithinDays": 14,
        "maxJobs": 3,
    },
    max_total_charge_usd=0.05,
)
for job in client.dataset(run.default_dataset_id).iterate_items():
    print(job["title"], "|", job["location"], "|", job["postedDate"])
Enter fullscreen mode Exit fullscreen mode

Run against NVIDIA, that printed jobs like Senior Data Engineer | US, CA, Remote | 2026-10-19, with allLocations listing all three locations instead of "3 Locations". (On 1.x versions of the client, use run["defaultDatasetId"].)

Let an AI agent pull the jobs (MCP)

If you use Claude, Cursor, VS Code or another MCP client, you can skip the code. Add Apify's hosted MCP server with just this tool enabled:

{
  "mcpServers": {
    "apify": {
      "url": "https://mcp.apify.com?tools=swillz/workday-jobs-scraper"
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

It signs you in to Apify with OAuth on first connect (or send Authorization: Bearer <APIFY_TOKEN>). Then ask something like:

"Find data engineering jobs posted in the last 14 days on NVIDIA's and Adobe's Workday sites, remote or in California, and give me a table with title, location, pay range and apply link."

The agent turns that into startUrls, searchText, postedWithinDays: 14 and locationKeywords: ["Remote", "CA"]. Tips for agent use:

  • Keep runs short. For a quick answer, have the agent set maxJobs to 50–200 and includeDetails: false. The MCP call-actor tool waits at most 45 seconds before handing back a run ID, so smaller runs come back in one turn.
  • Cap spend. Pass maxTotalChargeUsd in the call options.
  • Empty means explained. When a site returns nothing, you get a free status row such as "N jobs listed, 0 matched your filters" or "not_found" instead of a silent empty list, so the agent can fix the filter or the URL rather than guess.

A nice pattern is a daily schedule plus postedWithinDays: 1 across a list of target companies, piped into Slack or a spreadsheet: a personal "new jobs at companies I care about" feed.

Wrap-up

  • Workday career sites expose POST /wday/cxs/{tenant}/{site}/jobs; no browser needed.
  • Pages are capped at 20 results and searches at 2,000. Split by facets and de-duplicate to get everything.
  • The list view is a summary; full locations, descriptions and pay come from each job's detail JSON.
  • Treat "30+ Days Ago" as a lower bound, and parse salary out of the description text.
  • Go slowly, back off on errors, and respect the site's rules.

Happy (polite) scraping.

Top comments (0)