DEV Community

Dodo Data
Dodo Data

Posted on

Scraping Workday career sites without a browser, and the 2,000-job ceiling

#webscraping #api #jobs #typescript

Greenhouse, Lever, Ashby and half a dozen other applicant-tracking systems publish a job feed you can fetch in one request. Workday, the one running the career pages of most large enterprises, does not. No /jobs.json, no RSS, and the page itself is an empty JavaScript shell.

It still does not need a browser. Every Workday career site talks to its own backend over a plain POST, and that endpoint answers anyone.

The listing call 

POST https://<tenant>.<pod>.myworkdayjobs.com/wday/cxs/<tenant>/<site>/jobs
Content-Type: application/json

{ "appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "" }
Enter fullscreen mode Exit fullscreen mode

Everything you need is in the career-page URL. https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite gives tenant nvidia, pod wd5, site NVIDIAExternalCareerSite. Mind the en-US some sites insert before the site name.

The reply is small and clean:

{
  "total": 2000,
  "jobPostings": [
    {
      "title": "Senior Firmware Engineer – CSP Engagements",
      "externalPath": "/job/US-CA-Santa-Clara/Senior-Firmware-Engineer---CSP-Engagements_JR1999599",
      "locationsText": "US, CA, Santa Clara",
      "postedOn": "Posted Today",
      "bulletFields": ["JR1999599"]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Three things to know before you build on it.

limit caps at 20. Ask for 50 and you get a 400, not a truncated page. A full career site is total / 20 requests, minimum.

total is not the number of jobs. It is 2,000 on NVIDIA and 1,483 on Salesforce, and NVIDIA does not have exactly two thousand openings. 2,000 is the ceiling the site will paginate to. Past it you get empty pages. If a site reports exactly 2,000, assume you are seeing the 2,000 most recent and say so downstream rather than claiming a complete catalogue.

Only the first page carries total. Later pages return it as 0. Carry it forward yourself, or your loop will stop after page two.

The descriptions are in the HTML, as JSON-LD

The listing gives you title, location, date and requisition id, but no description, no country, no employment type. Those sit on each job's public page, and you do not have to parse the DOM for them: every Workday job page carries a schema.org JobPosting block.

const $ = load(html);
let ld: Json = {};
$('script[type="application/ld+json"]').each((_i, el) => {
    const d = JSON.parse($(el).text());
    if (d['@type'] === 'JobPosting') ld = d;
});
// ld.description, ld.datePosted, ld.employmentType,
// ld.jobLocation.address.addressCountry, ld.identifier.value
Enter fullscreen mode Exit fullscreen mode

That is one request per job, which is the whole cost of the job. Filter on the listing first, where title, location and date are all available, and only open what matches. A "remote engineering roles posted this week" run on a 2,000-job site is about 100 listing requests plus a handful of job pages, not 2,000.

"Posted Today" is not a date

postedOn is relative text: Posted Today, Posted Yesterday, Posted 3 Days Ago, Posted 30+ Days Ago. The first three convert cleanly. The last one does not. It is a lower bound, not a date, and turning it into "30 days ago" invents precision the site never gave you:

export function workdayPostedOn(text: string, now = Date.now()): string | null {
    const t = text.toLowerCase();
    let days: number | null = null;
    if (t.includes('today')) days = 0;
    else if (t.includes('yesterday')) days = 1;
    else {
        const m = t.match(/(\d+)(\+?)\s+days?/);
        if (m && !m[2]) days = Number(m[1]);   // "30+" falls through to null
    }
    return days === null ? null : new Date(now - days * 86_400_000).toISOString().slice(0, 10);
}
Enter fullscreen mode Exit fullscreen mode

The JSON-LD on the job page has a real datePosted, so a row that opens the page gets the exact date anyway. The relative text only matters when you are filtering on the listing, which is exactly when you want it to be honest.

robots.txt

https://<tenant>.<pod>.myworkdayjobs.com/robots.txt explicitly allows the career site and disallows /talentcommunity/ and /refreshFacet/. Read it per tenant rather than assuming. It is one request, and it is the difference between a public page and someone's candidate area.

What a real run looks like

500 jobs from one career site, with descriptions: 526 requests, 2 minutes 30 seconds, 4 timeouts that all succeeded on retry. The timeouts are worth planning for, because Workday's backend is slower than a static feed, and a 30-second request timeout with retries is not excessive.

Use it

If you would rather not maintain the pagination, the JSON-LD merge and the relative dates, it is on Apify Store as Workday Jobs Scraper & API. Paste career-site URLs, filter by title keywords, location or "posted within N days" before any job page is opened, $1 per 1,000 jobs:

https://apify.com/dododata/workday-jobs-scraper

Same 26-field shape as the Greenhouse, Lever and Ashby ones, so a multi-employer pipeline does not need a branch per system. If something changes, it is fixed within 48 hours, and that promise is on the listing.

Top comments (0)