DEV Community

rj
rj

Posted on

Build a niche job board from company ATS feeds (Greenhouse, Lever, Ashby & Workday)

#webscraping #javascript #api #tutorial

If you're building a niche job board, you don't need to scrape LinkedIn. You probably shouldn't.

Most companies post jobs on their applicant tracking system (ATS) first, and job boards and aggregators pick them up from there. The four ATSs that cover a large share of startups and big employers (Greenhouse, Lever, Ashby and Workday) all serve their public job boards from JSON endpoints. No login, no API key, no headless browser.

I run a small crypto job board, cryptogrind.com, and this is how it gets its listings. Here's what I learned wiring these up, including the parts that bit me.

Greenhouse

Board URLs look like boards.greenhouse.io/figma or job-boards.greenhouse.io/figma. The slug (figma) is all you need:

GET https://boards-api.greenhouse.io/v1/boards/figma/jobs
Enter fullscreen mode Exit fullscreen mode

Add ?content=true to get descriptions. Watch out: the HTML comes back entity-encoded (<p>), so decode it before you render or strip tags.

Useful fields: id, title, location.name, absolute_url, first_published, updated_at, departments, offices.

Lever

Board URLs look like jobs.lever.co/spotify:

GET https://api.lever.co/v0/postings/spotify?mode=json
Enter fullscreen mode Exit fullscreen mode

It's paginated with limit and skip. Companies on Lever's EU instance are served from api.eu.lever.co instead, and their boards live on jobs.eu.lever.co.

Useful fields: id, text (the title), categories.location, categories.team, categories.commitment, workplaceType, hostedUrl, createdAt (milliseconds since epoch), and sometimes a structured salaryRange.

Ashby

Board URLs look like jobs.ashbyhq.com/linear:

GET https://api.ashbyhq.com/posting-api/job-board/linear?includeCompensation=true
Enter fullscreen mode Exit fullscreen mode

Ashby is the nicest of the four: isRemote, workplaceType, employmentType, publishedAt, plain-text descriptions, and real compensation data.

One gotcha that cost me: salary lives in compensation.summaryComponents[], not in a minValue / maxValue pair on the compensation object. My first version read the wrong path and silently got no salaries at all.

Workday

Workday is where the big employers are, and it's the awkward one. Career sites look like nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite. The tenant (nvidia), the wdN host and the site name (NVIDIAExternalCareerSite) all go into the URL:

const res = await fetch(
  'https://nvidia.wd5.myworkdayjobs.com/wday/cxs/nvidia/NVIDIAExternalCareerSite/jobs',
  {
    method: 'POST',
    headers: { 'Content-Type': 'application/json', Accept: 'application/json' },
    body: JSON.stringify({ appliedFacets: {}, limit: 20, offset: 0, searchText: '' }),
  },
);
const { total, jobPostings } = await res.json();
Enter fullscreen mode Exit fullscreen mode

Things to know:

  • limit maxes out at 20, so you page with offset.
  • total only comes back on the first page.
  • Every search is capped at 2,000 results. NVIDIA reports total: 2000 but actually has over 2,700 open roles. To get them all, split the search by facet (job category or location, both returned in the response) and merge.
  • Dates are text: "Posted Today", "Posted 3 Days Ago", "Posted 30+ Days Ago". The real start date is only in the job detail.
  • Descriptions need a second request per job (GET the externalPath under the same /wday/cxs/... prefix), and Workday rate-limits each site to a few requests per second after an initial burst. Back off on 429s.

A minimal multi-ATS fetcher

Here's the core of it for the three simple ATSs, normalized into one shape. Node 18+ has fetch built in:

const sources = {
  greenhouse: async (board) => {
    const res = await fetch(`https://boards-api.greenhouse.io/v1/boards/${board}/jobs`);
    const { jobs } = await res.json();
    return jobs.map((j) => ({
      id: `greenhouse:${board}:${j.id}`,
      title: j.title,
      location: j.location?.name ?? null,
      url: j.absolute_url,
      postedAt: j.first_published ?? j.updated_at,
    }));
  },
  lever: async (company) => {
    const res = await fetch(`https://api.lever.co/v0/postings/${company}?mode=json`);
    const postings = await res.json();
    return postings.map((p) => ({
      id: `lever:${company}:${p.id}`,
      title: p.text,
      location: p.categories?.location ?? null,
      url: p.hostedUrl,
      postedAt: new Date(p.createdAt).toISOString(),
    }));
  },
  ashby: async (board) => {
    const res = await fetch(
      `https://api.ashbyhq.com/posting-api/job-board/${board}?includeCompensation=true`,
    );
    const { jobs } = await res.json();
    return jobs.map((j) => ({
      id: `ashby:${board}:${j.id}`,
      title: j.title,
      location: j.location ?? null,
      url: j.jobUrl,
      postedAt: j.publishedAt,
    }));
  },
};

const boards = [['greenhouse', 'figma'], ['lever', 'spotify'], ['ashby', 'linear']];
const jobs = (await Promise.all(boards.map(([ats, slug]) => sources[ats](slug)))).flat();
console.log(jobs.length, 'jobs'); // 255 when I ran it
Enter fullscreen mode Exit fullscreen mode

That gets you a working feed in an afternoon. The real work is everything after that.

The parts that take longer than you think

Finding which ATS a company uses. Most careers pages link or embed their ATS board, so you can fetch the page and look for greenhouse.io, lever.co, ashbyhq.com or myworkdayjobs.com links. Pages that load jobs with JavaScript won't show them in the raw HTML, and some companies use none of the four.

Normalizing. Every ATS says "remote" differently: a flag, a workplaceType, or just the word in the location string. Employment types, locations and dates all differ too. Pick one schema early and map everything into it.

Salaries. Ashby and some Lever boards give structured pay. Greenhouse mostly doesn't, so you parse ranges like $134.4K – $215K or £50,000–£65,000 per year out of the description text. US pay-transparency laws mean more listings include a range, which makes this worth doing.

Only new jobs. For a job board you want each run to return only jobs you haven't seen. Store the IDs you've seen per board, and only mark a job as seen after you've actually saved it. Otherwise a crashed run loses jobs forever.

Being polite. These are public endpoints, but they're not yours. Keep concurrency low, cache, and back off on 429s.

Or skip all of that

I packaged all of this, including the Workday splitting, salary parsing, ATS auto-detection and the "only new jobs" state, into two Apify Actors:

  • ATS Jobs Scraper: paste Greenhouse, Lever, Ashby or Workday board URLs, or just careers pages, and get one normalized schema back. Turn on Only new jobs and schedule it daily.
  • Workday Jobs Scraper: built just for Workday. It gets past the 2,000-result cap automatically and includes full descriptions.

Both cost $1.50 per 1,000 jobs, with no browser or proxy needed. Jobs filtered out or already seen cost nothing. You can call them straight from your backend:

curl -X POST "https://api.apify.com/v2/acts/rjmacarthy~ats-jobs-scraper/run-sync-get-dataset-items?token=YOUR_APIFY_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"companies":["https://boards.greenhouse.io/stripe","https://jobs.ashbyhq.com/ramp"],"onlyNewJobs":true,"stateKey":"my-board"}'
Enter fullscreen mode Exit fullscreen mode

Or schedule a run in Apify and point a webhook at your API to insert new jobs as they appear.

Niche board ideas

A board is only as good as the filter it applies to everyone else's data. A few that are underserved:

  • Roles that offer visa sponsorship
  • Europe-only remote roles, as opposed to "remote, but US only"
  • Jobs at AI startups. A lot of them are on Ashby.
  • One stack or one industry, done properly: Rust, climate, devtools, crypto

If you build something with this, or need a field or ATS I don't cover yet, leave a comment or open an issue on the Actor. I read every one.

I'm RJ, creator of twinny, an open-source AI coding assistant for VS Code.

Top comments (1)

Collapse
 
veristria profile image
StackShortlist •

Going straight to the ATS feeds is so much cleaner than scraping job sites. Nice walkthrough, the normalized fields part was especially useful.