DEV Community: Dodo Data

5,966 job posts: who actually publishes pay, and who only looks like they don't

Dodo Data — Mon, 21 Sep 2026 05:57:21 +0000

Every applicant-tracking system has a structured pay field. Greenhouse calls it pay_input_ranges, Ashby has compensation, Lever has salaryRange. If you are building anything on job data, that field is where you look for salary, and it is where most pipelines stop.

I pulled 24 public career sites, 5,966 open jobs, and counted. Then I read the descriptions of the boards that came back empty, and the number moved by 445 jobs.

Two of the 24 boards had no open jobs that day, so every count below comes from the 22 that did. The per-board numbers are published as open data at the end of this post, so you can check any of it.

The structured field

One request per board, no browser, no login. Greenhouse at boards-api.greenhouse.io/v1/boards/<board>/jobs, Ashby at api.ashbyhq.com/posting-api/job-board/<org>?includeCompensation=true, Lever at api.lever.co/v0/postings/<org>?mode=json.

System	Boards	Jobs	With structured pay
Ashby	8	1,293	72.8%
Greenhouse	12	3,435	56.2%
Lever	4	1,238	3.2%

Lever's 3.2% is the first thing worth noticing. It is not that Lever companies pay in secret. salaryRange exists on the Lever posting API, and almost nobody fills it in. Three of the four boards I pulled had it empty on every single job.

Per company, the spread inside one system is wider than the gap between systems:

Board	System	Jobs	Structured pay
Duolingo	Greenhouse	81	100.0%
Coinbase	Greenhouse	217	98.2%
Ramp	Ashby	148	95.3%
Anthropic	Greenhouse	611	89.4%
OpenAI	Ashby	816	82.4%
Databricks	Greenhouse	874	54.1%
Stripe	Greenhouse	665	0.0%
Palantir	Lever	313	0.0%
Notion	Ashby	128	0.0%
Discord	Greenhouse	47	0.0%

The zeroes are not all the same zero

Four of those 0.0% boards state a pay range in plain English in the job description. They just never filled in the field.

I ran a range extractor over the descriptions of every board that reported zero:

Board	Structured	Found in description text
Discord	0 of 47	44
Palantir	0 of 313	200
Notion	0 of 128	79
Affirm	0 of 199	121
Stripe	0 of 665	1
Spotify	0 of 71	0
Linear	0 of 32	0

Discord publishes pay on 94% of its jobs. Any dataset built on the structured field alone reports 0%.

That is the finding: an empty pay field is a data-entry fact, not a pay-transparency fact. Stripe and Spotify genuinely do not publish ranges. Discord, Palantir, Notion and Affirm do, in a sentence near the bottom of the post that no API returns as a number.

Across the whole sample, the structured field says 48.8% of jobs publish pay. Reading the text as well takes it to 56.3%, and the gain is concentrated entirely in the boards that looked silent.

Extracting a range from prose, without inventing numbers

The naive version of this is a regex for two dollar amounts with a dash between them, and it is wrong within about ten jobs. Here is what actually breaks.

Equity tables look exactly like salary ranges. Affirm puts this in most of its posts:

Equity Grade - 2 (awarded in US Dollars)
Equity grade: 2   New hire equity: $16,000-$24,000   Annual Refresh: $5,000

$16,000-$24,000 is a real range next to a real dollar sign, and it is not the salary. A blocklist on the preceding text is what saves you: equity, refresh, bonus, sign-on, 401, stipend, RSU, grant, option.

On-target earnings are not base pay. Sales roles quote OTE, which is base plus commission at 100% attainment. Rolling it into a salary_min column silently inflates every sales row in the dataset. I let those go unmatched on purpose, which is most of what the extractor misses on Affirm.

A number needs a cue word in front of it, not just a currency symbol. Requiring one of base pay, pay range, salary range, compensation range, hourly rate and so on within the preceding 90 characters removes nearly all of the false positives without costing real matches.

Hourly and annual are the same shape. $22 - $30 and $220,000 - $300,000 differ by three zeros. Decide the period explicitly from the surrounding words, and refuse to guess when there is neither an hourly nor an annual cue and the top of the range is under $10,000.

The whole thing is about thirty lines:

const MONEY = String.raw`(?:\$|USD\s?|£|€)\s?(\d{1,3}(?:,\d{3})+|\d{2,3}(?:\.\d+)?\s?[kK]\b|\d{2,6}(?:\.\d{2})?)`;
const RANGE = new RegExp(`${MONEY}\\s*(?:-|–|—|to|and)\\s*${MONEY}`, 'g');
const PAY_CUE = /(base pay|pay range|base salary|salary range|compensation range|annual salary|hourly rate|pay rate|salary of|compensation of|remuneration)/i;
const NOT_PAY = /(equity|refresh|bonus|sign[- ]?on|401|stipend|budget|revenue|funding|raise[ds]|valuation|grant|rsu|option)/i;

for (const m of flat.matchAll(RANGE)) {
    const before = flat.slice(Math.max(0, m.index - 90), m.index);
    if (!PAY_CUE.test(before) || NOT_PAY.test(before)) continue;
    // ...amounts, currency from the symbol, period from hourly/annual cues
}

It does not handle 35 000 €, the French spacing convention, and I left it that way rather than half-supporting it. A French board returns null, which is honest, instead of a number pulled out of a format the parser does not really read.

Keep the two sources apart

If you merge text-extracted ranges into the same column as the structured field and say nothing, you have made your dataset less trustworthy, not more. One came from a form the employer filled in. The other came from a regex over prose, and it can be wrong.

Carry a third column. Mine is salary_source, one of structured, description, or null, and a run over Affirm and Duolingo looks like this:

200 rows   structured: 81   description: 94   none: 25
  duolingo   structured: 81
  affirm     description: 94   none: 25
  e.g. Administrative Assistant IV        115,000 - 165,000 USD
  e.g. Affirm Bank Strategic Finance Mgr  185,000 - 245,000 USD

Anyone doing serious analysis can filter to structured only. Anyone building a job board gets a pay range on four times as many Affirm roles. Nobody gets a number without knowing where it came from.

The per-board numbers, as open data

Every board's counts, free and machine-readable, so you do not have to take the tables on trust:

https://api.apify.com/v2/datasets/6EJmXWfWFxmjjCUtl/items?clean=true&format=json
https://api.apify.com/v2/datasets/6EJmXWfWFxmjjCUtl/items?clean=true&format=csv

One row per board: jobs_open, jobs_with_structured_pay, structured_pay_pct, and where I also read the
descriptions, jobs_with_pay_in_description_text with description_text_checked saying so. The two boards with
no open jobs are in there as zeroes rather than quietly dropped, and every row carries the endpoint it came from.

Use it

The extractor ships in the Dodo Data job scrapers on Apify Store, across Greenhouse, Lever, Ashby, Workday and the multi-system one. Same 30-field row shape including salary_source, $1 per 1,000 jobs:

https://apify.com/dododata/career-site-jobs-scraper

If you want to re-run the survey yourself, it is one request per board and the endpoints are all listed above. The interesting number is not the average. It is which of your target employers are in the "publishes pay, never filled in the field" bucket, because that one is invisible until you read the text.

Scraping Workday career sites without a browser, and the 2,000-job ceiling

Dodo Data — Mon, 21 Sep 2026 03:09:50 +0000

Greenhouse, Lever, Ashby and half a dozen other applicant-tracking systems publish a job feed you can fetch in one request. Workday, the one running the career pages of most large enterprises, does not. No /jobs.json, no RSS, and the page itself is an empty JavaScript shell.

It still does not need a browser. Every Workday career site talks to its own backend over a plain POST, and that endpoint answers anyone.

The listing call

POST https://<tenant>.<pod>.myworkdayjobs.com/wday/cxs/<tenant>/<site>/jobs
Content-Type: application/json

{ "appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "" }

Everything you need is in the career-page URL. https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite gives tenant nvidia, pod wd5, site NVIDIAExternalCareerSite. Mind the en-US some sites insert before the site name.

The reply is small and clean:

{
  "total": 2000,
  "jobPostings": [
    {
      "title": "Senior Firmware Engineer – CSP Engagements",
      "externalPath": "/job/US-CA-Santa-Clara/Senior-Firmware-Engineer---CSP-Engagements_JR1999599",
      "locationsText": "US, CA, Santa Clara",
      "postedOn": "Posted Today",
      "bulletFields": ["JR1999599"]
    }
  ]
}

Three things to know before you build on it.

limit caps at 20. Ask for 50 and you get a 400, not a truncated page. A full career site is total / 20 requests, minimum.

total is not the number of jobs. It is 2,000 on NVIDIA and 1,483 on Salesforce, and NVIDIA does not have exactly two thousand openings. 2,000 is the ceiling the site will paginate to. Past it you get empty pages. If a site reports exactly 2,000, assume you are seeing the 2,000 most recent and say so downstream rather than claiming a complete catalogue.

Only the first page carries total. Later pages return it as 0. Carry it forward yourself, or your loop will stop after page two.

The descriptions are in the HTML, as JSON-LD

The listing gives you title, location, date and requisition id, but no description, no country, no employment type. Those sit on each job's public page, and you do not have to parse the DOM for them: every Workday job page carries a schema.org JobPosting block.

const $ = load(html);
let ld: Json = {};
$('script[type="application/ld+json"]').each((_i, el) => {
    const d = JSON.parse($(el).text());
    if (d['@type'] === 'JobPosting') ld = d;
});
// ld.description, ld.datePosted, ld.employmentType,
// ld.jobLocation.address.addressCountry, ld.identifier.value

That is one request per job, which is the whole cost of the job. Filter on the listing first, where title, location and date are all available, and only open what matches. A "remote engineering roles posted this week" run on a 2,000-job site is about 100 listing requests plus a handful of job pages, not 2,000.

"Posted Today" is not a date

postedOn is relative text: Posted Today, Posted Yesterday, Posted 3 Days Ago, Posted 30+ Days Ago. The first three convert cleanly. The last one does not. It is a lower bound, not a date, and turning it into "30 days ago" invents precision the site never gave you:

export function workdayPostedOn(text: string, now = Date.now()): string | null {
    const t = text.toLowerCase();
    let days: number | null = null;
    if (t.includes('today')) days = 0;
    else if (t.includes('yesterday')) days = 1;
    else {
        const m = t.match(/(\d+)(\+?)\s+days?/);
        if (m && !m[2]) days = Number(m[1]);   // "30+" falls through to null
    }
    return days === null ? null : new Date(now - days * 86_400_000).toISOString().slice(0, 10);
}

The JSON-LD on the job page has a real datePosted, so a row that opens the page gets the exact date anyway. The relative text only matters when you are filtering on the listing, which is exactly when you want it to be honest.

robots.txt

https://<tenant>.<pod>.myworkdayjobs.com/robots.txt explicitly allows the career site and disallows /talentcommunity/ and /refreshFacet/. Read it per tenant rather than assuming. It is one request, and it is the difference between a public page and someone's candidate area.

What a real run looks like

500 jobs from one career site, with descriptions: 526 requests, 2 minutes 30 seconds, 4 timeouts that all succeeded on retry. The timeouts are worth planning for, because Workday's backend is slower than a static feed, and a 30-second request timeout with retries is not excessive.

Use it

If you would rather not maintain the pagination, the JSON-LD merge and the relative dates, it is on Apify Store as Workday Jobs Scraper & API. Paste career-site URLs, filter by title keywords, location or "posted within N days" before any job page is opened, $1 per 1,000 jobs:

https://apify.com/dododata/workday-jobs-scraper

Same 26-field shape as the Greenhouse, Lever and Ashby ones, so a multi-employer pipeline does not need a branch per system. If something changes, it is fixed within 48 hours, and that promise is on the listing.

Every Shopify store publishes its whole catalogue at one URL

Dodo Data — Sat, 19 Sep 2026 09:14:35 +0000

If you have ever written a scraper for a Shopify store, you probably parsed the HTML. You did not have to.

Every Shopify storefront serves its own catalogue as JSON at /products.json. It is not a private API and not a leak. It is the storefront's own public endpoint, the same data the shop renders for any visitor:

https://www.allbirds.com/products.json?limit=250&page=1
https://www.allbirds.com/collections/mens/products.json?limit=250&page=1
https://www.allbirds.com/meta.json          store name, currency, country

250 products per page, page=N until you get an empty list. Collections work the same way, so you can scope a run to one part of a shop.

What a product actually contains

The interesting part is not the title, it is variants. Each variant is a real SKU with its own price, its own crossed-out price and its own stock flag:

{
  "id": 7205191974992,
  "title": "Daily Grind Women's Corduroy Workwear Jacket",
  "vendor": "Death Wish Coffee",
  "variants": [
    { "sku": "DWC-JKT-S", "price": "65.00", "compare_at_price": "90.00", "available": true,  "option1": "S" },
    { "sku": "DWC-JKT-M", "price": "65.00", "compare_at_price": "90.00", "available": false, "option1": "M" }
  ]
}

Two things follow from that shape, and both are easy to get wrong.

A discount only means something inside one variant. If you take the lowest price in the product and compare it with the highest compare_at_price in the product, a shop that sells a $10 accessory and a $100 jacket marked down from $120 will report a 92% discount. It is not on sale by 92%; those are two different things being subtracted from each other. Compute the discount per variant, then take the best one:

const discounts = variants
    .filter((v) => v.on_sale && v.price !== null && v.compare_at_price)
    .map((v) => Math.round((1 - v.price / v.compare_at_price) * 100));
const bestDiscount = discounts.length ? Math.max(...discounts) : null;

Stock is per variant too. "In stock" for a product usually means some size is in stock. If you are monitoring availability, one row per variant is the only honest shape.

Prices are strings, and the currency is somewhere else

price arrives as "65.00", a string. /products.json never tells you the currency, which lives in /meta.json alongside the shop's name and country. Fetch it once per store and attach it to every row, or your dataset silently mixes USD, EUR and GBP in one column.

A real run

Two stores (Allbirds and Death Wish Coffee), one run on 2026-09-17: 439 products, 3,282 variants, 9 seconds, 5 requests. 23 products were on sale, 246 had at least one variant in stock, and every row carried the store's currency. The deepest genuine discount was a pair of socks at $4 against a $16 compare-at price, computed inside one variant, which is why it can be trusted.

What the endpoint will not give you

Some shops turn /products.json off or put it behind a bot check; Gymshark, for one, returns 403. Treat that as a normal outcome, log it and move on. It is a per-store decision by the merchant, not a puzzle to defeat. And a site that is not on Shopify simply 404s.

Use it

If you would rather not maintain the pagination, the variant maths and the per-store currency, it is on Apify Store as Shopify Products Scraper & API. Paste store or collection URLs, choose one row per product or per variant, filter to in-stock or on-sale only, $1 per 1,000 rows:

https://apify.com/dododata/shopify-products-scraper

The id field is stable across runs (shopify:store:product:variant), so a daily run plus a diff is a price and stock monitor in about one line of code. If something changes, it is fixed within 48 hours, and that promise is on the listing.

Every open job at 500 companies, as JSON, without scraping a single job board

Dodo Data — Fri, 18 Sep 2026 07:50:29 +0000

Most "job data" projects start by scraping LinkedIn or Indeed, and most of them die there: captchas, logins, lawsuits, and listings that are already three reposts away from the employer.

There is a quieter source. Most tech companies run their career page on an applicant-tracking system: Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee, Workable, Personio, Breezy, Pinpoint, Rippling, Teamtailor. Every one of those systems publishes a job feed so the career page can be embedded on the company's site. First-party, current, no login, no anti-bot.

The feeds

Greenhouse       https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true&pay_transparency=true
Lever            https://api.lever.co/v0/postings/{company}?mode=json
Ashby            https://api.ashbyhq.com/posting-api/job-board/{company}?includeCompensation=true
SmartRecruiters  https://api.smartrecruiters.com/v1/companies/{company}/postings
Recruitee        https://{company}.recruitee.com/api/offers/
Workable         https://apply.workable.com/api/v1/widget/accounts/{company}?details=true
Personio         https://{company}.jobs.personio.de/xml
Breezy HR        https://{company}.breezy.hr/json
Pinpoint         https://{company}.pinpointhq.com/postings.json
Rippling         https://api.rippling.com/platform/api/ats/v1/board/{company}/jobs
Teamtailor       https://{career-site-host}/jobs.rss

{company} is the slug in the career-page URL: jobs.lever.co/spotify gives spotify.

The catch: eleven shapes

The feeds agree on nothing. Greenhouse sends the description as HTML that has been entity-escaped once (<p>), pay ranges in cents, and the location as { "name": ... }. Lever sends createdAt as a millisecond string and splits the description across descriptionPlain, lists[] and additionalPlain. Ashby nests compensation three levels deep. SmartRecruiters paginates and needs one more request per posting for the text. Personio is XML, Teamtailor is RSS with its own namespace, and Rippling's feed has no descriptions at all.

Un-escaping Greenhouse, for instance, is a two-step:

import { load } from 'cheerio';

export function htmlToText(html: unknown): string | null {
    if (!html) return null;
    let h = String(html);
    if (!h.includes('<') && h.includes('&lt;')) h = load(`<x>${h}</x>`)('x').text(); // un-escape once
    h = h.replace(/<\s*br\s*\/?>/gi, '\n').replace(/<\/(p|div|li|h[1-6]|ul|ol|tr)>/gi, '\n').replace(/<li[^>]*>/gi, '• ');
    return load(`<x>${h}</x>`)('x').text().trim() || null;
}

(Workday, the twelfth system, has no feed. Its career sites are allowed by robots.txt and every job page carries schema.org JobPosting as JSON-LD, so it is read the old-fashioned way: listing, then page.)

One shape out

I normalised all twelve into 26 fields: title, department, team, locations[], workplace_type (remote / hybrid / onsite), salary_min, salary_max, salary_currency, posted_at, apply_url, plain-text description, and a stable id (ats:company:job) so a daily run can be diffed in one line.

A real run on 2026-09-17, three boards (Duolingo on Greenhouse, Spotify on Lever, Ramp on Ashby): 304 jobs in 18 seconds, 224 of them (74%) with a published pay range already parsed to numbers.

Two things I deliberately leave out: recruiter names and mailboxes that some feeds carry, and any e-mail or phone number inside a description (replaced with [email removed]). It is job data, not people data.

Use it

If you would rather not maintain twelve parsers, it is on Apify Store as Career Site Jobs Scraper. Paste career-page URLs, filter by title keywords, location or "posted within N days", $1 per 1,000 jobs, callable from the API or from an AI agent over MCP:

https://apify.com/dododata/career-site-jobs-scraper

If a feed changes shape, it is fixed within 48 hours, and that promise is on the listing.