DEV Community: Dodo Data

What is actually in the We Work Remotely RSS feed, and four things that will trip your parser

Dodo Data — Mon, 05 Oct 2026 21:54:34 +0000

We Work Remotely publishes its jobs as RSS, and says so plainly on its RSS page: "Anyone can use the feed, all we ask is that you attribute the links back to We Work Remotely." That makes it one of the few big job boards you can build on without a terms-of-service argument. We built a tool on it at the end of September, and this is what the feed contains once you read all of it, measured on 25 September 2026.

Twelve feeds, 253 jobs

There is one feed of the newest jobs, https://weworkremotely.com/remote-jobs.rss, and one per category, listed on the RSS page. The newest-jobs feed held 80 jobs. All twelve together held 253 distinct jobs, because each category feed reaches further back than the newest-jobs feed does.

The same job appears in several of them. A Full-Stack job is in the newest-jobs feed, in the Full-Stack feed and in the "All Programming" feed, so if you read everything, de-duplicate on the job link. The newest-jobs feed even listed one job twice on the day we looked.

One feed was empty: Front-End Programming had no items at all. An empty feed is a valid answer, not an error, so do not treat zero items as a failure. Treat a response that is not RSS at all as the failure.

An item, and what each field really means

<item>
  <title>Headway: Senior Manager, Content &amp; Community Marketing</title>
  <region>Anywhere in the World</region>
  <country>🇺🇸 United States of America</country>
  <state>California</state>
  <skills>Community Growth, Digital Marketing, and Marketing</skills>
  <category>Sales and Marketing</category>
  <type>Full-Time</type>
  <description>&lt;p&gt;&lt;strong&gt;Headquarters:&lt;/strong&gt; ...</description>
  <pubDate>Fri, 07 Aug 2026 18:48:42 +0000</pubDate>
  <expires_at>Tue, 06 Oct 2026 18:48:42 +0000</expires_at>
  <guid>https://weworkremotely.com/remote-jobs/headway-senior-manager-content-community-marketing</guid>
  <link>https://weworkremotely.com/remote-jobs/headway-senior-manager-content-community-marketing</link>
</item>

title is two fields. It is the company and the job title joined by a colon: "Headway: Senior Manager, Content & Community Marketing". Split at the first ": ", not the last, because job titles contain colons too.

region says almost nothing. It was "Anywhere in the World" on 245 of the 253 jobs, including jobs whose country lists one country. When the two disagree, country is the one the job page shows.

country is a list with flags in it. Several countries come joined by commas and "and":

🇨🇦 Canada and 🇺🇸 United States of America
🇨🇺 Cuba, 🇺🇸 United States of America, and 🇷🇺 Russian Federation
🇬🇧 United Kingdom of Great Britain and Northern Ireland and 🇺🇸 United States of America

Splitting on " and " cuts the United Kingdom in half, and so does Bosnia and Herzegovina. Every country starts with a flag, which is two Unicode regional-indicator characters, so split only where the next flag begins:

const countries = text
    .split(/\s*(?:,|\band\b)?\s*(?=\p{Regional_Indicator}{2})/u)
    .map((c) => c.replace(/\p{Regional_Indicator}/gu, '').replace(/^[,\s]+|[,\s]+$/g, '').trim())
    .filter(Boolean);

state is not where the job is. It looks like a US state for a remote job, but it reads as the employer's address: one job listed France, Japan, Mexico, Norway, Türkiye and Viet Nam as its countries, with the state "California". The job page does not show it at all, which is the best hint about what it is. We leave it out.

description holds two more fields. It is entity-escaped HTML, and it always starts with a paragraph WWR writes itself:

<p><strong>Headquarters:</strong> Bridgetown, Saint Michael, Barbados
<br /><strong>URL:</strong> <a href="http://thehivecareers.co">http://thehivecareers.co</a></p>

That gives you the employer's website on 124 of the 253 jobs, and a headquarters text on 243. Read what the employer typed there as written: for jobs imported from an applicant tracking system it is often a place of work instead, like "REMOTE - US".

skills is filled on about a third of jobs (83 of 253 on 25 September), as a comma list. Most lists fit the job, but some do not: a sales development role lists "Architect, Architecture, Bug Tracking", and a marketing internship lists "Infrastructure Orchestration" and "Endpoint Security". Keep it as its own field and do not search on it, or a search for security engineers will return a marketing intern.

The older feeds are thinner. The "All Programming" feed carries no type, country or expires_at. The newest-jobs feed and the per-category feeds do.

Pay is in the text, and reading it is where we got it wrong

The feed has no pay field. 78 of the 253 descriptions state a range next to a word like Salary or Compensation, so we read it out with the same parser our other job tools use. Then we read all of the results by hand against their text. Two were wrong, and both bugs were in code already running in public tools.

The first was a currency: Compensation: US$65-$85 came out as Singapore dollars. The table of currency codes was checked in order, Singapore's S$ came before US$, and S$ is sitting inside US$. The short symbols now only count as a whole token:

[/SGD|(?<![A-Za-z])S\$/, 'SGD'],

The second was a period: "$42-60,000 USD Annual Salary ($3500-$5000" was read as a yearly salary of 3,500 to 5,000 dollars. The first range was skipped because it was malformed, and the second borrowed the word "Annual" from it. In dollars, pounds or euros no full-time yearly salary is under 10,000, so a range that small is now never called a year's pay. It is left out, because a wrong salary is worse than none.

The run that saved nothing

The first run of the finished tool on Apify returned 0 of 253 rows. The platform rejected every write with "Schema validation failed", because two fields in the dataset schema had been copied from a sibling tool where they are always empty and typed as null, and this one fills them. The feed was read perfectly and nothing was kept.

The fix took a minute. The lesson took a test: rows parsed from real feed items are now checked against the dataset schema before anything is deployed, so a mismatch fails on a laptop instead of on the platform.

The tool

We Work Remotely Jobs Scraper on Apify Store reads all twelve feeds and nothing else, de-duplicates them, and returns each job as JSON in 39 fields, with the countries, the employer's website, pay read out of the text and a link back to the job on weworkremotely.com, which is the one thing WWR asks for. The whole feed took 15 seconds. A monitor mode returns only the jobs you have not had yet. $1 per 1,000 jobs.

It returns the same core fields as our Remote Jobs Search, which reads Himalayas, Remote OK, Jobicy and Arbeitnow, so the two can go in one table.

Our job scraper was nearly 5x slower and costlier because of a proxy it never needed

Dodo Data — Mon, 05 Oct 2026 21:52:27 +0000

We built a tool that takes company names and returns their open jobs: type Databricks, OpenAI, Lyra Health, get their postings as JSON. Behind it is an index of more than 500 companies whose job boards on Greenhouse, Lever and Ashby we had proven belong to them (how, and why guessing gets it wrong half the time, is in an earlier post).

The first runs on the real platform found three problems that no unit test had. All three returned HTTP 200, and all three had been sitting in tools that were already public.

1. The first big company took the whole budget

Input: three companies, 60 jobs. Output: 60 jobs, all from Databricks. OpenAI and Lyra Health got nothing.

Nothing was broken in the usual sense. The boards were read in parallel, and every row that matched was billed and saved until the cap was hit. Databricks has 884 open jobs and answered first, so it filled the 60 before the others had spoken.

If someone names three companies, they expect three companies. The fix is the same one we use when reading four job boards at once: every company gets an equal share of the budget until all of them have answered, and what does not fit waits.

const share = Math.ceil(maxItems / totalBoards) - (savedBy.get(key) ?? 0);
const now = wanted.slice(0, share);
for (const r of wanted.slice(share)) leftovers.push([r, board]);

The first version of the hand-out was a plain round-robin over the held rows, one per company per turn. A test caught it: with two companies and 6 rows it gave 5 to 1, because the big company had already delivered its 3 and still got another turn. The held rows now go to whichever company has received the fewest so far. Three companies, 60 jobs: 20, 20, 20.

2. A 34 MB answer to "list the jobs"

Lever's public postings API returns a company's entire board in one response unless you ask for pages. For most companies that is fine. For TSMG, with 4,334 open jobs, it is 34 MB and 28.6 seconds from a fast connection, past the 30-second request timeout, so the request was retried and failed. The four biggest Lever boards in the index timed out on every run we made.

Lever supports limit and skip. We tried 250 per page first, and it was still wrong: Palantir's descriptions are long enough that 250 postings are 4.8 MB, and six of those downloading at once timed out. At 100 per page every request is under about 2 MB, and a board is read page by page, stopping once it has contributed its share.

One detail if you add paging to anything that tracks what disappeared since the last run: a first page is not the whole board. We record each page, and mark a board complete only on its last page, so a board read part-way never reports its unread jobs as closed.

While in there we checked Lever's robots.txt:

User-agent: *
Allow: /
Crawl-delay: 1

Nothing enforced that. We had a global cap of five requests a second, which is not the same thing when several Lever companies are read in parallel. Now requests are spaced per host, and the slot is reserved before the wait, so parallel requests queue instead of racing:

const at = Math.max(Date.now(), nextSlot.get(host) ?? 0);
nextSlot.set(host, at + gap);
if (at > Date.now()) await sleep(at - Date.now());

3. The proxy we did not need

With the company list empty, the tool searches every company in the index. data engineer plus remote, 60 jobs: correct results, 9 minutes 36 seconds, and more compute than 60 rows earn at $1 per 1,000.

We guessed at the cause twice and were wrong twice.

First guess: concurrency. Reading 20 boards at a time instead of 5 took it from 12.6 minutes to 10.5. Better, not fixed. Second guess: payload size. Greenhouse sends every full description unless you ask it not to, and Databricks alone is 10.4 MB with descriptions and 0.76 MB without. So we changed filtered searches to read the light list, filter on title, place and dates, and open only the matching jobs, 12 KB each. That was worth doing. It saved about a minute.

Then we read the log properly. The average request took 8.1 seconds. The light 0.76 MB lists download in about one second from a laptop. And there were 54 retries.

Every request went through a rotating proxy, because that is the default on the platform and because "use a proxy" is what scraping advice says. But these are public job-board APIs, published by the companies for their own career pages. Nobody was blocking us. One run with the proxy off:

Same search, same code	Through the proxy	Direct
Time	9 min 36 s	2 min 00 s
Compute units	0.160	0.034
Average request	8.1 s	1.3 s
Retries	54	0
429 or 403	n/a	0

The named-company run went from 81 seconds to 6. The same default had been on our public Greenhouse, Lever, Ashby, Workday and multi-system job tools. It is off now, with the switch still there for anyone who does get blocked. On pay-per-event pricing the compute is ours to pay, and one small run of each measured a two-to-eight-fold cut in what a customer run costs us.

What we would tell ourselves a week ago

Measure before you optimise, and measure the thing you are about to change. We spent two rounds tuning concurrency and payloads for a slowdown whose cause was one line in the input defaults, visible in the first log we should have read.

And "use a proxy" is advice for sites that block you. For an API that is published to be read, it is a toll you pay on every request.

The tool

Company Jobs API on Apify Store: type company names, get their open jobs from 510+ companies whose boards are verified, or search all of them at once. 30 fields per job, pay as numbers, a monitor mode that bills only new jobs. $1 per 1,000.

For any company outside the index, the Career Site Jobs Scraper reads a career page on 12 systems by URL; there are also single-system versions for Greenhouse, Lever, Ashby and Workday. Public endpoints only, no personal data in any row.

The same run returned 200 rows or 104, depending on which API answered first

Dodo Data — Tue, 29 Sep 2026 01:34:59 +0000

Four remote job boards publish free JSON feeds: Himalayas, Remote OK, Jobicy and Arbeitnow. No keys. Pulling all four into one shape is a small job, and it produced the most interesting bug I have written this year, because the same code with the same input returned either 200 rows or 104 and the difference was which server answered first.

The setup, which is the part that causes it

The customer asks for 200 rows. Four sources. The obvious implementation is to page each one until you have 200, and the obvious implementation is wrong: Himalayas answers fast and reports more than a hundred thousand jobs, so it fills the entire budget before the other three have finished their handshake. You ask for four boards and you get one board.

So you give each board an equal share of the budget and hold it there until every board has spoken. Once all four have answered, the cap lifts and whoever has rows left can fill the remainder. Fast sources stop crowding out slow ones, and nobody's rows are wasted.

That logic is right. The mistake was in what happens while a board waits.

The bug

const ok = await waitForShare(board);   // polls for 10 seconds
if (!ok) return;                        // caller reads false as "stop paging"

waitForShare polled for ten seconds and returned false. The caller treated false as stop, and dropped its cursor.

The comment above that function said the board "waits rather than abandoning its cursor". It waited, and then it abandoned it. The comment described the intent and the code described the timeout, and nobody had put the two sentences next to each other.

Why it passed the first time

I ran it, got 200 rows of 200, four boards represented, zero failures, and moved on.

That run passed by luck of ordering. Remote OK answered at 08:23:31, five seconds before Himalayas requested its first page at 08:23:36. The share was already open by the time anything could wait on it, so nothing ever waited and the broken branch never ran.

Four hours later, same input, same build:

12:35:28, Himalayas reaches its 50-row share and starts waiting.
12:35:38, ten seconds are up. It gives up and drops its cursor.
12:35:45, Remote OK finally answers on its retry and the share opens.

Seven seconds. The run ended with 104 rows of the 200 asked for, and the status line said 0 failures, because nothing had failed. Everything succeeded. It just stopped early.

That is the shape worth carrying away from this post. A pipeline that fans out to several sources has a failure mode that none of the usual signals can see: one source returning nothing looks exactly like one source having nothing, and the run is a success either way.

Three fixes, and the sizing one is the general lesson

Size the ceiling against the dependency's worst case, not its typical case. Ten seconds was picked from how long Remote OK usually takes. Across four measured runs it answered at 21s, 32s, 84s and 5s, the 84-second one after two retries. A timeout whose entire job is to outlast a slow dependency has to be sized against the tail. I raised it to 90 seconds and that was too thin: the very next run measured an 84-second wait. It held, with six seconds to spare, which is not a margin, it is a coin that landed the right way up. It is 150 now.

The longer ceiling costs a normal run nothing, because the poll is reactive at 250ms rather than sleeping the full duration. If you are tempted to keep a short timeout for safety, notice what that safety buys: it converts a slow run into a quietly incomplete one.

A source that fails for good has to release the gate. The cap lifts when every board has answered, so a board that never answers holds it shut forever and stalls the three that did. The crawler already routed permanently-failed requests to an onFailed hook; this code simply never listened to it. Any barrier that waits on N participants needs a path for a participant that is never going to arrive.

When the wait does expire, warn and name the board. A short run is now never silent about it. This is the cheapest of the three fixes and probably the most valuable, because it turns the next occurrence of any of this into a log line instead of an investigation.

Two smaller ones from the same Actor

Retrying a 429 is answering "slow down" with "no". Arbeitnow answers 429 when you page it quickly. The first version retried four times per run, against a free API, from a shared address. A 429 now stops that board for the run, paging is capped at eight pages, and pages are spaced 1.1 seconds apart.

I did that fix for Arbeitnow only, and left the other three boards retrying, because at the time only Arbeitnow had ever answered 429. Two days later Himalayas answered 429 and was retried four times. When you fix a policy for one participant, check whether it is a policy or a special case. It is now one decision applied to all four, with 429 separated from 403 and 503: a rate limit stops that board, a block stays retryable behind a fresh address.

A timestamp with no zone on it is UTC. Jobicy sends 2026-09-22T03:45:06+00:00, and a parser has to survive the day that timestamp arrives bare, as 2026-09-22 03:45:06. Parsed with the platform default, a bare timestamp is correct on a server running UTC and four hours wrong on my laptop in Mauritius, which means it would have shipped: the CI machine and the production machine would both have agreed, and only the developer's own runs would have been wrong. Caught by a unit test that pinned the expected instant, not by any run.

The obligation that lives in the code

Remote OK asks for a followed link back and a mention as the source. Jobicy asks that application buttons send the user to the original job URL. Both are met by pointing apply_url and url at the board's own job page and carrying source_name on every row.

There is a comment above that line telling the next person not to "improve" apply_url into the employer's own form, because that would look like a better field and would break Jobicy's terms. If you consume a free feed, the terms are part of the schema. Write them where someone editing the field will read them.

The Actor

Remote Jobs Scraper and API on Apify Store: all four boards in one 35-field shape, de-duplicated across them, with UTC offsets where the board publishes them. $1 per 1,000 rows.

Same core fields as the rest of what I publish (this one adds five, timezone offsets among them), so a pipeline over several of them needs no branching: ATS jobs across twelve systems, Greenhouse, Lever, Ashby, Workday. Public endpoints only, contact details stripped from every row.

The paging token an API hands you is not always the one it accepts

Dodo Data — Sun, 27 Sep 2026 02:49:26 +0000

Workable runs the hiring pages for a large number of companies, and the search on jobs.workable.com is served by a public JSON endpoint that its own site calls. No key, no OAuth. One GET gets you a page of jobs across every company on the platform, which at the time I measured it was 168,324 jobs, 48,183 of them remote and 18,048 posted in the last seven days.

Writing the client took an afternoon. Finding out what it was quietly doing wrong took three days, and every one of the four problems below returned HTTP 200 the whole time.

1. The token changes its name in transit

The response ends like this:

{ "jobs": [ ... ], "nextPageToken": "eyJmcm9tIjoyMH0" }

So you send it back:

GET /api/v1/jobs?query=data+engineer&limit=20&nextPageToken=eyJmcm9tIjoyMH0

And you get page one. Again. Forever.

The parameter is only accepted as pageToken. Send it under the name the response gave it and the field is ignored, the request is valid, and you are served the first page with a 200 and a fresh token that also does nothing.

Here is the part worth internalising: a row count cannot see this. Ask for 120 rows at 20 per page and you get 120 rows. Six requests, six pages, no failures, and every number in your log looks correct. What you have is the same twenty jobs six times.

The check that finds it is identity, not volume:

const first = await page(null);
const second = await page(first.nextPageToken);
const overlap = new Set(first.jobs.map(j => j.id));
if (second.jobs.every(j => overlap.has(j.id))) throw new Error('page 2 is page 1');

Any time you page an API you did not write, assert that page two contains an id page one did not. It is two lines and it is the only thing standing between you and a dataset that is one page wide.

2. A filter that is ignored looks exactly like a filter that matched everything

workplace=remote works. remote=true does not exist, and sending it is not an error. You get back the unfiltered corpus, 200, no warning, and if you are searching a term where most jobs happen to be remote you will not notice for a week.

Same family: limit above 20 is a 400, which at least tells you. The silent one is worse than the loud one.

The habit I have settled on is to test every filter by its complement. Run the search with the filter and without it. If the two row counts are identical, the filter did nothing, whatever the documentation says. That takes one extra request and it is the difference between a filter and a decoration.

3. A cursor can hand you the same job twice, and you will bill for it twice

On a 120-row run I got 119 distinct ids. One job was the last row of page 3 and the first row of page 4, identical in every field except the timestamp my own code stamped on it.

This one matters more than it looks, because I sell these rows at a price per row. A duplicate is not an aesthetic problem, it is a customer paying twice for one job. There is no version of that which is acceptable, and a cursor over a corpus that is being written to while you read it will always be able to do it.

The failure in my code was subtle and worth naming, because it is a shape that recurs. I already had a deduplicating guard. It lived on a change tracker, and the change tracker was only constructed when the customer had asked for monitoring:

if (tracker.emittedThisRun(SOURCE, r.id)) continue;

On a plain search there was no tracker, so there was no guard, and the code path that ran most often was the one with no protection at all. The guard now stands on its own, built every run:

const seenIds = new Set();

When I found it I checked the whole catalogue for the same shape rather than only fixing the one Actor, because a guard attached to an optional feature is a guard that is off by default.

4. A field can be filled in the schema and empty in every row

The last one I found this week, two days before the thing shipped.

Two fields in my row, team and experience_level, are hardcoded to null. That is deliberate: this endpoint publishes neither, and the keys stay so that rows from this source have the same shape as rows from the career-page scrapers, where the ATS does publish them. Fine.

What was not fine was how they were described. The schema said:

experience_level: Seniority when the ATS provides it

Read that as a buyer. It says sometimes. It is never. And the sentence did a second, worse thing: I have a script that reports how often each field is actually filled and excludes fields the schema documents as conditional, and "when the ATS provides it" is exactly the phrasing it treats as conditional. The one check I had built to catch hollow fields was being silenced by a sentence that was not true.

Both now read "Always null for this Actor", and say why the key is there.

If you publish data, the coverage of a field is part of its documentation. "String or null" tells the reader nothing. Null on 100% of rows, and here is the reason, tells them everything.

The thread running through all four

None of these four showed up as an error. Every run was a 200 with the right number of rows. What caught them was checking a different thing each time: that page two held new ids, that a filter changed the count, that ids were distinct, that fields were filled.

Status tells you the request worked. It has never once told you the data was right.

The Actor

Jobs API on Apify Store: search company career pages by words, location and remote, with none of the above left as an exercise. 31 fields per job, pay parsed out of the description where the employer writes it in prose, and a monitor mode that returns only jobs that are new or changed since your last run. $1 per 1,000 rows.

The endpoint is public and robots.txt allows /api/, which is what this calls. It disallows the /search HTML pages, which this never requests, and it carries Content-Signal: search=yes, ai-input=yes, ai-train=no, which is respected.

If you would rather reach one company's board directly: Greenhouse, Lever, Ashby, Workday, or the one that detects which system a career page runs on. Public endpoints only, no personal data in any row.

Half the ATS job boards you find by guessing belong to a different company

Dodo Data — Wed, 23 Sep 2026 05:34:52 +0000

If you are building anything on job data, sooner or later you want a map from company name to job board. Greenhouse boards live at boards-api.greenhouse.io/v1/boards/<slug>/jobs, Lever at api.lever.co/v0/postings/<slug>, Ashby at api.ashbyhq.com/posting-api/job-board/<slug>. All three are public, documented, and need no key.

The obvious move is to guess the slug from the company name. Lowercase it, strip the punctuation, try it.

It works about a quarter of the time. And half of what it finds belongs to a different company.

I measured this while building an index across 2,127 companies, and the verification step turned out to matter more than the discovery step.

The naive version, and what it costs you

Take "Airspace Link". Strip it down and you get three candidates: airspacelink, airspace-link, airspace. The third one hits: boards-api.greenhouse.io/v1/boards/airspace/jobs returns 15 open jobs, HTTP 200, real data.

It is not Airspace Link's board.

Here is the same failure four times over, all from one 40-company sample:

Company	slug that returned jobs	actually theirs?
Airspace Link	`greenhouse/airspace`	no
ICON plc	`ashby/icon`	no
Sanctuary Computer	`ashby/sanctuary`	no
Dispatch	`greenhouse/dispatch`	no

Every one is a generic first token that some other company owns. A job index built this way does not merely have gaps. It confidently attributes one employer's open roles to another, and nothing in the response tells you.

Across the first 654: 26% of companies had a findable board, and only 47% of those survived verification.

Verification is harder than it looks, because the three APIs are not equal

The obvious check is to compare the company name against the board's company name. That works on exactly one of the three.

Greenhouse publishes it. GET boards-api.greenhouse.io/v1/boards/<slug> returns:

{"name": "Dispatch Technologies, Inc", "content": "..."}

Ashby does not. The entire payload is {"jobs": [...], "apiVersion": "..."} and a job looks like this:

address, applyUrl, department, descriptionHtml, descriptionPlain,
employmentType, id, isListed, isRemote, jobUrl, location,
publishedAt, secondaryLocations, team, title, workplaceType

No organisation name. No company field. The closest thing is team, which on the board I was checking said "Construction".

Lever does not either. Its fields are text, categories, hostedUrl, applyUrl, description and friends. categories gives you commitment, department, location and team. No company.

So for two of the three systems, the board itself will never tell you whose it is.

The check I wrote first was circular, and passed everything

My first verification compared the company name against the hostedUrl in the response. For jobs.lever.co/jumpcloud/... and company "JumpCloud", that matches.

Of course it matches. The URL contains the slug I just guessed. The test was asking whether the slug I guessed was the slug I guessed.

It reported 8 of 8 confirmed. Every one of the four false positives above sailed through it.

If you are writing this kind of check, the question to ask is: could this test ever fail? If the evidence is derived from the thing you are testing, it cannot.

What actually works: match a job you have already seen

The signal that is available on all three systems is the job list itself. If you already know one real job at that company, from any other source, check whether it is on the board.

def norm(s):
    return re.sub(r"[^a-z0-9]", "", (s or "").lower())

matched = known_titles & {norm(t) for t in board_titles}
if matched:
    ...  # this is their board

That is what caught all four. greenhouse/airspace has 15 jobs and not one of them is the backend role we had seen at Airspace Link.

Two things to know about it:

It gives false negatives, and you must not treat those as proof of anything. greenhouse/shopmy failed the job match, and its board is literally called "ShopMy". The remote board we sourced titles from was listing a role the ATS board no longer carried. So use the name when the ATS gives you one, and fall back to the job match when it does not.

Neither signal firing means you do not know. That is a third state, not a no. Those go in a candidates file, not the index, and a later pass with more known titles for that company can promote one.

Two smaller things that cost me time

api.lever.co publishes Crawl-delay: 1. My first throttle was a single global 0.15s pause, which sent Lever roughly four requests a second against the one second it asks for. Worth checking the exact host you are hitting, not the one that looks similar: jobs.ashbyhq.com/robots.txt disallows /api/, but api.ashbyhq.com is a different host and serves no robots.txt at all.

Filter the company list before you probe it. I measured a 67% hit rate on a sample, then watched it collapse to about 10% on the full run. The sample had been filtered to engineering roles, which is exactly the population that runs an ATS. Most employers on a remote job board are staffing and VA agencies that never had a Greenhouse board to find. Filtering first removed roughly 6,700 requests that were only ever going to 404 on somebody else's API, which is both faster and more polite.

The one that got past all of it

Everything above was in place, and a board named NATIONAL still ended up in the index three times over: as National Church Residences, as National Laboratory of the Rockies, and as National Louis University. Same for a board called Paradigm and one called Public.

The name check was doing containment in both directions. Any board whose name is a common word swallows every company that starts with it. Worse, it re-admitted greenhouse/airspace for Airspace Link, which the job match had already disproved, because "Airspace" is inside "Airspace Link".

Containment is the wrong test. One name has to lead with the other. That single change is the difference between these:

Company	Board name	Same company?	Containment says	Prefix says
Dispatch	`Dispatch Technologies, Inc`	yes	match	match
Domino Data Lab	`Domino`	yes	match	match
Beam	`Bridge to Enter Advanced Mathematics (BEAM)`	no	match	no match
Remote	`General Assembly Remote Jobs`	no	match	no match

Two more things came out of that round:

The slug is evidence on its own. greenhouse/dominodatalab is not a guess that happened to land, it is the company name spelled out. Trusting an exact slug recovered nine boards a stricter pass had wrongly thrown away.

Add a consistency check that does not care how a row was confirmed. A board belongs to one company. If two companies both claim it, at most one is right and you cannot tell which, so demote all of them. That backstop is what caught NATIONAL, and it would have caught it no matter which rule let it in.

The result

264 verified boards, 20,929 open jobs, from 2,127 companies. Every row carries which evidence confirmed it:

{
  "company": "Help Scout",
  "ats": "ashby",
  "slug": "helpscout",
  "open_jobs": 10,
  "evidence": "job_match",
  "confirmed_by": "directorofproductdesign"
}

148 confirmed on the board's own name, 94 on a job match, 13 on both, 9 on an exact slug. 104 companies sit in the candidates file. Final audit: zero boards claimed by two companies, zero short-slug matches outside the company name, zero rows without evidence.

Twelve percent of the companies I started with. The other 88% either have no board on these three systems or could not be proven, and saying so is the entire point.

If you want the job data without building this

The same three endpoints, with the row shape and the ATS handling already done, are on Apify Store:
Greenhouse, Lever, Ashby, Workday, or the multi-system one that auto-detects which system a career page runs on. Public endpoints only, no personal data, $1 per 1,000 jobs.

If you build your own instead, the one thing I would take from this post is the third state. A board you found but cannot prove is not a board you found.

I measured what ranks an Apify Actor. It is not your title.

Dodo Data — Tue, 22 Sep 2026 09:39:05 +0000

I published eight Actors on Apify Store in five days. Careful titles, real READMEs written from real runs, dataset schemas, published task examples, honest pay-per-event pricing. Then I searched the Store for the exact thing I had built and could not find myself.

Not "found low". Not found at all in the first fifty results.

So I stopped guessing and measured it. The answer is one number, it is not the number I had been optimising, and the whole week of title tuning was wasted effort.

The thing that looked like proof

Earlier in the week I renamed an Actor and watched it vanish from search, then reappear a few hours later at positions 12, 7 and 26 for three related terms. Another Actor appeared at position 10 for its main keyword three hours after I published it.

Rename, then rank. Publish, then rank. Obvious: Store search matches on title.

That was wrong, and the shape of the error is worth naming. I had two events and an interpretation, and I never ran the control.

The control

Search the Store for greenhouse jobs. Look at who is above me.

#    users30  actor
1       114   fantastic-jobs/greenhouse-jobs-api
...
9         0   vnx0/greenhouse-jobs-scraper
13        0   skootle/greenhouse-jobs
15        0   charliemorrisondev/greenhouse-jobs-scraper

Three Actors at positions 9, 13 and 15 with zero users in the last thirty days. My dododata/greenhouse-jobs-scraper has the same name, a longer README, a dataset schema, published task examples and a working price, and it is at 74.

Identical names. Sixty places apart. Title match cannot explain that.

The difference is in a field I had not been looking at, because the Store UI shows you the monthly number:

Actor	position	users, 30 days	users, lifetime	runs, lifetime
vnx0/greenhouse-jobs-scraper	9	0	12	236
skootle/greenhouse-jobs	13	0	5	196
charliemorrisondev/greenhouse-jobs-scraper	15	0	4	193
dododata/greenhouse-jobs-scraper	74	1	2	11

The correlation

One request per term, then Spearman's rho between result position and each stat over the first thirty to forty results.

import json, urllib.request, urllib.parse

def spearman(x, y):
    def rank(v):
        order = sorted(range(len(v)), key=lambda i: v[i])
        r = [0] * len(v)
        for pos, i in enumerate(order):
            r[i] = pos
        return r
    rx, ry = rank(x), rank(y)
    n = len(x)
    d2 = sum((rx[i] - ry[i]) ** 2 for i in range(n))
    return 1 - 6 * d2 / (n * (n * n - 1))

url = "https://api.apify.com/v2/store?search=" + urllib.parse.quote("greenhouse jobs") + "&limit=40"
items = json.load(urllib.request.urlopen(url))["data"]["items"]

inv_position = [-i for i, _ in enumerate(items, 1)]
lifetime = [(a.get("stats") or {}).get("totalUsers", 0) for a in items]
monthly  = [(a.get("stats") or {}).get("totalUsers30Days", 0) for a in items]

print(spearman(inv_position, lifetime), spearman(inv_position, monthly))

Three terms:

Search term	lifetime users	lifetime runs	30-day users
greenhouse jobs	+0.934	+0.645	+0.111
app store reviews	+0.930	+0.864	+0.692
shopify products	+0.871	+0.574	+0.387

Rank is cumulative distinct users, all time. The thirty-day number that the UI puts in front of you barely registers.

This also explains the two events that fooled me. A rename does not change your rank; it briefly removes you from the index and then puts you back where you always were. And an Actor that "appeared at position 10" appeared there for a keyword where the whole field was near zero, so two lifetime users was enough.

What that means if you are new

The metric never decays. That cuts both ways.

It is a ratchet, so every real user you ever get is permanent rank equity. It is also a wall, because the people above you banked theirs months ago and keep them for free. Paging the search API to the end, here is where five days of work actually put me:

Term	the Actor	my rank	lifetime runs of the lowest Actor in the top 20
workday jobs	workday-jobs-scraper	49	57
greenhouse jobs	greenhouse-jobs-scraper	74	57
shopify products	shopify-products-scraper	117	102
app store reviews	app-store-reviews-scraper	174	153

You cannot fake it either, and not only because faking things on a platform that counts them is a bad way to run a business. totalUsers counts distinct users, so running your own Actor a thousand times moves nothing.

The practical conclusion is uncomfortable and clarifying at the same time. On-platform discovery for a new account is a closed loop: search is how people find you, search is ordered by usage, and usage comes from search. Your first users cannot come from the Store. They have to come from somewhere else, and then the Store starts working for you.

The one surface that is not gated on usage

There is exactly one: sortBy=newest. It orders by publication date and nothing else.

https://api.apify.com/v2/store?sortBy=newest&limit=100

My newest Actor was at position 17 four hours after publishing, while the seven older ones sat at 728 to 745. Measuring the age of the Actor at each depth gives the decay:

Position	Age of the Actor there
1	1 hour
101	17 hours
301	33 hours
701	61 hours

About six new Actors an hour, platform-wide. So publishing buys you roughly three hours on the first page of newest and about a day in the top hundred.

Three hours is not much. But it is the only window a zero-history account gets, and it has a scheduling consequence I had completely backwards: never publish two Actors on the same day. I had three finishing staging together and was going to ship them in one batch. That would have spent three windows to buy one. They now go out on three separate days, each one paired with whatever off-platform writing lands the same afternoon, because those hours are the only time the platform itself is sending anyone.

The bug in my own measurement

Worth repeating because it nearly became a published claim.

My first pass said my Actors were missing from the Jobs category page entirely, which would have been a much more dramatic post. The loop looked like this:

off = 0
while True:
    items = get(f"...&category=JOBS&limit=100&offset={off}")
    if not items:
        break      # <-- wrong
    off += 100

The Store listing endpoint intermittently returns an empty page in the middle of a result set. Treating that as the end of the list stopped the scan at 994 records, my Actors were not in those 994, and the conclusion wrote itself. Retrying an empty page shows the category holds 6,066 Actors and mine are all in it, at positions 1,192 to 1,213. Buried by the same variable as everything else, not excluded.

If you paginate this API, retry an empty page before you believe it.

What I would tell myself on day one

Spend day one on distribution and day two on the Actor. Not the other way round, which is what I did.

The build quality still matters, just not for the reason you think. It does not rank you. It is what makes the handful of people who arrive from somewhere else come back, and a returning user is the only input to the one number that does rank you.

The Actors every number above is measured on are at apify.com/dododata. Public sources only, no personal data, robots.txt respected, pay per event. Ask me anything about the measurement and I will run it again with your search term.

5,966 job posts: who actually publishes pay, and who only looks like they don't

Dodo Data — Mon, 21 Sep 2026 05:57:21 +0000

Every applicant-tracking system has a structured pay field. Greenhouse calls it pay_input_ranges, Ashby has compensation, Lever has salaryRange. If you are building anything on job data, that field is where you look for salary, and it is where most pipelines stop.

I pulled 24 public career sites, 5,966 open jobs, and counted. Then I read the descriptions of the boards that came back empty, and the number moved by 445 jobs.

Two of the 24 boards had no open jobs that day, so every count below comes from the 22 that did. The per-board numbers are published as open data at the end of this post, so you can check any of it.

The structured field

One request per board, no browser, no login. Greenhouse at boards-api.greenhouse.io/v1/boards/<board>/jobs, Ashby at api.ashbyhq.com/posting-api/job-board/<org>?includeCompensation=true, Lever at api.lever.co/v0/postings/<org>?mode=json.

System	Boards	Jobs	With structured pay
Ashby	8	1,293	72.8%
Greenhouse	12	3,435	56.2%
Lever	4	1,238	3.2%

Lever's 3.2% is the first thing worth noticing. It is not that Lever companies pay in secret. salaryRange exists on the Lever posting API, and almost nobody fills it in. Three of the four boards I pulled had it empty on every single job.

Per company, the spread inside one system is wider than the gap between systems:

Board	System	Jobs	Structured pay
Duolingo	Greenhouse	81	100.0%
Coinbase	Greenhouse	217	98.2%
Ramp	Ashby	148	95.3%
Anthropic	Greenhouse	611	89.4%
OpenAI	Ashby	816	82.4%
Databricks	Greenhouse	874	54.1%
Stripe	Greenhouse	665	0.0%
Palantir	Lever	313	0.0%
Notion	Ashby	128	0.0%
Discord	Greenhouse	47	0.0%

The zeroes are not all the same zero

Four of those 0.0% boards state a pay range in plain English in the job description. They just never filled in the field.

I ran a range extractor over the descriptions of every board that reported zero:

Board	Structured	Found in description text
Discord	0 of 47	44
Palantir	0 of 313	200
Notion	0 of 128	79
Affirm	0 of 199	121
Stripe	0 of 665	1
Spotify	0 of 71	0
Linear	0 of 32	0

Discord publishes pay on 94% of its jobs. Any dataset built on the structured field alone reports 0%.

That is the finding: an empty pay field is a data-entry fact, not a pay-transparency fact. Stripe and Spotify genuinely do not publish ranges. Discord, Palantir, Notion and Affirm do, in a sentence near the bottom of the post that no API returns as a number.

Across the whole sample, the structured field says 48.8% of jobs publish pay. Reading the text as well takes it to 56.3%, and the gain is concentrated entirely in the boards that looked silent.

Extracting a range from prose, without inventing numbers

The naive version of this is a regex for two dollar amounts with a dash between them, and it is wrong within about ten jobs. Here is what actually breaks.

Equity tables look exactly like salary ranges. Affirm puts this in most of its posts:

Equity Grade - 2 (awarded in US Dollars)
Equity grade: 2   New hire equity: $16,000-$24,000   Annual Refresh: $5,000

$16,000-$24,000 is a real range next to a real dollar sign, and it is not the salary. A blocklist on the preceding text is what saves you: equity, refresh, bonus, sign-on, 401, stipend, RSU, grant, option.

On-target earnings are not base pay. Sales roles quote OTE, which is base plus commission at 100% attainment. Rolling it into a salary_min column silently inflates every sales row in the dataset. I let those go unmatched on purpose, which is most of what the extractor misses on Affirm.

A number needs a cue word in front of it, not just a currency symbol. Requiring one of base pay, pay range, salary range, compensation range, hourly rate and so on within the preceding 90 characters removes nearly all of the false positives without costing real matches.

Hourly and annual are the same shape. $22 - $30 and $220,000 - $300,000 differ by three zeros. Decide the period explicitly from the surrounding words, and refuse to guess when there is neither an hourly nor an annual cue and the top of the range is under $10,000.

The whole thing is about thirty lines:

const MONEY = String.raw`(?:\$|USD\s?|£|€)\s?(\d{1,3}(?:,\d{3})+|\d{2,3}(?:\.\d+)?\s?[kK]\b|\d{2,6}(?:\.\d{2})?)`;
const RANGE = new RegExp(`${MONEY}\\s*(?:-|–|—|to|and)\\s*${MONEY}`, 'g');
const PAY_CUE = /(base pay|pay range|base salary|salary range|compensation range|annual salary|hourly rate|pay rate|salary of|compensation of|remuneration)/i;
const NOT_PAY = /(equity|refresh|bonus|sign[- ]?on|401|stipend|budget|revenue|funding|raise[ds]|valuation|grant|rsu|option)/i;

for (const m of flat.matchAll(RANGE)) {
    const before = flat.slice(Math.max(0, m.index - 90), m.index);
    if (!PAY_CUE.test(before) || NOT_PAY.test(before)) continue;
    // ...amounts, currency from the symbol, period from hourly/annual cues
}

It does not handle 35 000 €, the French spacing convention, and I left it that way rather than half-supporting it. A French board returns null, which is honest, instead of a number pulled out of a format the parser does not really read.

Keep the two sources apart

If you merge text-extracted ranges into the same column as the structured field and say nothing, you have made your dataset less trustworthy, not more. One came from a form the employer filled in. The other came from a regex over prose, and it can be wrong.

Carry a third column. Mine is salary_source, one of structured, description, or null, and a run over Affirm and Duolingo looks like this:

200 rows   structured: 81   description: 94   none: 25
  duolingo   structured: 81
  affirm     description: 94   none: 25
  e.g. Administrative Assistant IV        115,000 - 165,000 USD
  e.g. Affirm Bank Strategic Finance Mgr  185,000 - 245,000 USD

Anyone doing serious analysis can filter to structured only. Anyone building a job board gets a pay range on four times as many Affirm roles. Nobody gets a number without knowing where it came from.

The per-board numbers, as open data

Every board's counts, free and machine-readable, so you do not have to take the tables on trust:

https://api.apify.com/v2/datasets/6EJmXWfWFxmjjCUtl/items?clean=true&format=json
https://api.apify.com/v2/datasets/6EJmXWfWFxmjjCUtl/items?clean=true&format=csv

One row per board: jobs_open, jobs_with_structured_pay, structured_pay_pct, and where I also read the
descriptions, jobs_with_pay_in_description_text with description_text_checked saying so. The two boards with
no open jobs are in there as zeroes rather than quietly dropped, and every row carries the endpoint it came from.

Use it

The extractor ships in the Dodo Data job scrapers on Apify Store, across Greenhouse, Lever, Ashby, Workday and the multi-system one. Same 30-field row shape including salary_source, $1 per 1,000 jobs:

https://apify.com/dododata/career-site-jobs-scraper

If you want to re-run the survey yourself, it is one request per board and the endpoints are all listed above. The interesting number is not the average. It is which of your target employers are in the "publishes pay, never filled in the field" bucket, because that one is invisible until you read the text.

Scraping Workday career sites without a browser, and the 2,000-job ceiling

Dodo Data — Mon, 21 Sep 2026 03:09:50 +0000

Greenhouse, Lever, Ashby and half a dozen other applicant-tracking systems publish a job feed you can fetch in one request. Workday, the one running the career pages of most large enterprises, does not. No /jobs.json, no RSS, and the page itself is an empty JavaScript shell.

It still does not need a browser. Every Workday career site talks to its own backend over a plain POST, and that endpoint answers anyone.

The listing call

POST https://<tenant>.<pod>.myworkdayjobs.com/wday/cxs/<tenant>/<site>/jobs
Content-Type: application/json

{ "appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "" }

Everything you need is in the career-page URL. https://nvidia.wd5.myworkdayjobs.com/NVIDIAExternalCareerSite gives tenant nvidia, pod wd5, site NVIDIAExternalCareerSite. Mind the en-US some sites insert before the site name.

The reply is small and clean:

{
  "total": 2000,
  "jobPostings": [
    {
      "title": "Senior Firmware Engineer – CSP Engagements",
      "externalPath": "/job/US-CA-Santa-Clara/Senior-Firmware-Engineer---CSP-Engagements_JR1999599",
      "locationsText": "US, CA, Santa Clara",
      "postedOn": "Posted Today",
      "bulletFields": ["JR1999599"]
    }
  ]
}

Three things to know before you build on it.

limit caps at 20. Ask for 50 and you get a 400, not a truncated page. A full career site is total / 20 requests, minimum.

total is not the number of jobs. It is 2,000 on NVIDIA and 1,483 on Salesforce, and NVIDIA does not have exactly two thousand openings. 2,000 is the ceiling the site will paginate to. Past it you get empty pages. If a site reports exactly 2,000, assume you are seeing the 2,000 most recent and say so downstream rather than claiming a complete catalogue.

Only the first page carries total. Later pages return it as 0. Carry it forward yourself, or your loop will stop after page two.

The descriptions are in the HTML, as JSON-LD

The listing gives you title, location, date and requisition id, but no description, no country, no employment type. Those sit on each job's public page, and you do not have to parse the DOM for them: every Workday job page carries a schema.org JobPosting block.

const $ = load(html);
let ld: Json = {};
$('script[type="application/ld+json"]').each((_i, el) => {
    const d = JSON.parse($(el).text());
    if (d['@type'] === 'JobPosting') ld = d;
});
// ld.description, ld.datePosted, ld.employmentType,
// ld.jobLocation.address.addressCountry, ld.identifier.value

That is one request per job, which is the whole cost of the job. Filter on the listing first, where title, location and date are all available, and only open what matches. A "remote engineering roles posted this week" run on a 2,000-job site is about 100 listing requests plus a handful of job pages, not 2,000.

"Posted Today" is not a date

postedOn is relative text: Posted Today, Posted Yesterday, Posted 3 Days Ago, Posted 30+ Days Ago. The first three convert cleanly. The last one does not. It is a lower bound, not a date, and turning it into "30 days ago" invents precision the site never gave you:

export function workdayPostedOn(text: string, now = Date.now()): string | null {
    const t = text.toLowerCase();
    let days: number | null = null;
    if (t.includes('today')) days = 0;
    else if (t.includes('yesterday')) days = 1;
    else {
        const m = t.match(/(\d+)(\+?)\s+days?/);
        if (m && !m[2]) days = Number(m[1]);   // "30+" falls through to null
    }
    return days === null ? null : new Date(now - days * 86_400_000).toISOString().slice(0, 10);
}

The JSON-LD on the job page has a real datePosted, so a row that opens the page gets the exact date anyway. The relative text only matters when you are filtering on the listing, which is exactly when you want it to be honest.

robots.txt

https://<tenant>.<pod>.myworkdayjobs.com/robots.txt explicitly allows the career site and disallows /talentcommunity/ and /refreshFacet/. Read it per tenant rather than assuming. It is one request, and it is the difference between a public page and someone's candidate area.

What a real run looks like

500 jobs from one career site, with descriptions: 526 requests, 2 minutes 30 seconds, 4 timeouts that all succeeded on retry. The timeouts are worth planning for, because Workday's backend is slower than a static feed, and a 30-second request timeout with retries is not excessive.

Use it

If you would rather not maintain the pagination, the JSON-LD merge and the relative dates, it is on Apify Store as Workday Jobs Scraper & API. Paste career-site URLs, filter by title keywords, location or "posted within N days" before any job page is opened, $1 per 1,000 jobs:

https://apify.com/dododata/workday-jobs-scraper

Same 26-field shape as the Greenhouse, Lever and Ashby ones, so a multi-employer pipeline does not need a branch per system. If something changes, it is fixed within 48 hours, and that promise is on the listing.

Every Shopify store publishes its whole catalogue at one URL

Dodo Data — Sat, 19 Sep 2026 09:14:35 +0000

If you have ever written a scraper for a Shopify store, you probably parsed the HTML. You did not have to.

Every Shopify storefront serves its own catalogue as JSON at /products.json. It is not a private API and not a leak. It is the storefront's own public endpoint, the same data the shop renders for any visitor:

https://www.allbirds.com/products.json?limit=250&page=1
https://www.allbirds.com/collections/mens/products.json?limit=250&page=1
https://www.allbirds.com/meta.json          store name, currency, country

250 products per page, page=N until you get an empty list. Collections work the same way, so you can scope a run to one part of a shop.

What a product actually contains

The interesting part is not the title, it is variants. Each variant is a real SKU with its own price, its own crossed-out price and its own stock flag:

{
  "id": 7205191974992,
  "title": "Daily Grind Women's Corduroy Workwear Jacket",
  "vendor": "Death Wish Coffee",
  "variants": [
    { "sku": "DWC-JKT-S", "price": "65.00", "compare_at_price": "90.00", "available": true,  "option1": "S" },
    { "sku": "DWC-JKT-M", "price": "65.00", "compare_at_price": "90.00", "available": false, "option1": "M" }
  ]
}

Two things follow from that shape, and both are easy to get wrong.

A discount only means something inside one variant. If you take the lowest price in the product and compare it with the highest compare_at_price in the product, a shop that sells a $10 accessory and a $100 jacket marked down from $120 will report a 92% discount. It is not on sale by 92%; those are two different things being subtracted from each other. Compute the discount per variant, then take the best one:

const discounts = variants
    .filter((v) => v.on_sale && v.price !== null && v.compare_at_price)
    .map((v) => Math.round((1 - v.price / v.compare_at_price) * 100));
const bestDiscount = discounts.length ? Math.max(...discounts) : null;

Stock is per variant too. "In stock" for a product usually means some size is in stock. If you are monitoring availability, one row per variant is the only honest shape.

Prices are strings, and the currency is somewhere else

price arrives as "65.00", a string. /products.json never tells you the currency, which lives in /meta.json alongside the shop's name and country. Fetch it once per store and attach it to every row, or your dataset silently mixes USD, EUR and GBP in one column.

A real run

Two stores (Allbirds and Death Wish Coffee), one run on 2026-09-17: 439 products, 3,282 variants, 9 seconds, 5 requests. 23 products were on sale, 246 had at least one variant in stock, and every row carried the store's currency. The deepest genuine discount was a pair of socks at $4 against a $16 compare-at price, computed inside one variant, which is why it can be trusted.

What the endpoint will not give you

Some shops turn /products.json off or put it behind a bot check; Gymshark, for one, returns 403. Treat that as a normal outcome, log it and move on. It is a per-store decision by the merchant, not a puzzle to defeat. And a site that is not on Shopify simply 404s.

Use it

If you would rather not maintain the pagination, the variant maths and the per-store currency, it is on Apify Store as Shopify Products Scraper & API. Paste store or collection URLs, choose one row per product or per variant, filter to in-stock or on-sale only, $1 per 1,000 rows:

https://apify.com/dododata/shopify-products-scraper

The id field is stable across runs (shopify:store:product:variant), so a daily run plus a diff is a price and stock monitor in about one line of code. If something changes, it is fixed within 48 hours, and that promise is on the listing.

Every open job at 500 companies, as JSON, without scraping a single job board

Dodo Data — Fri, 18 Sep 2026 07:50:29 +0000

Most "job data" projects start by scraping LinkedIn or Indeed, and most of them die there: captchas, logins, lawsuits, and listings that are already three reposts away from the employer.

There is a quieter source. Most tech companies run their career page on an applicant-tracking system: Greenhouse, Lever, Ashby, SmartRecruiters, Recruitee, Workable, Personio, Breezy, Pinpoint, Rippling, Teamtailor. Every one of those systems publishes a job feed so the career page can be embedded on the company's site. First-party, current, no login, no anti-bot.

The feeds

Greenhouse       https://boards-api.greenhouse.io/v1/boards/{company}/jobs?content=true&pay_transparency=true
Lever            https://api.lever.co/v0/postings/{company}?mode=json
Ashby            https://api.ashbyhq.com/posting-api/job-board/{company}?includeCompensation=true
SmartRecruiters  https://api.smartrecruiters.com/v1/companies/{company}/postings
Recruitee        https://{company}.recruitee.com/api/offers/
Workable         https://apply.workable.com/api/v1/widget/accounts/{company}?details=true
Personio         https://{company}.jobs.personio.de/xml
Breezy HR        https://{company}.breezy.hr/json
Pinpoint         https://{company}.pinpointhq.com/postings.json
Rippling         https://api.rippling.com/platform/api/ats/v1/board/{company}/jobs
Teamtailor       https://{career-site-host}/jobs.rss

{company} is the slug in the career-page URL: jobs.lever.co/spotify gives spotify.

The catch: eleven shapes

The feeds agree on nothing. Greenhouse sends the description as HTML that has been entity-escaped once (<p>), pay ranges in cents, and the location as { "name": ... }. Lever sends createdAt as a millisecond string and splits the description across descriptionPlain, lists[] and additionalPlain. Ashby nests compensation three levels deep. SmartRecruiters paginates and needs one more request per posting for the text. Personio is XML, Teamtailor is RSS with its own namespace, and Rippling's feed has no descriptions at all.

Un-escaping Greenhouse, for instance, is a two-step:

import { load } from 'cheerio';

export function htmlToText(html: unknown): string | null {
    if (!html) return null;
    let h = String(html);
    if (!h.includes('<') && h.includes('&lt;')) h = load(`<x>${h}</x>`)('x').text(); // un-escape once
    h = h.replace(/<\s*br\s*\/?>/gi, '\n').replace(/<\/(p|div|li|h[1-6]|ul|ol|tr)>/gi, '\n').replace(/<li[^>]*>/gi, '• ');
    return load(`<x>${h}</x>`)('x').text().trim() || null;
}

(Workday, the twelfth system, has no feed. Its career sites are allowed by robots.txt and every job page carries schema.org JobPosting as JSON-LD, so it is read the old-fashioned way: listing, then page.)

One shape out

I normalised all twelve into 26 fields: title, department, team, locations[], workplace_type (remote / hybrid / onsite), salary_min, salary_max, salary_currency, posted_at, apply_url, plain-text description, and a stable id (ats:company:job) so a daily run can be diffed in one line.

A real run on 2026-09-17, three boards (Duolingo on Greenhouse, Spotify on Lever, Ramp on Ashby): 304 jobs in 18 seconds, 224 of them (74%) with a published pay range already parsed to numbers.

Two things I deliberately leave out: recruiter names and mailboxes that some feeds carry, and any e-mail or phone number inside a description (replaced with [email removed]). It is job data, not people data.

Use it

If you would rather not maintain twelve parsers, it is on Apify Store as Career Site Jobs Scraper. Paste career-page URLs, filter by title keywords, location or "posted within N days", $1 per 1,000 jobs, callable from the API or from an AI agent over MCP:

https://apify.com/dododata/career-site-jobs-scraper

If a feed changes shape, it is fixed within 48 hours, and that promise is on the listing.