DEV Community

Daniel Meshulam
Daniel Meshulam

Posted on

Indeed 403s you with zero bytes. The companies it aggregates publish free APIs.

#webscraping #python #api #datascience

I spent an afternoon measuring which job sources actually answer a request, and
the result was cleaner than I expected: the aggregators are shut and the
sources they aggregate from are wide open.

Everything below was measured on 2026-08-01 from an ordinary residential
connection with normal browser headers. Not read from documentation.

The aggregators 

indeed.com/jobs          403      0 bytes
glassdoor.com/Job/...    403      0 bytes
ziprecruiter.com         403      0 bytes
upwork.com/nx/search     403      0 bytes
Enter fullscreen mode Exit fullscreen mode

Zero bytes. Cloudflare rejects these before the request reaches an application,
so there is nothing to parse and nothing to be clever about. That is from a
residential address, which is the friendliest IP you can bring. A datacenter
address does worse.

Reddit is worth a mention because it fails in a more interesting way:

reddit.com/r/programming/hot.json    403   189,908 bytes
old.reddit.com/.../hot.json          403   189,908 bytes
reddit.com/r/programming/.rss        200    40,349 bytes   <- once
reddit.com/r/programming/.rss        429         0 bytes   <- and every time after
Enter fullscreen mode Exit fullscreen mode

Note the 189,908 bytes on a 403. If you check len(response.content) instead of
response.status_code, that block page reads as a large successful response. I
have written that bug. It is the most expensive one-line mistake in scraping,
because it does not fail, it lies.

The sources

Indeed does not originate job data. Neither does Glassdoor, ZipRecruiter or
LinkedIn Jobs. They aggregate from company career pages, and those career pages
run on applicant tracking systems. Every major one publishes a public JSON API
with no key, because companies want their openings indexed.

GET https://boards-api.greenhouse.io/v1/boards/stripe/jobs
200    548 jobs

GET https://api.ashbyhq.com/posting-api/job-board/openai
200    753 jobs

GET https://api.lever.co/v0/postings/palantir?mode=json
200    302 jobs

GET https://api.rippling.com/platform/api/ats/v1/board/rippling/jobs
200    768 jobs
Enter fullscreen mode Exit fullscreen mode

Also open: Workable, Recruitee, SmartRecruiters, Personio. No key, no proxy, no
bot wall, and fresher than any aggregator, because an aggregator shows you its
last crawl and this shows you the board.

One distinction worth making, because these are not all the same thing.
Greenhouse, Lever, Ashby, Recruitee and SmartRecruiters document these
endpoints publicly as needing no authentication. Greenhouse says it plainly:
"Job Board data is publicly available, so authentication is not required for any
GET endpoints." Lever goes further and says published postings "may be scraped
by third parties."

Rippling and Workable are different. Rippling's developer documentation sits
behind a login, and Workable's documented API is a separate authenticated one.
The endpoints I used for those two are the ones their own public careers pages
call, they answer an anonymous request, and they serve postings a company chose
to publish. But "answers without a key" is not the same claim as "the vendor
documents it as public", and I would rather say which is which than let the
list imply all seven are equivalent.

Trap 1: you need the board token, and there is no directory

Every one of those URLs contains a token: stripe, openai, palantir. There
is no public mapping from a company to its token. This is the actual reason
people reach for Indeed instead.

The obvious fix is to fetch the company's careers page and look for an ATS link.
I tried that on 12 companies and it found 6. Stripe's careers page is a
JavaScript app that names no board anywhere in the HTML.

The fix that works is the other way round: derive candidates from the domain
and verify each against the live API.

def tokens(domain):
    stem = domain.split("//")[-1].split("/")[0].removeprefix("www.").split(".")[0]
    return [stem, stem.replace("-", "")]

PROBES = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever":      "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby":      "https://api.ashbyhq.com/posting-api/job-board/{}",
    "recruitee":  "https://{}.recruitee.com/api/offers/",
    "workable":   "https://apply.workable.com/api/v1/widget/accounts/{}?details=true",
    "rippling":   "https://api.rippling.com/platform/api/ats/v1/board/{}/jobs",
}
Enter fullscreen mode Exit fullscreen mode

stripe.com gives stripe, and boards-api.greenhouse.io/v1/boards/stripe/jobs
returns 548 jobs. Both methods, measured:

Method Sample Resolved
Read the careers page for a link 12 companies 50%
Derive the token and verify it 30 companies 90%

Trap 2: why guessing is sound here, and where it stops being sound

Guessing is only safe if a wrong guess is distinguishable from a right one. I
checked each API with a real token and a deliberately bogus one:

greenhouse   real 200      bogus 404
lever        real 200      bogus 404
ashby        real 200      bogus 404
recruitee    real 200      bogus 404
workable     real 200      bogus 404
rippling     real 200      bogus 404
Enter fullscreen mode Exit fullscreen mode

A 404 for an unknown board is what makes a 200 proof rather than a maybe.
And a real board with nothing open is a third, distinct state:

ashby/deel     200   {"jobs":[],"apiVersion":"1"}
lever/kraken   200   []
Enter fullscreen mode Exit fullscreen mode

So 404 means no such board, 200 [] means the board exists and is empty.
Clean semantics, and they let you tell "wrong token" apart from "not hiring".

SmartRecruiters is the exception, and it is worth knowing before you trust it:

smartrecruiters  real company Visa            200  {"totalFound":2,   "content":[...]}
smartrecruiters  company with nothing open    200  {"totalFound":0,   "content":[]}
smartrecruiters  company that does not exist  200  {"totalFound":0,   "content":[]}
Enter fullscreen mode Exit fullscreen mode

The last two are byte-identical. The obvious second signal does not rescue it
either: GET /v1/companies/Visa returns 404, for a company whose postings
endpoint returns real jobs. So on SmartRecruiters there is no way to confirm a
guessed id, and the honest move is to require an exact one rather than emit a
confident zero. Company ids there are also case sensitive: Visa works, visa
does not.

This is the same shape as a family of bugs I keep meeting: an absence being
returned as an answer. RDAP does it too, where a 404 means "no server for this
TLD" and gets read as "domain available".

Trap 3: descriptions cost 18x, and change the schema

Greenhouse takes ?content=true. Same board, measured:

boards-api.greenhouse.io/v1/boards/figma/jobs                  111 KB
boards-api.greenhouse.io/v1/boards/figma/jobs?content=true    1.97 MB
Enter fullscreen mode Exit fullscreen mode

Eighteen times the payload. What is less obvious is that departments and
offices only exist when content is on, so if you want the department of a
role you are paying for every description whether you wanted them or not.

Rippling goes the other way: its board API returns five fields and no
description at any setting. If you build a uniform schema across platforms, that
field is empty because the source has none, not because you missed it. Worth
saying in your output rather than shipping a null that reads as a bug.

What you get for the trouble

One request per company, no key, no proxy, and data that has not been through
somebody else's crawl schedule. Stripe's 548 roles, Databricks' 803, OpenAI's

  1. Titles, locations, departments, offices, descriptions.

And on Ashby, salary. Of the seven JSON sources it is the only one that
publishes a compensation band, and it hides it behind a parameter:

GET /posting-api/job-board/ramp                             no compensation key
GET /posting-api/job-board/ramp?includeCompensation=true    "$211.4K - $290.6K"
Enter fullscreen mode Exit fullscreen mode

Every Ramp and OpenAI posting carries a real band. Notion and Linear return the
field full of nulls, which is a company choosing not to publish rather than a
missing feature, and the two are worth telling apart.

Correction, added after publishing. I first wrote that Ashby was the only
one of the eight sources here with salary. That is wrong: Personio, the XML one,
publishes it too, and in a better shape. Ashby gives you a formatted string,
"$211.4K - $290.6K". Personio gives you the numbers:

<salaryInformation><min>45000.00</min><max>55000.00</max></salaryInformation>
<currencyCode>EUR</currencyCode><currencySymbol>&#8364;</currencySymbol>
Enter fullscreen mode Exit fullscreen mode

Structured min, max and currency, which is what you want if you are sorting or
filtering rather than displaying. Personio is a European mid-market system, so
this is worth knowing if that is your segment. I had this parameter wired
to the description flag in my own client and was silently dropping salary on
every run that did not ask for descriptions, which is a good argument for
reading a response you think you already understand.

The aggregators are not gatekeepers of this data. They are readers of it, same
as you, and they are the only ones who have to be let in.

Every endpoint in this post is public and the measurements stand on their own,
so nothing here needs a product. I am packaging the eight clients and the domain
resolver as Actors on the Apify Store under
glitchbound, which is also where my other
scrapers live if this is the kind of thing you use.

Top comments (0)