DEV Community

Daniel Meshulam
Daniel Meshulam

Posted on

Merging job postings from ten ATS platforms into one schema, and the four fields that fight back

#api #python #datascience

Reading one ATS job board API is easy. Every one of them is public JSON with
no key. Reading ten and getting rows you can actually query together is a
different job, and it is almost entirely about four fields.

I run an index across Greenhouse, Ashby, Workable, Workday, BambooHR, Breezy,
Teamtailor, Personio, Recruitee and Rippling. Here is what actually broke,
with the measurements.

1. Remote is not a boolean, and one API will lie to you about it

Ashby postings carry isRemote. It looks like exactly the field you want.

Measured on Ramp's board, 1 August 2026: isRemote was true for 117 of 126
postings, while the same postings' workplaceType said Hybrid 101, Remote
16, OnSite 9.

Ashby's isRemote means "not fully on-site". Trust it and a user asking for
remote work gets a hundred hybrid roles at a New York office.

workplaceType is the honest field, and it has three values, because hybrid
is a real third state that a boolean cannot hold. Normalise to the three-way
enum and derive the boolean from it, never the reverse:

workplace_type = raw.get("workplaceType")            # Remote | Hybrid | OnSite
is_remote = (workplace_type == "Remote") if workplace_type else None
Enter fullscreen mode Exit fullscreen mode

Note the None. A platform that publishes no arrangement at all gets unknown,
not False. "We do not know" and "we know it is not remote" are different
answers and collapsing them is how a filter silently drops good rows.

2. Dates arrive in three shapes, one of which is relative

ISO 8601 from some. Epoch milliseconds from others. And Workday hands you a
human sentence: "Posted 4 Days Ago", "Posted Today",
"Posted 30+ Days Ago".

Epoch handling has a cheap tell:

# Milliseconds since the epoch are 13 digits until the year 2286.
secs = n / 1000 if n > 10_000_000_000 else n
Enter fullscreen mode Exit fullscreen mode

The relative ones need parsing, and one of them must be refused:

_AGO = re.compile(r"^posted\s+(\d+)\s+days?\s+ago$", re.I)

def workday_posted(text):
    t = (text or "").strip().lower()
    if t == "posted today":     days = 0
    elif t == "posted yesterday": days = 1
    else:
        m = _AGO.match(t)
        if not m:
            return None          # "posted 30+ days ago" lands here, on purpose
        days = int(m.group(1))
    return (datetime.now(timezone.utc).date() - timedelta(days=days)).isoformat()
Enter fullscreen mode Exit fullscreen mode

"Posted 30+ Days Ago" returns None deliberately. It could be 31 days or

  1. Turning it into "31 days" produces a date that sorts, filters and charts perfectly and is fiction. Keep the original string in a separate field and let the caller see what the source actually said.

3. Employment type can vary between two boards on the SAME platform

This one surprised me. It is not just that platforms disagree with each other.
Workday's timeType is tenant-configured, so two customers of the same ATS
publish different things.

Measured 2026-08-08 across ten Workday boards: two publish timeType on
every posting, and eight publish it on none. The only value that appeared at
all was "Full time".

So "does this platform have employment type" is not a question with an answer,
and any per-platform lookup table encodes a guess. Read the field where the
board provides it, map it through one shared vocabulary, and leave it null
everywhere else. I had this branch hard-coded to None for a while, which
meant 258,876 rows carried no employment type even where Workday had plainly
stated one.

The trap on the other side is a filter for "full-time" that silently excludes
the majority of your index, because most rows are null rather than false.
Filter on != 'part-time' if you mean "not part-time", and say in your docs
which one you did.

4. Company name is often not a company name

Several platforms return the board token where you expect the employer, so you
get acmecorp instead of Acme Corp, and on Workday you can get the hostname.
Pull the display name from the board envelope rather than the posting, and if
it is absent, say so instead of shipping a slug that looks like a name.

The rule underneath all four

Every one of these bugs has the same shape: a field existed, so it got
trusted. isRemote exists. A parsed "30+ days" exists. A board token
exists where a company name goes. Each one produces a value that is
well-formed, sorts correctly, and is wrong.

The fix is always the same and it is not technical. Carry unknown as unknown.
Three-state where reality has three states. Null where the source said nothing.
It makes your schema uglier and your answers true, and in this data the null
rate is itself information: it tells you which platform to stop promising
things about.

The index, if you want the rows rather than the code

24,280 company boards, 10 ATS platforms, 688,711 open roles in one schema,
rebuilt nightly, no login and no company list. On Apify as
ATS Jobs Search API, or
single-platform:
Greenhouse,
Workday,
Ashby,
Workable,
BambooHR.

Posted date is populated on 76.2% of rows and country on 78.0%. Those are the
real numbers, not the ones I would like, and the missing quarter is mostly
Workday's "30+ days" refusing to become a date.

Top comments (0)