Normalizing public job-board data with Python

Abdulwahab — Sun, 27 Sep 2026 15:01:03 +0000

Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boards. You don't need a key or a login. The catch is that the three APIs describe a job in three different ways, so putting several boards in one spreadsheet means mapping three shapes into one.

This post covers:

how the three APIs differ,
a small standard-library Python script that normalizes title, location and apply link (plus a few optional fields),
real output from a run on 27 September 2026 (UTC),
JSON and CSV export,
missing values, source limits and common mistakes.

How the three APIs differ

Each board is identified by a slug: the board name in the vendor URL, such as palantir in jobs.lever.co/palantir or ramp in jobs.ashbyhq.com/ramp. Greenhouse's docs call it the board_token.

	Greenhouse	Lever	Ashby
List endpoint	`/v1/boards/{slug}/jobs`	`/v0/postings/{slug}?mode=json`	`/posting-api/job-board/{slug}`
Response	`{"jobs": [...], "meta": ...}`	bare JSON array	`{"apiVersion": ..., "jobs": [...]}`
Title	`title`	`text`	`title`
Location	`location.name`	`categories.location`	`location`
Apply link	`absolute_url` (posting page)	`applyUrl` (form), `hostedUrl` (page)	`applyUrl` (form), `jobUrl` (page)
Date	`first_published`, ISO with offset	`createdAt`, epoch milliseconds	`publishedAt`, ISO
Workplace	not provided	`workplaceType`	`workplaceType`, `isRemote`
Department	not in the list	`categories.department` or `team`	`department`
Salary	not in the list; per job via `/jobs/{id}?pay_transparency=true` (`pay_input_ranges`)	optional `salaryRange`	with `?includeCompensation=true`

A few details decide how you call each one.

Greenhouse keeps the list light: title, a free-text location, a URL and dates. The list has no department names. You can get them two ways: ?content=true swaps the light list for a much heavier one that carries every job description, or a separate /departments call nests the jobs under their departments. The script below leaves department empty for Greenhouse; if you need it, call /departments once per board and join on the job id. Also note that absolute_url can point at the company's own careers site rather than greenhouse.io.

Lever returns a bare array, and every posting carries its full description in several forms. In this run, 321 postings came to 6.19 MB. The documented limit and skip parameters let you page or probe a board without downloading all of it. Boards on Lever's EU instance live on api.eu.lever.co, not api.lever.co. The docs list workplaceType values as on-site, remote, hybrid or unspecified, but the board used here returned onsite without the hyphen, so the script accepts both. createdAt isn't in the README's field table, but it is in the responses as epoch milliseconds.

Ashby always includes the HTML and plain-text description. With ?includeCompensation=true it adds a compensation object with per-tier details and a summaryComponents list. That list isn't ordered by type: on the first Ramp posting in this run, the equity component came before the salary one, so you need to search for compensationType == "Salary" and not take the first item. Ashby also has isListed (skip false) and two workplace signals that can disagree. The same Ramp posting had workplaceType: "Hybrid" and isRemote: true. The script keeps workplaceType.

The script

Every posting becomes one flat row with the same 13 keys: source, board, job_id, title, location, workplace, department, apply_url, posted_at, and four salary_* fields. Anything a vendor doesn't provide stays None.

The endpoints:

ENDPOINTS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever": "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true",
}

The fetch step keeps three outcomes apart. A 404 means no board with that slug exists on that vendor. A 200 with an empty list means the board exists but has nothing open. A 429, 5xx or network error (including a connection that drops mid-download) means you don't know yet. Retry a few times with a growing wait, honour Retry-After, then give up and say the status is unknown. Never record that case as "company not found".

def fetch(url, tries=4):
    """Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors."""
    for attempt in range(1, tries + 1):
        time.sleep(1)  # never more than one request per second
        try:
            request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
            with urllib.request.urlopen(request, timeout=60) as response:
                body = response.read()
                return json.loads(body), len(body)
        except (OSError, http.client.HTTPException) as err:  # HTTP errors, timeouts, dropped reads
            code = getattr(err, "code", None)
            if code == 404:
                return None, 0
            if (code and code != 429 and code < 500) or attempt == tries:
                raise
            header = err.headers.get("Retry-After", "") if code else ""
            wait = min(int(header), 60) if header.isdigit() else 2 ** attempt
            print(f"  retry {attempt}/{tries - 1} in {wait}s: {url}", file=sys.stderr)
            time.sleep(wait)

The mapping itself is one function with a branch per vendor:

def normalize(source, job):
    """Map one raw posting to the shared field names. Anything a vendor lacks stays None."""
    if source == "greenhouse":  # the /jobs list has no workplace, department or salary
        return {"job_id": job["id"], "title": job.get("title"),
                "location": (job.get("location") or {}).get("name"),
                "apply_url": job.get("absolute_url"), "posted_at": job.get("first_published")}
    if source == "lever":
        cats, pay = job.get("categories") or {}, job.get("salaryRange") or {}
        return {"job_id": job["id"], "title": job.get("text"), "location": cats.get("location"),
                "workplace": job.get("workplaceType"),  # docs: on-site/remote/hybrid/unspecified
                "department": cats.get("department") or cats.get("team"),
                "apply_url": job.get("applyUrl") or job.get("hostedUrl"),
                "posted_at": job.get("createdAt"),  # epoch milliseconds
                "salary_min": pay.get("min"), "salary_max": pay.get("max"),
                "salary_currency": pay.get("currency"), "salary_interval": pay.get("interval")}
    parts = (job.get("compensation") or {}).get("summaryComponents") or []  # Ashby
    pay = next((p for p in parts if p.get("compensationType") == "Salary"), {})  # any order
    return {"job_id": job["id"], "title": job.get("title"), "location": job.get("location"),
            "workplace": job.get("workplaceType"), "department": job.get("department"),
            "apply_url": job.get("applyUrl") or job.get("jobUrl"),
            "posted_at": job.get("publishedAt"),
            "salary_min": pay.get("minValue"), "salary_max": pay.get("maxValue"),
            "salary_currency": pay.get("currencyCode"), "salary_interval": pay.get("interval")}

After mapping, each row gets the same cleanup: whitespace is trimmed and collapsed, dates are converted to UTC, and the workplace value is reduced to onsite, hybrid, remote or None. The whitespace step matters because titles arrive with stray spaces. Ramp's board, for example, sends " Security Engineer, Cloud" with a leading space, which would otherwise break exact matching and de-duplication.

        row.update(job_id=str(row["job_id"]), posted_at=iso_utc(row["posted_at"]),
                   **{key: " ".join(row[key].split()) or None  # trim, collapse whitespace
                      for key in ("title", "location", "department") if isinstance(row[key], str)})
        kind = str(row["workplace"]).lower().replace("-", "")  # "OnSite", "on-site", "onsite"
        row["workplace"] = kind if kind in ("onsite", "hybrid", "remote") else None

The full script is 122 lines and needs Python 3.11 or newer, with no third-party packages:

normalize_jobs.py (full script)

"""Normalize public Greenhouse, Lever and Ashby job boards into one JSON and one CSV.
Usage: python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp
Python 3.11+, standard library only, no API key (these are public job-board APIs).
"""
import csv
import http.client
import json
import sys
import time
import urllib.request
from datetime import datetime, timezone

USER_AGENT = "job-board-normalizer-example/1.0 (tutorial script; one request per board)"
ENDPOINTS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever": "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true",
}
FIELDS = ("source board job_id title location workplace department apply_url posted_at "
          "salary_min salary_max salary_currency salary_interval").split()
UTC = "%Y-%m-%dT%H:%M:%SZ"


def fetch(url, tries=4):
    """Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors."""
    for attempt in range(1, tries + 1):
        time.sleep(1)  # never more than one request per second
        try:
            request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
            with urllib.request.urlopen(request, timeout=60) as response:
                body = response.read()
                return json.loads(body), len(body)
        except (OSError, http.client.HTTPException) as err:  # HTTP errors, timeouts, dropped reads
            code = getattr(err, "code", None)
            if code == 404:
                return None, 0
            if (code and code != 429 and code < 500) or attempt == tries:
                raise
            header = err.headers.get("Retry-After", "") if code else ""
            wait = min(int(header), 60) if header.isdigit() else 2 ** attempt
            print(f"  retry {attempt}/{tries - 1} in {wait}s: {url}", file=sys.stderr)
            time.sleep(wait)


def iso_utc(value):
    """ISO 8601 with an offset, or epoch milliseconds (Lever) -> 'YYYY-MM-DDTHH:MM:SSZ'."""
    if isinstance(value, (int, float)):
        return datetime.fromtimestamp(value / 1000, timezone.utc).strftime(UTC)
    return datetime.fromisoformat(value).astimezone(timezone.utc).strftime(UTC) if value else None


def normalize(source, job):
    """Map one raw posting to the shared field names. Anything a vendor lacks stays None."""
    if source == "greenhouse":  # the /jobs list has no workplace, department or salary
        return {"job_id": job["id"], "title": job.get("title"),
                "location": (job.get("location") or {}).get("name"),
                "apply_url": job.get("absolute_url"), "posted_at": job.get("first_published")}
    if source == "lever":
        cats, pay = job.get("categories") or {}, job.get("salaryRange") or {}
        return {"job_id": job["id"], "title": job.get("text"), "location": cats.get("location"),
                "workplace": job.get("workplaceType"),  # docs: on-site/remote/hybrid/unspecified
                "department": cats.get("department") or cats.get("team"),
                "apply_url": job.get("applyUrl") or job.get("hostedUrl"),
                "posted_at": job.get("createdAt"),  # epoch milliseconds
                "salary_min": pay.get("min"), "salary_max": pay.get("max"),
                "salary_currency": pay.get("currency"), "salary_interval": pay.get("interval")}
    parts = (job.get("compensation") or {}).get("summaryComponents") or []  # Ashby
    pay = next((p for p in parts if p.get("compensationType") == "Salary"), {})  # any order
    return {"job_id": job["id"], "title": job.get("title"), "location": job.get("location"),
            "workplace": job.get("workplaceType"), "department": job.get("department"),
            "apply_url": job.get("applyUrl") or job.get("jobUrl"),
            "posted_at": job.get("publishedAt"),
            "salary_min": pay.get("minValue"), "salary_max": pay.get("maxValue"),
            "salary_currency": pay.get("currencyCode"), "salary_interval": pay.get("interval")}


def collect(spec):
    """'lever:palantir' -> list of normalized rows. Problems print a message and return []."""
    source, _, board = spec.partition(":")
    if source not in ENDPOINTS or not board.replace("-", "").replace("_", "").isalnum():
        print(f"{spec}: expected greenhouse:<slug>, lever:<slug> or ashby:<slug>")
        return []
    try:
        data, size = fetch(ENDPOINTS[source].format(board))
    except (OSError, http.client.HTTPException, ValueError) as err:  # ValueError: bad JSON
        print(f"{spec}: request failed ({err}) - status unknown, try again later")
        return []
    if data is None:
        print(f"{spec}: board not found (HTTP 404) - check the slug and the vendor")
        return []
    jobs = [job for job in (data if source == "lever" else data.get("jobs", []))
            if job.get("isListed", True)]  # Ashby can mark a job as unlisted
    if not jobs:
        print(f"{spec}: board exists but has no open jobs right now")
        return []
    rows = []
    for job in jobs:
        row = dict.fromkeys(FIELDS)  # every row gets every column
        row.update(normalize(source, job), source=source, board=board)
        row.update(job_id=str(row["job_id"]), posted_at=iso_utc(row["posted_at"]),
                   **{key: " ".join(row[key].split()) or None  # trim, collapse whitespace
                      for key in ("title", "location", "department") if isinstance(row[key], str)})
        kind = str(row["workplace"]).lower().replace("-", "")  # "OnSite", "on-site", "onsite"
        row["workplace"] = kind if kind in ("onsite", "hybrid", "remote") else None
        rows.append(row)
    filled = {key: sum(row[key] is not None for row in rows)
              for key in ("location", "workplace", "department", "salary_min")}
    print(f"{spec}: {len(rows)} jobs, {size / 1e6:.2f} MB, non-empty: {filled}")
    return rows


if __name__ == "__main__":
    captured_at = datetime.now(timezone.utc).strftime(UTC)
    print(f"capture started at {captured_at}")
    rows = [row for spec in sys.argv[1:] for row in collect(spec)]
    with open("jobs.json", "w", encoding="utf-8") as f:
        json.dump({"captured_at": captured_at, "jobs": rows}, f, ensure_ascii=False, indent=2)
    with open("jobs.csv", "w", newline="", encoding="utf-8-sig") as f:  # BOM: Excel sees UTF-8
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)
    print(f"wrote {len(rows)} rows to jobs.json and jobs.csv")

Real results

Here is the run against one public board per vendor, plus a made-up Lever slug to show the 404 path:

$ python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp lever:no-such-board-20260927
capture started at 2026-09-27T12:06:53Z
greenhouse:stripe: 701 jobs, 0.44 MB, non-empty: {'location': 701, 'workplace': 0, 'department': 0, 'salary_min': 0}
lever:palantir: 321 jobs, 6.19 MB, non-empty: {'location': 321, 'workplace': 321, 'department': 321, 'salary_min': 0}
ashby:ramp: 158 jobs, 2.86 MB, non-empty: {'location': 158, 'workplace': 158, 'department': 158, 'salary_min': 151}
lever:no-such-board-20260927: board not found (HTTP 404) - check the slug and the vendor
wrote 1180 rows to jobs.json and jobs.csv

This capture ran on 27 September 2026; afterwards the script gained one fix, retrying a connection that drops mid-download, which does not change the output above. None of these boards was empty or rate-limited in this run. The "no open jobs" and back-off paths were checked with offline unit tests that fake the HTTP responses (12 tests, all passing), and so was Lever's salaryRange mapping, since no Palantir posting had one.

The table below shows the first three distinct "Engineer" titles per source, in response order.

Captured 2026-09-27 12:06:53 UTC. Postings change constantly, and some of these may already be closed. The companies are examples of public boards that responded on 27 September 2026. This article is not affiliated with or endorsed by them or by Greenhouse, Lever or Ashby.

Source	Board	Title	Location	Apply link
Greenhouse	stripe	Abuse Research Engineer	Remote from the US	stripe.com
Greenhouse	stripe	AI Engineer	Chicago	stripe.com
Greenhouse	stripe	Android BSP Engineer	Taipei, Taiwan	stripe.com
Lever	palantir	Backend Software Engineer - Application Development	London, United Kingdom	jobs.lever.co
Lever	palantir	Backend Software Engineer - Defense	Washington, D.C.	jobs.lever.co
Lever	palantir	Backend Software Engineer - Infrastructure	New York, NY	jobs.lever.co
Ashby	ramp	Security Engineer, Cloud	New York, NY (HQ)	jobs.ashbyhq.com
Ashby	ramp	Mobile Engineer, Android	New York, NY (HQ)	jobs.ashbyhq.com
Ashby	ramp	Software Engineer, Frontend	New York, NY (HQ)	jobs.ashbyhq.com

Exporting JSON and CSV

The last block of the script writes both files. The JSON keeps the capture time next to the rows, so a saved file always says when it was taken:

    with open("jobs.json", "w", encoding="utf-8") as f:
        json.dump({"captured_at": captured_at, "jobs": rows}, f, ensure_ascii=False, indent=2)
    with open("jobs.csv", "w", newline="", encoding="utf-8-sig") as f:  # BOM: Excel sees UTF-8
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)

One row from this run's jobs.json:

{
  "source": "ashby",
  "board": "ramp",
  "job_id": "34413f8d-26bf-4bbc-8ade-eb309a0e2245",
  "title": "Security Engineer, Cloud",
  "location": "New York, NY (HQ)",
  "workplace": "hybrid",
  "department": "Engineering",
  "apply_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application",
  "posted_at": "2026-04-07T17:12:35Z",
  "salary_min": 211400,
  "salary_max": 290600,
  "salary_currency": "USD",
  "salary_interval": "1 YEAR"
}

Three details make the CSV behave:

utf-8-sig writes a byte-order mark so Excel opens the file as UTF-8. Nine of the titles in this run contain an em or en dash (for example "Account Executive, Funded Startups — Hunter, Italy"), and without the BOM Excel can show them as garbage.
newline="" is what the csv docs require. Without it you get blank lines between rows on Windows.
csv.DictWriter quotes commas and quotes for you, and writes None as an empty cell. In this run's file, 1,029 of 1,180 rows have an empty salary_min, which is honest. A 0 or "N/A" there would be wrong.

If you run this on a schedule, de-duplicate on (source, board, job_id). Titles and locations aren't unique, and Palantir alone lists the same title in several cities.

Missing values, source limits and common mistakes

Missing values are information. Keep them as None or empty. The non-empty counts in the run log show where the gaps are: Greenhouse gave no workplace for any of its 701 rows, but 109 of its location strings contain the word "remote". If you infer a workplace from text, put it in a separate column so the reader can tell reported values from guessed ones. On Ashby, 151 of 158 Ramp postings published a salary. The currencies were USD (140), CAD (6), GBP (4) and SEK (1), and the intervals were 1 YEAR (147) and 1 MONTH (4).

Source limits:

It's a snapshot. Counts and postings change during the day.
Only three vendors are covered. Many employers use other systems or their own careers site, so a 404 on all three says nothing about whether a company is hiring.
Lever EU boards need api.eu.lever.co, and the script only calls the global host.
Location is free text. Greenhouse strings like "Chicago, IL; Atlanta, GA" pack several places into one field (22 of Stripe's rows in this run contain a ;). Lever's allLocations and Ashby's secondaryLocations hold extra locations that this script doesn't export.
Ashby's summary gives one salary range per posting. When tiers differ by location, keep the raw compensation object if you need per-location pay.
All 701 Greenhouse links in this run point to stripe.com, not greenhouse.io. That is the company's choice, and it is still the right link to send people to.

Common mistakes:

Treating a 429 or a timeout as "board not found", or treating a 200 with an empty list as a 404.
Taking the first Ashby compensation component as the salary.
Reading Lever's createdAt as seconds. datetime.fromtimestamp(1786469891368) fails (with OSError on Windows), so divide by 1000 first.
Comparing salaries without checking currency and interval.
Assuming a slug belongs to the company you meant. A 200 proves a board exists, not whose it is. Take the slug from the company's own careers page.
Fetching too fast. The script waits one second between requests, sends a descriptive User-Agent and fetches each board once per run. These are public boards meant for distribution, so link back to the original posting for applying.

Official sources

Greenhouse Job Board API: https://docs.greenhouse.io/job-board.html
Lever Postings API: https://github.com/lever/postings-api
Ashby public job posting API: https://developers.ashbyhq.com/docs/public-job-posting-api
Python csv module: https://docs.python.org/3/library/csv.html
Python json module: https://docs.python.org/3/library/json.html

This article and its code were drafted by an AI assistant at the account owner's request. The code was run against the live public APIs, and the results and table were captured on 27 September 2026 at 12:06 UTC.