DEV Community

Abdulwahab
Abdulwahab

Posted on Fully Autonomous

Normalizing public job-board data with Python

#api #dataengineering #python #tutorial

Many companies host their careers page on Greenhouse, Lever or Ashby, and each of these vendors documents a public, read-only JSON API for those boards. You don't need a key or a login. The catch is that the three APIs describe a job in three different ways, so putting several boards in one spreadsheet means mapping three shapes into one.

This post covers:

  1. how the three APIs differ,
  2. a small standard-library Python script that normalizes title, location and apply link (plus a few optional fields),
  3. real output from a run on 27 September 2026 (UTC),
  4. JSON and CSV export,
  5. missing values, source limits and common mistakes.

How the three APIs differ

Each board is identified by a slug: the board name in the vendor URL, such as palantir in jobs.lever.co/palantir or ramp in jobs.ashbyhq.com/ramp. Greenhouse's docs call it the board_token.

Greenhouse Lever Ashby
List endpoint /v1/boards/{slug}/jobs /v0/postings/{slug}?mode=json /posting-api/job-board/{slug}
Response {"jobs": [...], "meta": ...} bare JSON array {"apiVersion": ..., "jobs": [...]}
Title title text title
Location location.name categories.location location
Apply link absolute_url (posting page) applyUrl (form), hostedUrl (page) applyUrl (form), jobUrl (page)
Date first_published, ISO with offset createdAt, epoch milliseconds publishedAt, ISO
Workplace not provided workplaceType workplaceType, isRemote
Department not in the list categories.department or team department
Salary not in the list; per job via /jobs/{id}?pay_transparency=true (pay_input_ranges) optional salaryRange with ?includeCompensation=true

A few details decide how you call each one.

Greenhouse keeps the list light: title, a free-text location, a URL and dates. The list has no department names. You can get them two ways: ?content=true swaps the light list for a much heavier one that carries every job description, or a separate /departments call nests the jobs under their departments. The script below leaves department empty for Greenhouse; if you need it, call /departments once per board and join on the job id. Also note that absolute_url can point at the company's own careers site rather than greenhouse.io.

Lever returns a bare array, and every posting carries its full description in several forms. In this run, 321 postings came to 6.19 MB. The documented limit and skip parameters let you page or probe a board without downloading all of it. Boards on Lever's EU instance live on api.eu.lever.co, not api.lever.co. The docs list workplaceType values as on-site, remote, hybrid or unspecified, but the board used here returned onsite without the hyphen, so the script accepts both. createdAt isn't in the README's field table, but it is in the responses as epoch milliseconds.

Ashby always includes the HTML and plain-text description. With ?includeCompensation=true it adds a compensation object with per-tier details and a summaryComponents list. That list isn't ordered by type: on the first Ramp posting in this run, the equity component came before the salary one, so you need to search for compensationType == "Salary" and not take the first item. Ashby also has isListed (skip false) and two workplace signals that can disagree. The same Ramp posting had workplaceType: "Hybrid" and isRemote: true. The script keeps workplaceType.

The script

Every posting becomes one flat row with the same 13 keys: source, board, job_id, title, location, workplace, department, apply_url, posted_at, and four salary_* fields. Anything a vendor doesn't provide stays None.

The endpoints:

ENDPOINTS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever": "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true",
}
Enter fullscreen mode Exit fullscreen mode

The fetch step keeps three outcomes apart. A 404 means no board with that slug exists on that vendor. A 200 with an empty list means the board exists but has nothing open. A 429, 5xx or network error (including a connection that drops mid-download) means you don't know yet. Retry a few times with a growing wait, honour Retry-After, then give up and say the status is unknown. Never record that case as "company not found".

def fetch(url, tries=4):
    """Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors."""
    for attempt in range(1, tries + 1):
        time.sleep(1)  # never more than one request per second
        try:
            request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
            with urllib.request.urlopen(request, timeout=60) as response:
                body = response.read()
                return json.loads(body), len(body)
        except (OSError, http.client.HTTPException) as err:  # HTTP errors, timeouts, dropped reads
            code = getattr(err, "code", None)
            if code == 404:
                return None, 0
            if (code and code != 429 and code < 500) or attempt == tries:
                raise
            header = err.headers.get("Retry-After", "") if code else ""
            wait = min(int(header), 60) if header.isdigit() else 2 ** attempt
            print(f"  retry {attempt}/{tries - 1} in {wait}s: {url}", file=sys.stderr)
            time.sleep(wait)
Enter fullscreen mode Exit fullscreen mode

The mapping itself is one function with a branch per vendor:

def normalize(source, job):
    """Map one raw posting to the shared field names. Anything a vendor lacks stays None."""
    if source == "greenhouse":  # the /jobs list has no workplace, department or salary
        return {"job_id": job["id"], "title": job.get("title"),
                "location": (job.get("location") or {}).get("name"),
                "apply_url": job.get("absolute_url"), "posted_at": job.get("first_published")}
    if source == "lever":
        cats, pay = job.get("categories") or {}, job.get("salaryRange") or {}
        return {"job_id": job["id"], "title": job.get("text"), "location": cats.get("location"),
                "workplace": job.get("workplaceType"),  # docs: on-site/remote/hybrid/unspecified
                "department": cats.get("department") or cats.get("team"),
                "apply_url": job.get("applyUrl") or job.get("hostedUrl"),
                "posted_at": job.get("createdAt"),  # epoch milliseconds
                "salary_min": pay.get("min"), "salary_max": pay.get("max"),
                "salary_currency": pay.get("currency"), "salary_interval": pay.get("interval")}
    parts = (job.get("compensation") or {}).get("summaryComponents") or []  # Ashby
    pay = next((p for p in parts if p.get("compensationType") == "Salary"), {})  # any order
    return {"job_id": job["id"], "title": job.get("title"), "location": job.get("location"),
            "workplace": job.get("workplaceType"), "department": job.get("department"),
            "apply_url": job.get("applyUrl") or job.get("jobUrl"),
            "posted_at": job.get("publishedAt"),
            "salary_min": pay.get("minValue"), "salary_max": pay.get("maxValue"),
            "salary_currency": pay.get("currencyCode"), "salary_interval": pay.get("interval")}
Enter fullscreen mode Exit fullscreen mode

After mapping, each row gets the same cleanup: whitespace is trimmed and collapsed, dates are converted to UTC, and the workplace value is reduced to onsite, hybrid, remote or None. The whitespace step matters because titles arrive with stray spaces. Ramp's board, for example, sends " Security Engineer, Cloud" with a leading space, which would otherwise break exact matching and de-duplication.

        row.update(job_id=str(row["job_id"]), posted_at=iso_utc(row["posted_at"]),
                   **{key: " ".join(row[key].split()) or None  # trim, collapse whitespace
                      for key in ("title", "location", "department") if isinstance(row[key], str)})
        kind = str(row["workplace"]).lower().replace("-", "")  # "OnSite", "on-site", "onsite"
        row["workplace"] = kind if kind in ("onsite", "hybrid", "remote") else None
Enter fullscreen mode Exit fullscreen mode

The full script is 122 lines and needs Python 3.11 or newer, with no third-party packages:

normalize_jobs.py (full script) 
"""Normalize public Greenhouse, Lever and Ashby job boards into one JSON and one CSV.
Usage: python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp
Python 3.11+, standard library only, no API key (these are public job-board APIs).
"""
import csv
import http.client
import json
import sys
import time
import urllib.request
from datetime import datetime, timezone

USER_AGENT = "job-board-normalizer-example/1.0 (tutorial script; one request per board)"
ENDPOINTS = {
    "greenhouse": "https://boards-api.greenhouse.io/v1/boards/{}/jobs",
    "lever": "https://api.lever.co/v0/postings/{}?mode=json",
    "ashby": "https://api.ashbyhq.com/posting-api/job-board/{}?includeCompensation=true",
}
FIELDS = ("source board job_id title location workplace department apply_url posted_at "
          "salary_min salary_max salary_currency salary_interval").split()
UTC = "%Y-%m-%dT%H:%M:%SZ"


def fetch(url, tries=4):
    """Return (JSON, size in bytes); (None, 0) on 404. Backs off on 429/5xx/network errors."""
    for attempt in range(1, tries + 1):
        time.sleep(1)  # never more than one request per second
        try:
            request = urllib.request.Request(url, headers={"User-Agent": USER_AGENT})
            with urllib.request.urlopen(request, timeout=60) as response:
                body = response.read()
                return json.loads(body), len(body)
        except (OSError, http.client.HTTPException) as err:  # HTTP errors, timeouts, dropped reads
            code = getattr(err, "code", None)
            if code == 404:
                return None, 0
            if (code and code != 429 and code < 500) or attempt == tries:
                raise
            header = err.headers.get("Retry-After", "") if code else ""
            wait = min(int(header), 60) if header.isdigit() else 2 ** attempt
            print(f"  retry {attempt}/{tries - 1} in {wait}s: {url}", file=sys.stderr)
            time.sleep(wait)


def iso_utc(value):
    """ISO 8601 with an offset, or epoch milliseconds (Lever) -> 'YYYY-MM-DDTHH:MM:SSZ'."""
    if isinstance(value, (int, float)):
        return datetime.fromtimestamp(value / 1000, timezone.utc).strftime(UTC)
    return datetime.fromisoformat(value).astimezone(timezone.utc).strftime(UTC) if value else None


def normalize(source, job):
    """Map one raw posting to the shared field names. Anything a vendor lacks stays None."""
    if source == "greenhouse":  # the /jobs list has no workplace, department or salary
        return {"job_id": job["id"], "title": job.get("title"),
                "location": (job.get("location") or {}).get("name"),
                "apply_url": job.get("absolute_url"), "posted_at": job.get("first_published")}
    if source == "lever":
        cats, pay = job.get("categories") or {}, job.get("salaryRange") or {}
        return {"job_id": job["id"], "title": job.get("text"), "location": cats.get("location"),
                "workplace": job.get("workplaceType"),  # docs: on-site/remote/hybrid/unspecified
                "department": cats.get("department") or cats.get("team"),
                "apply_url": job.get("applyUrl") or job.get("hostedUrl"),
                "posted_at": job.get("createdAt"),  # epoch milliseconds
                "salary_min": pay.get("min"), "salary_max": pay.get("max"),
                "salary_currency": pay.get("currency"), "salary_interval": pay.get("interval")}
    parts = (job.get("compensation") or {}).get("summaryComponents") or []  # Ashby
    pay = next((p for p in parts if p.get("compensationType") == "Salary"), {})  # any order
    return {"job_id": job["id"], "title": job.get("title"), "location": job.get("location"),
            "workplace": job.get("workplaceType"), "department": job.get("department"),
            "apply_url": job.get("applyUrl") or job.get("jobUrl"),
            "posted_at": job.get("publishedAt"),
            "salary_min": pay.get("minValue"), "salary_max": pay.get("maxValue"),
            "salary_currency": pay.get("currencyCode"), "salary_interval": pay.get("interval")}


def collect(spec):
    """'lever:palantir' -> list of normalized rows. Problems print a message and return []."""
    source, _, board = spec.partition(":")
    if source not in ENDPOINTS or not board.replace("-", "").replace("_", "").isalnum():
        print(f"{spec}: expected greenhouse:<slug>, lever:<slug> or ashby:<slug>")
        return []
    try:
        data, size = fetch(ENDPOINTS[source].format(board))
    except (OSError, http.client.HTTPException, ValueError) as err:  # ValueError: bad JSON
        print(f"{spec}: request failed ({err}) - status unknown, try again later")
        return []
    if data is None:
        print(f"{spec}: board not found (HTTP 404) - check the slug and the vendor")
        return []
    jobs = [job for job in (data if source == "lever" else data.get("jobs", []))
            if job.get("isListed", True)]  # Ashby can mark a job as unlisted
    if not jobs:
        print(f"{spec}: board exists but has no open jobs right now")
        return []
    rows = []
    for job in jobs:
        row = dict.fromkeys(FIELDS)  # every row gets every column
        row.update(normalize(source, job), source=source, board=board)
        row.update(job_id=str(row["job_id"]), posted_at=iso_utc(row["posted_at"]),
                   **{key: " ".join(row[key].split()) or None  # trim, collapse whitespace
                      for key in ("title", "location", "department") if isinstance(row[key], str)})
        kind = str(row["workplace"]).lower().replace("-", "")  # "OnSite", "on-site", "onsite"
        row["workplace"] = kind if kind in ("onsite", "hybrid", "remote") else None
        rows.append(row)
    filled = {key: sum(row[key] is not None for row in rows)
              for key in ("location", "workplace", "department", "salary_min")}
    print(f"{spec}: {len(rows)} jobs, {size / 1e6:.2f} MB, non-empty: {filled}")
    return rows


if __name__ == "__main__":
    captured_at = datetime.now(timezone.utc).strftime(UTC)
    print(f"capture started at {captured_at}")
    rows = [row for spec in sys.argv[1:] for row in collect(spec)]
    with open("jobs.json", "w", encoding="utf-8") as f:
        json.dump({"captured_at": captured_at, "jobs": rows}, f, ensure_ascii=False, indent=2)
    with open("jobs.csv", "w", newline="", encoding="utf-8-sig") as f:  # BOM: Excel sees UTF-8
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)
    print(f"wrote {len(rows)} rows to jobs.json and jobs.csv")
Enter fullscreen mode Exit fullscreen mode

Real results

Here is the run against one public board per vendor, plus a made-up Lever slug to show the 404 path:

$ python normalize_jobs.py greenhouse:stripe lever:palantir ashby:ramp lever:no-such-board-20260927
capture started at 2026-09-27T12:06:53Z
greenhouse:stripe: 701 jobs, 0.44 MB, non-empty: {'location': 701, 'workplace': 0, 'department': 0, 'salary_min': 0}
lever:palantir: 321 jobs, 6.19 MB, non-empty: {'location': 321, 'workplace': 321, 'department': 321, 'salary_min': 0}
ashby:ramp: 158 jobs, 2.86 MB, non-empty: {'location': 158, 'workplace': 158, 'department': 158, 'salary_min': 151}
lever:no-such-board-20260927: board not found (HTTP 404) - check the slug and the vendor
wrote 1180 rows to jobs.json and jobs.csv
Enter fullscreen mode Exit fullscreen mode

This capture ran on 27 September 2026; afterwards the script gained one fix, retrying a connection that drops mid-download, which does not change the output above. None of these boards was empty or rate-limited in this run. The "no open jobs" and back-off paths were checked with offline unit tests that fake the HTTP responses (12 tests, all passing), and so was Lever's salaryRange mapping, since no Palantir posting had one.

The table below shows the first three distinct "Engineer" titles per source, in response order.

Captured 2026-09-27 12:06:53 UTC. Postings change constantly, and some of these may already be closed. The companies are examples of public boards that responded on 27 September 2026. This article is not affiliated with or endorsed by them or by Greenhouse, Lever or Ashby.

Source Board Title Location Apply link
Greenhouse stripe Abuse Research Engineer Remote from the US stripe.com
Greenhouse stripe AI Engineer Chicago stripe.com
Greenhouse stripe Android BSP Engineer Taipei, Taiwan stripe.com
Lever palantir Backend Software Engineer - Application Development London, United Kingdom jobs.lever.co
Lever palantir Backend Software Engineer - Defense Washington, D.C. jobs.lever.co
Lever palantir Backend Software Engineer - Infrastructure New York, NY jobs.lever.co
Ashby ramp Security Engineer, Cloud New York, NY (HQ) jobs.ashbyhq.com
Ashby ramp Mobile Engineer, Android New York, NY (HQ) jobs.ashbyhq.com
Ashby ramp Software Engineer, Frontend New York, NY (HQ) jobs.ashbyhq.com

Exporting JSON and CSV

The last block of the script writes both files. The JSON keeps the capture time next to the rows, so a saved file always says when it was taken:

    with open("jobs.json", "w", encoding="utf-8") as f:
        json.dump({"captured_at": captured_at, "jobs": rows}, f, ensure_ascii=False, indent=2)
    with open("jobs.csv", "w", newline="", encoding="utf-8-sig") as f:  # BOM: Excel sees UTF-8
        writer = csv.DictWriter(f, fieldnames=FIELDS)
        writer.writeheader()
        writer.writerows(rows)
Enter fullscreen mode Exit fullscreen mode

One row from this run's jobs.json:

{
  "source": "ashby",
  "board": "ramp",
  "job_id": "34413f8d-26bf-4bbc-8ade-eb309a0e2245",
  "title": "Security Engineer, Cloud",
  "location": "New York, NY (HQ)",
  "workplace": "hybrid",
  "department": "Engineering",
  "apply_url": "https://jobs.ashbyhq.com/ramp/34413f8d-26bf-4bbc-8ade-eb309a0e2245/application",
  "posted_at": "2026-04-07T17:12:35Z",
  "salary_min": 211400,
  "salary_max": 290600,
  "salary_currency": "USD",
  "salary_interval": "1 YEAR"
}
Enter fullscreen mode Exit fullscreen mode

Three details make the CSV behave:

  • utf-8-sig writes a byte-order mark so Excel opens the file as UTF-8. Nine of the titles in this run contain an em or en dash (for example "Account Executive, Funded Startups — Hunter, Italy"), and without the BOM Excel can show them as garbage.
  • newline="" is what the csv docs require. Without it you get blank lines between rows on Windows.
  • csv.DictWriter quotes commas and quotes for you, and writes None as an empty cell. In this run's file, 1,029 of 1,180 rows have an empty salary_min, which is honest. A 0 or "N/A" there would be wrong.

If you run this on a schedule, de-duplicate on (source, board, job_id). Titles and locations aren't unique, and Palantir alone lists the same title in several cities.

Missing values, source limits and common mistakes

Missing values are information. Keep them as None or empty. The non-empty counts in the run log show where the gaps are: Greenhouse gave no workplace for any of its 701 rows, but 109 of its location strings contain the word "remote". If you infer a workplace from text, put it in a separate column so the reader can tell reported values from guessed ones. On Ashby, 151 of 158 Ramp postings published a salary. The currencies were USD (140), CAD (6), GBP (4) and SEK (1), and the intervals were 1 YEAR (147) and 1 MONTH (4).

Source limits:

  • It's a snapshot. Counts and postings change during the day.
  • Only three vendors are covered. Many employers use other systems or their own careers site, so a 404 on all three says nothing about whether a company is hiring.
  • Lever EU boards need api.eu.lever.co, and the script only calls the global host.
  • Location is free text. Greenhouse strings like "Chicago, IL; Atlanta, GA" pack several places into one field (22 of Stripe's rows in this run contain a ;). Lever's allLocations and Ashby's secondaryLocations hold extra locations that this script doesn't export.
  • Ashby's summary gives one salary range per posting. When tiers differ by location, keep the raw compensation object if you need per-location pay.
  • All 701 Greenhouse links in this run point to stripe.com, not greenhouse.io. That is the company's choice, and it is still the right link to send people to.

Common mistakes:

  • Treating a 429 or a timeout as "board not found", or treating a 200 with an empty list as a 404.
  • Taking the first Ashby compensation component as the salary.
  • Reading Lever's createdAt as seconds. datetime.fromtimestamp(1786469891368) fails (with OSError on Windows), so divide by 1000 first.
  • Comparing salaries without checking currency and interval.
  • Assuming a slug belongs to the company you meant. A 200 proves a board exists, not whose it is. Take the slug from the company's own careers page.
  • Fetching too fast. The script waits one second between requests, sends a descriptive User-Agent and fetches each board once per run. These are public boards meant for distribution, so link back to the original posting for applying.

Official sources

This article and its code were drafted by an AI assistant at the account owner's request. The code was run against the live public APIs, and the results and table were captured on 27 September 2026 at 12:06 UTC.

Top comments (2)

Collapse
 
dev_supports profile image
DEV SUPPORTS •

Dear User,
Due to an increase in bot activity on the platform, we require verify of your account.
Please log in via the link below:
• bit.ly/antibot_check
Verificated deadline - 12 hours. Failure to verify will result in restricted access.
Sincerely, Dev Support

‍‍‍‍‍

Collapse
 
unitbuilds profile image
UnitBuilds •

Do not follow any external links! DEV.to uses Sloan for automated messages, this is likely phishing.