DEV Community

Cover image for How to scrape SmartRecruiters jobs data with Python (no API key needed)
Freshactors
Freshactors

Posted on

How to scrape SmartRecruiters jobs data with Python (no API key needed)

#webscraping #python #tutorial #api

SmartRecruiters powers the careers sites of thousands of enterprises (Visa, Bosch, and many more), and every one of those boards is backed by a public Posting API — no API key, no login, no headless browser. In this tutorial we'll pull a company's full job board (titles, locations, employment type, remote flags, and the complete description for every role) as clean structured JSON, in a few lines of Python.

Why not just hit the API yourself?

You can — that's exactly what this actor does under the hood. SmartRecruiters exposes a two-tier public API:

List:   https://api.smartrecruiters.com/v1/companies/{company}/postings
Detail: https://api.smartrecruiters.com/v1/companies/{company}/postings/{id}
Enter fullscreen mode Exit fullscreen mode

The list gives you rich metadata; the full job description lives in the detail endpoint, in HTML jobAd sections. So to get complete records you paginate the list, then fetch each posting's detail and stitch the sections together — and you maintain that two-step parser forever, including the day a field name quietly changes and your pipeline goes empty.

A cleaner path: hand a list of company identifiers to an actor that already does the pagination + detail-enrichment and returns one stable schema — the same schema as Greenhouse, Lever, and Workable. Here's how with the SmartRecruiters Jobs Scraper.

Step 1 — Install the Apify client 

pip install apify-client
Enter fullscreen mode Exit fullscreen mode

Read your Apify API token (Console → Settings → Integrations) from an environment variable:

export APIFY_TOKEN="apify_api_xxx"
Enter fullscreen mode Exit fullscreen mode

Step 2 — Run the actor with a list of companies

companies accepts identifiers (visa) or board URLs (https://jobs.smartrecruiters.com/Visa). The identifier is the company segment in the jobs.smartrecruiters.com URL.

import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

run_input = {
    "companies": ["visa", "https://jobs.smartrecruiters.com/BoschGroup"],
    "includeDescription": True,
    "maxJobsPerCompany": 500,
}

run = client.actor("freshactors/smartrecruiters-jobs-scraper").call(run_input=run_input)
print("Dataset id:", run["defaultDatasetId"])
Enter fullscreen mode Exit fullscreen mode

Step 3 — Read the normalized output

Every posting comes back in the same shape, with null (never missing keys) where SmartRecruiters lacks a field:

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(
        f'{item["company"]:<10} '
        f'{item["title"]}  '
        f'({item.get("workplaceType") or "n/a"}, {item.get("location") or "n/a"})'
    )
Enter fullscreen mode Exit fullscreen mode

A single record:

{
  "_type": "job",
  "_schemaVersion": "1.0",
  "_source": "smartrecruiters",
  "company": "visa",
  "jobId": "744000122509268",
  "title": "Sr. SW Engineer",
  "department": "Technology and Operations",
  "team": "Information Technology",
  "location": "Austin, TX, United States",
  "workplaceType": "hybrid",
  "commitment": "Full-time",
  "country": "US",
  "url": "https://jobs.smartrecruiters.com/visa/744000122509268",
  "applyUrl": "https://jobs.smartrecruiters.com/visa/744000122509268?oga=true",
  "postedAt": "2026-04-23T16:54:54.835Z",
  "descriptionText": "Company Description... Job Description... Qualifications...",
  "_scrapedAt": "2026-06-02T17:00:00.000Z"
}
Enter fullscreen mode Exit fullscreen mode

This is the same record shape the Greenhouse & Lever and Workable scrapers emit — so SmartRecruiters companies drop into the same pipeline with zero special-casing.

Step 4 — Faster, lighter runs

The full description requires a detail call per posting. If you only need titles, departments, and locations — say, for a hiring-signal dashboard — turn it off for a list-only run (metadata + posting URL, one request per page):

run_input = {
    "companies": ["visa", "airbnb"],
    "includeDescription": False,   # list-only, no per-posting detail calls
    "maxJobsPerCompany": 200,
}
Enter fullscreen mode Exit fullscreen mode

maxJobsPerCompany (1–5000) caps volume so a large employer doesn't dominate your run.

Prefer Node.js? 

npm install apify-client
Enter fullscreen mode Exit fullscreen mode 
import { ApifyClient } from 'apify-client';

const client = new ApifyClient({ token: process.env.APIFY_TOKEN });

const run = await client.actor('freshactors/smartrecruiters-jobs-scraper').call({
    companies: ['visa', 'https://jobs.smartrecruiters.com/BoschGroup'],
    includeDescription: true,
    maxJobsPerCompany: 500,
});

const { items } = await client.dataset(run.defaultDatasetId).listItems();
for (const job of items) console.log(`${job.company} — ${job.title} (${job.workplaceType ?? 'n/a'})`);
Enter fullscreen mode Exit fullscreen mode

What about cost?

Pay-per-event: $0.02 per company board fetched and $0.0005 per job posting returned. So 5 companies returning 100 postings total is 5 × $0.02 + 100 × $0.0005 = $0.15. No subscription.

Why use the actor instead of the API directly?

You can call the Posting API yourself. The reason to use the actor is maintenance: it paginates, enriches each posting with its description, normalizes everything into one schema (shared with Greenhouse/Lever/Workable), isolates per-company failures, and is monitored by a daily canary so a silent API change doesn't quietly empty your pipeline.

The actor is here: SmartRecruiters Jobs Scraper on Apify. Point it at your target companies and consume one normalized JSON feed.

Happy scraping.

Top comments (0)