DEV Community

Devil Scrapes
Devil Scrapes

Posted on

SmartRecruiters Jobs Scraper: Pull Any Employer's Postings From One Keyless API

SmartRecruiters Jobs Scraper: Pull Any Employer's Postings From One Keyless API

Thousands of mid-size and enterprise employers run their careers page on SmartRecruiters — Visa, Bosch, and a long tail of companies you'd recognize. If you're building recruiting pipelines, a job-board aggregator, or a competitive hiring-intel feed, SmartRecruiters coverage is table stakes. This post walks through what the data looks like, why the naive scraping approach breaks down at scale, and how the SmartRecruiters Jobs Scraper gets you clean rows instead.

Why SmartRecruiters data is worth pulling

Every SmartRecruiters careers board (jobs.smartrecruiters.com/{companyId}) exposes structured posting data — title, location, department, function, employment type, and a canonical apply URL — through the same public JSON API the page itself calls. That's useful for:

  • Recruiters and sourcers tracking open reqs at target employers by function and location
  • Job-board aggregator operators who need SmartRecruiters alongside Workday, Greenhouse, Lever, and Ashby
  • Competitive-intel analysts inferring a rival's growth areas from where they're actively hiring
  • ATS/data-pipeline builders wiring structured job rows into a CRM or workflow tool

The naive approach, and where it falls apart

api.smartrecruiters.com/v1/companies/{companyId}/postings is a real, documented-ish public endpoint, and a single curl against it will return JSON. The friction shows up once you're doing this for more than one company: you need to paginate correctly, apply the server-side filters (q, department, city, country) without silently dropping rows, handle the detail endpoint for full descriptions without doubling your request count unnecessarily, and survive transient 429s across a batch of dozens of employers without the whole run dying on employer #12. None of that is hard in isolation — it's the compounding maintenance burden that gets expensive.

What we built instead

The SmartRecruiters Jobs Scraper hits that same public postings API directly and absorbs the parts that make multi-employer batches fragile: curl-cffi browser-fingerprint impersonation so requests present a real handshake, retries with exponential backoff on 429/5xx, and normalized output regardless of how many company IDs you pass in one run. Server-side filters (searchQuery, department, city, country) narrow the result set before you're billed for a row.

Run it via the Apify Python SDK:

from apify_client import ApifyClient

client = ApifyClient("APIFY_TOKEN")
run = client.actor("DevilScrapes/smartrecruiters-jobs-scraper").call(run_input={
    "companyIds": ["Visa"],
    "maxResultsPerCompany": 25,
    "includeDescription": True,
    "proxyConfiguration": {"useApifyProxy": True},
})

for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    print(item["title"], item["location_city"], item["department"])
Enter fullscreen mode Exit fullscreen mode

Or drop the same JSON straight into the input form on the Store page — no code needed.

Sample output row:

{
  "posting_id": "743999...abc",
  "title": "Sr. Manager, Data Platform",
  "company_id": "Visa",
  "company_name": "Visa",
  "location_city": "Austin",
  "location_region": "Texas",
  "location_country": "US",
  "is_remote": false,
  "department": "Technology",
  "employment_type": "Full-time",
  "apply_url": "https://jobs.smartrecruiters.com/Visa/743999...abc",
  "description_html": "Visa is looking for a Sr. Manager...",
  "scraped_at": "2026-07-21T14:00:00Z"
}
Enter fullscreen mode Exit fullscreen mode

The technically interesting bit

SmartRecruiters' companyId isn't always the display name you'd guess — it's a slug pulled straight from the careers URL, and it's case-sensitive with its own quirks per tenant. Most "generic" scrapers treat this as a solved problem and just string-match against a display name, which silently returns zero results for a lot of real companies. We take the company ID as-is from the URL structure the site itself uses, so jobs.smartrecruiters.com/Visa maps directly to companyIds: ["Visa"] without any guessing layer that can quietly fail.

Pricing — no fine print

Pay-Per-Event: $0.005 per run plus $0.0015 per job posting written to the dataset. A 1,000-posting pull costs $1.51. No matching postings, no charge beyond the small warm-up fee. New Apify accounts get $5 of free credit, no card required.

Honest limitations

We only reach the public, unauthenticated postings boards — not internal employee-only views. includeDescription adds one detail request per posting, so leave it off if you just need metadata fast. And you supply the SmartRecruiters companyId, not a freeform company name — a display name like "Bosch" can differ from its actual slug, so pull the ID straight from the careers URL.

Try it, and the rest of the fleet

SmartRecruiters is one platform in a bigger landscape. We also run the Workday Jobs Scraper, the Workable Jobs Scraper, and the Multi-ATS Jobs Scraper for Greenhouse, Lever, and Ashby in one normalized schema — same keyless, pay-per-result approach across the board.

Grab the SmartRecruiters Jobs Scraper on the Apify Store and try it with the free $5 credit. Found a company that behaves differently, or need a field we don't expose? Open an issue on the Actor's Issues tab — we ship fixes weekly. Browse the rest of the fleet at apify.com/DevilScrapes.

Which employer's SmartRecruiters board would you pull first?

Top comments (0)