DEV Community

Kata Omel
Kata Omel

Posted on

I scraped 378 company ATS boards to find out who is hiring SDRs (stdlib only, MIT)

#python #webscraping #opensource #startup

A month ago I started building a lead-gen service for companies that hire outbound sales roles.
The core assumption: a company hiring its first SDR needs pipeline before the seat is filled —
and the 60–90 day hiring gap is a service opportunity.

To find those companies I needed one signal: live SDR/BDR job postings, with the company's own job board as the source.
No paid databases. Just public ATS boards.

What I built

A stdlib-only Python scanner (MIT, on GitHub) that:

  1. Takes company slugs from an open dataset of Greenhouse/Lever/Ashby boards.
  2. Hits each board's public JSON API — no HTML scraping, no API keys, no rate-limit games:
    • Greenhouse: boards-api.greenhouse.io/v1/boards/<slug>/jobs
    • Lever: api.lever.co/v0/postings/<slug>?mode=json
    • Ashby: api.ashbyhq.com/posting-api/job-board/<slug>
  3. Filters postings by outbound-sales roles (SDR, BDR, outbound, sales development).
  4. Emits one row per role: company, title, location, board URL, timestamp.

What one full run produced

  • 378 company boards scanned (from an open dataset of 2,600+ boards; many were dead or non-US)
  • 11 fetch errors (timeouts, moved boards) — logged, not retried aggressively
  • 2,661 outbound-role postings at 212 companies

Each row carries its source URL, so every record is auditable. That matters because the #1 complaint
about bought lead lists is "I can't verify where this came from".

Things I got wrong first

  • Generic emails are useless. sales@, info@, careers@ — every reply was an autoresponder. I now mine named emails from the JD text itself (recruiter contacts) and from GitHub commit history of company OSS repos — with a junk filter for template addresses (jsmith@acme.com inside a JD is an example, not a real person — sending there bounces and burns your sender reputation).
  • Job boards lie about freshness. Some postings sat for 6+ months. I keep updated_at per row and only trust postings from the last 60 days as a hiring signal.
  • Gmail cold outreach hits a spam wall. DKIM/SPF/DMARC all pass, but consumer-mailbox cold mail from @gmail.com lands in Spam regardless of content. Lesson learned the expensive way; the fix is a real domain with warm-up, or inbound channels instead.

Try it 

git clone https://github.com/Alex20Sas12/jd-play-scraper
python jd_play.py   # stdlib only, no requirements.txt
Enter fullscreen mode Exit fullscreen mode

If you run it for a different vertical, the role filter is one regex constant at the top.
PRs welcome — especially more ATS providers (Workday's JSON endpoint is the obvious next one,
it just needs a per-tenant URL map).

The service itself (evidence-backed lead lists, $299/100 records, dead-row cash-back) is at
pipelinefactory.vercel.app if you want the done-for-you version.

Top comments (0)