DEV Community

Cover image for Querying 780K+ Live Jobs & Tech Signals Across 7 ATS Platforms Without Headless Browsers (Python + MCP)
Noctua Data
Noctua Data

Posted on Edited on

Querying 780K+ Live Jobs & Tech Signals Across 7 ATS Platforms Without Headless Browsers (Python + MCP)

Most recruiting pipelines and B2B outbound teams treat job postings as static text for candidates. For GTM engineers, however, live job descriptions are one of the highest-signal public technographic datasets on the internet.

When a mid-market company opens a role asking for "Snowflake, dbt, Looker, and HubSpot experience" with a $160,000–$195,000 base salary range, they are publicly broadcasting their active internal stack, budget tier, and departmental headcount expansion.

The bottleneck is extraction: enterprise employers fragment their job boards across Workday, Greenhouse, Ashby, Lever, SmartRecruiters, Recruitee, and Workable. Throwing headless browsers at thousands of career portals burns gigabytes of memory and breaks constantly.

Here is how we unified 4,066+ verified employer boards (~780,000 active jobs, including 1,534 Workday portals) into a single 256 MB async Python & MCP engine with zero browser overhead.


1. Every Major ATS Exposes Clean Public JSON (Even Workday)

You almost never need DOM scraping or Playwright to read an ATS board if you speak its native public feed protocol:

  • Greenhouse: GET https://boards-api.greenhouse.io/v1/boards/{slug}/jobs?content=true — Returns full job descriptions, locations, departments, and publish timestamps in one JSON payload.
  • Ashby: GET https://api.ashbyhq.com/posting-api/job-board/{slug}?includeCompensation=true — Cleanly exposes structured compensation tiers (minValue, maxValue, currencyCode, interval) alongside workplace types.
  • Lever: GET https://api.lever.co/v0/postings/{slug}?mode=json — Returns structured plain-text lists, commitment types, and salary ranges.
  • SmartRecruiters: GET https://api.smartrecruiters.com/v1/companies/{slug}/postings — Paginates active roles with clean location and department metadata.
  • Recruitee & Workable: Expose public /api/offers/ and /spi/v3/accounts/{slug}/jobs endpoints respectively.
  • Workday (myworkdayjobs.com): Often mislabeled as "impossible without Puppeteer." In reality, every Workday tenant (https://{tenant}.wd5.myworkdayjobs.com/{site}) serves a fast internal JSON endpoint at:
  POST /wday/cxs/{tenant}/{site}/jobs
  Content-Type: application/json

  {"appliedFacets": {}, "limit": 20, "offset": 0, "searchText": "Data Engineer"}
Enter fullscreen mode Exit fullscreen mode

Each job's externalPath then resolves via GET /wday/cxs/{tenant}/{site}{externalPath} in ~150ms.


2. Bare Company Domain -> Live ATS Auto-Resolution

In real GTM workflows (like enriching a CRM or Clay table), you rarely start with boards.greenhouse.io/airtable — you start with a bare domain like airtable.com or mistral.ai.

To keep container memory safely below 90 MB inside a 256 MB serverless worker, we split resolution into two stages:

  1. O(1) Compressed Route Index: A gzipped JSONL index of 4,066 pre-verified live employers maps bare company domains directly to (ats_provider, tenant_slug) in < 2ms while keeping resident memory under 2 MB.
  2. Live Career-Page Probing: For unknown domains not in the index, an async HTTP probe checks /careers and /jobs for embedded ATS iframe/script signatures (boards.greenhouse.io, jobs.ashbyhq.com, myworkdayjobs.com, etc.) and verifies company ownership to prevent slug collisions.

3. Deterministic Compensation & 60+ B2B Tech Signals

Instead of paying per-job LLM token costs just to extract tools, regex word-boundary dictionaries with disambiguation guards (e.g., matching Go / Golang or Linear only in engineering contexts rather than generic prose) reliably tag 60+ B2B cloud, data, CRM, security, and AI tools (AWS, Snowflake, Databricks, Terraform, Kubernetes, Salesforce, HubSpot, Gong, Okta, OpenAI, etc.) in microseconds.

Hourly, monthly, and annual pay bands are simultaneously normalized into salaryMin, salaryMax, and currency so downstream filters like minSalary: 140000 work uniformly across all 7 ATS providers.


Mounting Live Hiring Signals into Claude Desktop / Cursor (MCP)

We published this unified engine on Apify Store (Unified ATS Jobs & Hiring Signals), where it can be queried directly via REST API, Clay HTTP enrichment, or mounted into any MCP client alongside our ecommerce and app-churn tools:

{
  "mcpServers": {
    "noctua-b2b-intelligence": {
      "command": "npx",
      "args": [
        "-y",
        "@apify/actors-mcp-server",
        "--actors",
        "kkhome/unified-ats-jobs,kkhome/store-tech-leads,kkhome/shopify-chrome-app-reviews"
      ],
      "env": {
        "APIFY_TOKEN": "<YOUR_APIFY_TOKEN>"
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

Full configuration recipes and Python examples are available in our open integration repo:
👉 GitHub: https://github.com/kkhomej33-netizen/noctua-b2b-mcp

How is your team currently combining hiring intent, technographics, or product-review signals in your GTM or data workflows?

Top comments (0)