I Tested 16 Job Sources With a Scraping API, here's What Worked!

I Tested 16 Job Sources With a Scraping API. Here's What Worked

Job hunting can get messy quickly.

You find a role on LinkedIn, another on a startup's career page, and a few more on remote job boards. Soon, you have dozens of tabs open and no single place to track everything 😅

I started thinking about building a small tool that collects job listings from different sources and organizes them in one place.

As a product manager working with Spicrawl, I wanted to explore how its web scraping API could help.

In our October 2026 tests, 15 out of 16 job sources returned usable listings. Here's what we learned.

1. Not every website needs a browser

Some job sources provide data through public APIs or RSS feeds. Others rely on JavaScript to display listings.

Here's a snapshot of our results:

Source	What worked	Credits
Remote OK API	100 job items	1
Remotive API	Structured job data	1
LinkedIn single posting	Full job description	1
LinkedIn job search	Rendered listings	3
Y Combinator jobs	Rendered listings	3
Glassdoor jobs	Rendered listings	3
Indeed search	Listings after scrolling	8

These are results from our tests, not guaranteed results for every request.

The takeaway? Start with the simplest request and only add browser rendering when you need it.

2. Collect job listings with Spicrawl

You'll need a Spicrawl API key, which you can get through Spicrawl.

For a JavaScript-rendered job page, you can use:

curl https://api.spicrawl.com/v1/scrape \
  -H "Authorization: Bearer $SPICRAWL_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://www.ycombinator.com/jobs",
    "js_render": true,
    "cache": false
  }'

This loads the page in a browser so its JavaScript can build the listings.

In our October tests, this returned around 28 KB of content for three credits.

For a source that already provides JSON, you can use a plain fetch instead. For example, Remote OK returned 100 items for one credit.

3. What about pages that load more jobs while scrolling?

This was one of the more interesting results from our tests.

Indeed returned only a tiny, 60-byte page when we tried JavaScript rendering alone. The request technically succeeded, but the job listings weren't there.

We added scrolling actions:

{
  "url": "https://www.indeed.com/jobs?q=data+engineer&l=Remote",
  "js_render": true,
  "wait": 2000,
  "actions": [
    {"scroll": {"to_bottom": true}},
    {"scroll": {"to_bottom": true}}
  ]
}

The response grew to around 68 KB of listings. This request cost eight credits because browser actions use the full-browser rendering tier.

A successful HTTP response doesn't always mean you received useful data. Check the returned content and target status before saving results.

4. Turn scraped pages into a job tracker

Once you've collected the page content, you can extract fields such as:

• Job title and company
• Location and employment type
• Salary, when available
• Required skills
• Application URL

Spicrawl's autoparse: true option can return structured data embedded in a page, including schema.org JobPosting data when available.

You can store the results in a database or spreadsheet and build features such as job filters, application tracking, and AI-powered comparisons against your skills.

Just remember that not every page contains every field, and some job listings may be outdated.

5. What we learned

A few lessons stood out:

• Use APIs and feeds when available. They're often cheaper and easier to process.
• Render JavaScript when needed. Some listing pages don't contain their data in the initial HTML.
• Check the actual response. Empty pages can still return HTTP 200.
• Respect website rules. Check terms and robots.txt, follow rate limits, and collect only the data you need.

What would you build?

A job collector is just one use case. The same idea could help organize internships, scholarships, research opportunities, or listings for a job aggregator.

If you were building a job-hunting assistant, what would you want it to do first?

Find relevant jobs, compare opportunities, match jobs to your skills, or track applications?

I'm exploring these use cases while working on Spicrawl, and I'd genuinely love to hear your ideas.

Want to explore more job boards? Check out the Spicrawl job-postings scraping guide for tested API requests, credit costs, JavaScript rendering, and tips for scraping different job sources.

Explore Spicrawl currently free during beta, with no credit card required.

Comments

[Bipul Nath]

Are they providing any free credits?