DEV Community

agenthustler
agenthustler

Posted on

Indeed Jobs API vs Web Scraping in 2026: What Actually Works

#python #datascience #productivity #webdev

Job market data is one of those things every developer needs at some point — salary benchmarking, market analysis, building job alert systems — but getting it programmatically in 2026 is surprisingly painful.

Let's talk about what actually works.

The Problem: Indeed's API Is Gone

Indeed shut down their official Publisher API years ago. If you search for "Indeed API" today, you'll find outdated docs, dead endpoints, and Stack Overflow answers from 2019 telling you to "just use the API."

The reality: no major job board offers bulk data access anymore. LinkedIn locked theirs down. Glassdoor's is invite-only. Indeed's is deprecated. If you need job listing data at scale, you're on your own.

What Developers Actually Need

When I talk to people building HR tech tools or doing labor market research, they don't need a single job posting. They need:

  • Bulk listings — hundreds or thousands of results per query
  • Salary data — the field most job boards hide or make hard to extract
  • Remote filtering — especially post-2024, remote vs hybrid vs onsite matters
  • Date ranges — new postings vs stale listings that have been up for months
  • Structured output — JSON, not HTML to parse yourself

None of these are easy to get from Indeed's public-facing site.

Indeed's Unofficial Patterns

Indeed does have internal API endpoints that their frontend calls. You can find them in your browser's network tab. Here's what you'll discover:

  • There's no auth required for basic search queries
  • Results come back as JSON (good)
  • Rate limits kick in fast — you'll get blocked within minutes of automated requests
  • The response structure changes without warning
  • Geo-targeting and pagination have quirks that break naive implementations

You can build a scraper from scratch, but you'll spend most of your time handling anti-bot measures rather than working with the data you actually need.

When Scraping Makes Sense

To be clear — scraping job boards for individual job hunting is a waste of time. Just use the site.

But there are legitimate bulk use cases where scraping is the only option:

  • Salary research: What's the real market rate for a Senior Python Developer in Berlin?
  • Market analysis: How many AI/ML roles were posted this quarter vs last?
  • Alert systems: Notify me when a company posts a role matching specific criteria
  • Academic research: Labor market trends, geographic distribution of remote work
  • Competitive intelligence: What roles are competitors hiring for?

For these, you need structured data at scale.

Working With the Data in Python

Here's what clean Indeed job data looks like once extracted, and how you'd work with it:

import json
from collections import Counter

# Load scraped results (JSON output from your scraper)
with open("indeed_results.json") as f:
    jobs = json.load(f)

# Each job has structured fields
for job in jobs[:3]:
    print(f"{job['title']} at {job['company']}")
    print(f"  Location: {job['location']}")
    print(f"  Salary: {job.get('salary', 'Not listed')}")
    print(f"  Apply: {job['url']}")
    print()

# Quick salary analysis
salaries = [j for j in jobs if j.get("salary")]
print(f"{len(salaries)}/{len(jobs)} listings include salary data")

# Most common locations
locations = Counter(j["location"] for j in jobs)
for loc, count in locations.most_common(5):
    print(f"  {loc}: {count} jobs")
Enter fullscreen mode Exit fullscreen mode

The key fields you get: title, company, location, salary, url, description, datePosted, and jobType. That covers 90% of analysis use cases.

Skip the Maintenance Headache

If you don't want to build and maintain your own scraper (and deal with Indeed's constant anti-bot updates), there are ready-made tools that handle the infrastructure.

I built Indeed Jobs Scraper on Apify that handles pagination, proxy rotation, and outputs clean JSON. You set your search query, location, and filters — it returns structured data you can pipe directly into your analysis pipeline.

The output format matches the code example above, so you can start analyzing immediately instead of writing parsing logic.

Over to You

What job market data are you tracking? I've seen people build everything from salary negotiation tools to geographic hiring heatmaps. Any unusual use cases I should know about?

Top comments (0)