You search Indeed for "Data Engineer New York $120k+". 2,345 results. No export button.
Most people copy-paste. Here's how to pull all of it into a spreadsheet in under 5 minutes — without writing a line of code and without getting blocked.
Why Python Scrapers Fail on Indeed Immediately
Before getting to the solution, here's why the obvious approach doesn't work.
Indeed runs on JavaScript rendering. When your requests library fetches indeed.com/jobs, it gets back this:
<div id="mosaic-provider-jobcards"></div>
Empty. The job cards don't exist yet — JavaScript loads them after the page opens. BeautifulSoup has nothing to parse.
Even if you switch to Playwright or Puppeteer to handle the JS rendering, Indeed's CloudFront layer analyzes your TLS fingerprint. Headless browsers send different signatures than real Chrome. Indeed's detection rate for headless traffic is ~31% — nearly 3× higher than the average job board.
The third layer is IP rate limiting. Indeed flags data center IPs immediately. Residential proxies help but cost $8-40/GB and add setup complexity.
The Approach That Actually Works
A Chrome extension runs inside your real browser tab — after JavaScript has rendered, using your actual cookies and session. There's no fingerprint mismatch because it isn't headless. Indeed sees a normal Chrome session at normal browsing speed.
Here's the full workflow with Clura's Indeed scraper:
- Run your Indeed search — job title, location, salary filter, date posted. Let results load fully.
- Open Clura from the Chrome toolbar — it detects the repeating job card structure automatically.
- Review detected fields — job title, company, location, salary range, date posted, job URL. The Indeed template pre-maps all of these.
- Export to CSV — one row per job, one column per field.
- Paginate — Clura handles auto-pagination across all result pages.
What You Can Extract
| Field | Notes |
|---|---|
| Job title | Always present |
| Company name | Always present |
| Location | City, state, remote flag |
| Salary range | Present on ~40% of listings |
| Job type | Full-time, contract, remote, etc. |
| Date posted | Relative (1 day ago → absolute date) |
| Job URL | Direct link to full description |
Tool Comparison
| Tool | Block Rate on Indeed | Setup | Cost |
|---|---|---|---|
| Python + requests | ~85% (immediate) | 2-4 hrs | Free (fails) |
| Playwright | ~31% | 4-8 hrs | Free |
| Apify cloud scraper | ~22% (shared IPs) | 30-45 min | $49/mo+ |
| Bright Data | ~8% (residential) | 1-2 hrs | $500+/mo |
| Chrome extension (Clura) | ~4% (real session) | 2 min | Free tier |
The Use Cases Worth Knowing
Salary benchmarking — Indeed shows salary ranges on 40% of postings, higher than most job boards. 200 "Senior Engineer" listings across 3 cities gives your HR team real-time market rate data without a $15k compensation survey.
Competitor hiring intelligence — scrape a competitor's company page weekly. Track new roles by type and location. 12 new "Account Executive" postings in one quarter is a signal their sales team is scaling.
B2B lead generation — job postings are buying signals. A company hiring a "Head of Data" is probably in the market for data infrastructure. Scrape weekly, filter by role, build a target account list.
Is Scraping Indeed Legal?
The hiQ v. LinkedIn ruling (9th Circuit, 2022) established that scraping publicly accessible data doesn't violate the CFAA. Indeed's job search requires no login — it's public data.
Indeed's ToS prohibit automated collection, but ToS violations aren't criminal. Indeed enforces via technical blocking, not legal action against individual users. Operating at human browsing speed through a real Chrome session keeps you well within the normal use pattern.
Full breakdown including scheduled automation options and the complete tool comparison: Indeed Scraper Guide
Top comments (0)