For a month I built scrapers the way most indie devs do: pick a site I find interesting, build it, publish, hope. Thirty-seven actors later the honest scorecard was: one real earner (a Whop clipping-campaign scraper), a handful with a single paying user each, and a long tail of "zero starts".
So I stopped guessing and pulled the market instead.
Step 1: pull the whole store
Apify's public Store API returns every actor with its 30-day run stats, user counts, rating and price. I pulled the top 8,758 by monthly users (54k listed in total, the rest are dead) and bucketed them by keyword cluster. Monthly active users per cluster, roughly:
| Cluster | MAU | Cluster | MAU |
|---|---|---|---|
| 133k | Ads (Google/Meta) | 35k | |
| 114k | YouTube | 34k | |
| Jobs | 59k | 16k | |
| Google Maps | 57k | Leads / contacts | 12k |
| TikTok | 50k | Transcripts | 11k |
Then the fun part: inside each cluster, is the leader hated (≥250 MAU but ≤3.6★), expensive (≥$10 per 1,000 results) or broken (≥35% failed runs)? Those are the three gaps a newcomer can actually take.
Some examples that fell out of the data:
- Airbnb: 1,900 MAU across 48 actors, leader at 754 MAU with 2.5★.
- YouTube transcripts: 11k MAU, leader charges $10 / 1,000 and sits at 3.7★.
- Google Trends: leader has a 37% failure rate.
- LinkedIn jobs: 47k MAU, leader is fine ($2 / 1,000, 16k MAU) — the gap is depth, not quality.
Step 2: probe before building
A gap is worthless if the site blocks datacenter IPs. So each candidate got a 10-minute HTTP probe from a plain server: fetch the listing page, fetch a detail page, hit whatever JSON the front-end calls, count 403s.
Airbnb, LinkedIn's guest job API and YouTube's watch page all answered 200. Google Trends (429), eBay (403 even with browser fingerprints), TikTok Shop (login wall) and Google Jobs did not. They went on a "needs residential relay" list instead of the build list.
One thing I learned the hard way: my sandbox turned out to have a residential IP, so "passed the datacenter probe" only became true after the first cloud run. Verify from the platform you deploy to.
Step 3: build the three that passed
Same day, three actors, each with unit tests against fixtures, a live gate script (10 checks against the real site) and one verification run in the cloud:
Airbnb Market Intel Scraper — search results plus, per listing, the 12-month availability calendar. From that you get occupancy and estimated revenue, which is what investors pay AirDNA for. Live gate today for Austin: median nightly price $166, p25 $130, p75 $222; a sample listing at 57% 30-day occupancy, ~$4.8k estimated monthly revenue. $4 / 1,000 listings, $6 with calendar.
LinkedIn Jobs Scraper — Full Description, Salary & Applicants — the guest API gives the card; the detail page gives description, criteria, salary when posted and applicant count ("Over 200 applicants" on two of three sampled data-analyst roles today). It also emits free "company signal" rows: employers ranked by open roles, which is a lead list for recruiters. $1.60 / 1,000 detailed jobs vs the leader's $2.
YouTube Transcript Scraper — the one that failed the cloud test. YouTube returns the caption track list but the timedtext endpoint answers 200 with 0 bytes from datacenter IPs. Fix: the actor tries direct first, then falls back to a tiny relay running on a Mac mini behind a Tailscale funnel. Verification run: 7 of 8 videos transcribed, 38 seconds, $0.003. $6 / 1,000 transcripts vs $10.
What the survey changed
The old approach optimized for "can I scrape this?". The survey optimizes for "will anyone start this?" — demand first, gap second, feasibility third, build last. One day in: zero paying users, which is exactly what day one looks like. I'll post the 8-week numbers either way.
The survey script is ~150 lines of Python against one public endpoint. If you publish on any marketplace with a public stats API, do this before your next build.
All three are HTTP-only, pay per result, and callable as MCP tools. If you try one, a review on the store page is the only signal other buyers have — and open an issue if a field is wrong; I fix those fast.
Top comments (0)