I Built 77 Web Scrapers in 2 Weeks — Here's What I Learned About What People Actually Need

The Experiment

Two weeks ago, I set myself an ambitious challenge: build as many web scrapers as possible and publish them on Apify Store.

The result? 77 scrapers. And almost nobody used them.

Here's what I learned about building things people actually want.

The Mistake: Building What's Cool vs. What's Needed

I started by building scrapers for platforms I thought were interesting:

• Bluesky profile scraper
• Trustpilot review extractor
• Academic paper searchers

Guess how many users? Zero.

Then I built a Reddit scraper — basically the same technical complexity — and it got its first user within days.

Why?

Because I was solving MY problems, not OTHER PEOPLE'S problems.

Nobody woke up thinking "I need a Bluesky scraper." But thousands of marketers, researchers, and founders wake up thinking "I need Reddit data."

The 3 Rules I Wish I Knew Before Building 77 Products

1. Search Volume > Technical Complexity

I spent 3 hours building an elegant arXiv paper scraper with citation parsing. 0 users.

I spent 30 minutes wrapping Reddit's API. 1 user in the first week.

The lesson: Google what people search for before you build. Not after.

2. README Is Your Landing Page

My first 20 scrapers had 3-line READMEs. "Scrapes X. Input: URL. Output: JSON."

Then I rewrote one README with:

• A problem statement ("Ever tried to manually copy 500 reviews from Trustpilot?")
• Use cases (market research, competitor analysis, sentiment tracking)
• Screenshots of the output
• A comparison with alternatives

That scraper got 3x more page views than the others.

3. One Product With 10 Users > 77 Products With 0 Users

This is the hardest lesson. As developers, we love building new things. The dopamine hit of git push on a new repo is addictive.

But the boring work — writing better docs, responding to issues, adding the feature that ONE user requested — that's what turns 0 revenue into actual revenue.

What I'm Doing Differently Now

1. Researching demand FIRST — checking search volume, Reddit questions, forum posts
2. Building 1 thing well instead of 10 things poorly
3. Writing READMEs like landing pages — problem → solution → proof
4. Talking to the 1 user who showed up, instead of chasing 1000 new ones

My Numbers (Full Transparency)

Metric	Count
Scrapers built	77
Total users	1
Revenue	$0
Lessons learned	Priceless (okay, actually expensive in time)

Your Turn

Have you ever built something nobody used? What made you realize you were solving the wrong problem?

I'd love to hear your stories — especially if you found a way to pivot from "cool project" to "thing people pay for."

---

I'm documenting this entire journey publicly. If you're interested in the intersection of web scraping, APIs, and building products people actually want, follow me on GitHub where I'm open-sourcing everything I learn.