I Built a Production Job Scraper in 4 Days — Here's What I Learned

Ahmed sayed sayed Madbouly — Sat, 15 Aug 2026 22:06:30 +0000

I spent the last 4 days building a multi-site job scraper. Not a toy project
— an actual production tool that runs on Apify and works reliably.

This is what I learned.

The Problem

I was manually checking LinkedIn and government job portals every week for
Singapore IT roles. 20-30 minutes each time. Repetitive, soul-crushing, easy
to forget.

I kept thinking: there has to be a better way.

The Solution

Built a scraper that just... does it automatically.

The results:

1,939 IT jobs in a single run
8 minutes end-to-end
$0.13 per execution
100% success rate (no silent failures)
Zero code changes to search different keywords or locations

The last part was the win. I didn't want fragile code that breaks every time
I change a search. Just fill in the form and it works.

Why This Matters (Beyond Just Saving Time)

Most people ask: "Why would you spend 4 days building something that takes
20 minutes to do manually?"

They're missing the point.

It's not about the 20 minutes. It's about never having to do it again. About
my brain being free. About knowing it runs every Sunday without me thinking
about it.

Someone here put it perfectly: "I'd do it again even if the time saved never
catches up to the time I spent building it."

That's the real win.

The Technical Approach

I chose Apify + Playwright + Node.js for a reason — not because it's
trendy, but because it's reliable.

Architecture:

Modular handlers for each site (LinkedIn, MyCareersFuture, etc.)
One orchestrator that iterates through sites × keywords × pages
Graceful error handling (no silent failures)
Proxy rotation (Apify handles it)
Cloud-deployed (not local, not fragile)

Why modular?
Because I want to add more sites later. The whole point of building this
was to prove you could make a truly extensible scraper — not a one-off hack.

Adding a new site takes ~50 lines of code. Same pattern every time.

What Took The Most Time

Not the coding. The debugging.

Getting selectors right when websites change. Handling edge cases. Making sure
it doesn't break on empty pages or network timeouts. Testing different proxy
configurations.

The actual "scrape a website" code? Maybe 30% of the work. The other 70%?
Making sure it's production-grade.

That's the difference between a script and a system.

The Honest Part

I'm probably never going to "save" 4 days worth of manual work. I'd need to
run this scraper 12+ times before the time investment pays for itself.

But again — that's not the point. The point is I don't have to think about
it anymore.

My brain is free for better things.

The Business Side

I published this on Apify Marketplace. Free tier (1 run/month), paid tier
for unlimited runs.

Not because I'm expecting to get rich. But because I genuinely believe people
would pay for reliability. And because building something and giving it away
for free feels incomplete.
**"It is totally free currently, but I may change to tiers in the future."
**Open source code, paid hosted service. Both have their place.

What I'd Do Differently

Start with the output format first. I spent too much time on architecture
before I knew what the output should look like. Lesson learned.
Test with real proxies earlier. Found issues with proxy rotation late.
Would've caught sooner if I'd tested production-like conditions earlier.
Document as you go. Retroactive documentation is brutal. Do it live.

What I Got Right

Modular from day 1. Not retrofitting structure later. Made everything
cleaner.
Actual error handling. Not just try/catch and hope. Real graceful
fallbacks.
Proof before polish. Got 1,939 jobs working before I touched the README.
Proof comes first.

For Anyone Thinking About Building Automation

A few thoughts:

You don't need AI agents for everything. Simple, reliable code beats
fancy tech every time. Make the business money first. Fancy comes later.

Build for reliability, not speed. A scraper that runs 100 times and
fails once is worse than one that runs 10 times perfectly.

Open source your code, charge for the service. The code is the trust
signal. The hosting is the value.

Don't ask if the time investment "pays for itself." If your brain gets
freedom, it pays for itself.

The Next Step

GitHub repo is public (full code):
Multi-Site Jobs Scraper

Try it on Apify (no coding needed):
Try on Apify Marketplace

Free right now. I'm collecting feedback, not revenue.

Final Thought

Building something you'd actually use is wildly underrated. It forces you to
solve real problems. Not theoretical ones. Real ones.

Start there.

DEV Community: Ahmed sayed sayed Madbouly