DEV Community: Ken-Mutisya

How I Run Dozens of Scrapers as One Person

Ken-Mutisya — Sun, 28 Jun 2026 15:00:24 +0000

I maintain a lot of scrapers by myself. People ask how that does not collapse into chaos. The honest answer is that almost all of it is boring discipline, not clever code. Here is the setup.

One repo, many small actors

Every scraper is its own Apify actor, but they all live in one repo and share a small set of libraries. Cookie extraction, a link policy filter for promotion, a few parsing helpers. When I fix a parser bug once, every actor that imports it gets the fix.

A boring, identical shape

Every actor is Node with Crawlee and Playwright, ESM, a single src/main.js entry point. Same structure, same scripts, same deploy command. Sameness is the point. I never have to relearn a project. A new actor is mostly copy the skeleton, change the parsing, ship.

Test before every deploy

One broken actor that a user reports publicly costs more than ten quiet good ones. So nothing deploys without a check and a smoke test. It is not fancy. It just has to run before every push.

Fail fast on anti bot

Scrapers hang. A target throws a captcha, a page never settles, and a run that should take two minutes burns twenty. The fix that saved me was a wall clock soft deadline inside each actor. If it is past the budget, stop cleanly, return what you have, and never charge for an empty result.

Let the platform bill

Apify charges per event, so I do not run servers or chase invoices. I do watch one number per actor though, the compute cost per run against the revenue per run. A heavy browser actor with thin output can quietly run at a loss. Light HTTP and JSON actors win.

Watch for silent failure

The scary failures are the quiet ones. An actor that returns zero rows because the site changed, not because there was no data. So I keep probes that tell the difference between a real empty result and a block, and I get flagged when an actor goes quiet.

Automate the boring promotion too

Even the marketing runs on scripts. Posting, logging what went where, respecting per platform limits so nothing gets an account banned. If I had to do it by hand I would not do it at all.

None of this is clever. It is the same boring shape repeated, with guardrails, so one person can keep a lot of plates spinning. The full set lives on Apify at https://apify.com/scrapemint and I think out loud about the build in the Discord at https://discord.gg/Ed2VNSHbr.

If you run a lot of small projects solo, what is the one piece of discipline that keeps it from falling apart?

Keyless Public APIs Most Developers Do Not Know About

Ken-Mutisya — Sun, 28 Jun 2026 14:43:40 +0000

Building scrapers taught me that a lot of scraping is unnecessary. A surprising amount of valuable public data ships as clean JSON behind endpoints with no key and no signup. Here are the ones I reach for most.

SEC EDGAR

The US securities regulator publishes every corporate filing as JSON. The path data.sec.gov/submissions/CIK##########.json gives a company's entire filing history, and efts.sec.gov runs full text search across filings. No key. The only rules are a fair use rate limit and a descriptive user agent. Insider trades, 8-K events, fund holdings, all there.

openFDA

openFDA exposes drug approvals, recalls, adverse events, and device clearances at api.fda.gov. One gotcha that cost me an hour: drug sponsor names are case sensitive and want uppercase, while device applicant names are not. A 404 just means an empty result set, not an error.

USAspending

Every US federal contract and grant is queryable at api.usaspending.gov. You POST a JSON filter and get awards with amounts, recipients, and dates. If you want new awards rather than old multi year giants, filter by date signed, not the default action date.

npm and PyPI

The npm registry search at registry.npmjs.org/-/v1/search returns packages with maintainers and metadata. On the Python side, pypi.org/simple is the entire package index as one document, and pypi.org/pypi/{package}/json gives structured metadata per project. No key, very stable, great for mapping an ecosystem.

A few more worth a look

ClinicalTrials.gov has a clean v2 API for trials. Google Patents has an undocumented xhr query endpoint that returns JSON. Hacker News ships a Firebase API. Wikipedia and OpenStreetMap both have generous public endpoints.

Why this matters

Half the time I see someone reaching for a paid data vendor or spinning up a headless browser, the data is already sitting behind one of these. Read the docs slowly, send a polite user agent, respect the rate limits, and you skip a lot of pain.

These endpoints quietly power a good chunk of the actors I publish on Apify at https://apify.com/scrapemint, and I keep notes on new ones in the Discord at https://discord.gg/Ed2VNSHbr.

What is your favorite keyless API that more people should know about?

I Ship a New Data Scraper Every Few Days. Here Is What I Have Learned

Ken-Mutisya — Sun, 28 Jun 2026 13:15:22 +0000

A while back I kept hitting the same wall. I wanted some specific slice of public data, a list of local businesses, fresh SEC filings, app store reviews, and the options were always the same. Pay a bloated subscription for a dashboard I did not need, or write yet another scraper from scratch.

So I started building the scrapers anyway, and then I did the obvious thing. I put each one online as a small pay per use API. As I write this there are around 85 of them live, and I ship a new one every few days, so by the time you read this the count is higher.

Here is what the stack looks like and what I learned.

The stack

Every actor is Node with Crawlee and Playwright, deployed on Apify. ESM modules, a single src/main.js entry point, automated checks before every deploy. Apify handles the hosting and the pay per event billing, so a buyer is charged per result instead of a flat monthly fee. You run it, you pay for what comes out.

That billing model changed how I think about products. A free tier is not charity, it is the top of the funnel. People test on a few rows, then run a real job.

The interesting problems

Most of the work was not the happy path, it was the edges.

Anti bot was the constant fight. Some targets fold to residential proxies and a good browser fingerprint. Others, like a couple of the big marketplaces, defeat both and need a different door entirely.

Keyless public APIs turned out to be gold. SEC EDGAR, openFDA, USAspending, the npm registry search, the PyPI simple index. No key, no signup, just clean JSON if you read the docs closely. A surprising number of useful datasets sit behind endpoints nobody talks about.

The pattern that paid off most was chaining. A single scraper is worth a little. A pipeline that takes the output of one and enriches it with two more is worth far more, because it does the boring glue work the buyer would otherwise do by hand.

What actually moved the needle

Positioning beat features every time. Renaming a generic scraper to the outcome a buyer wants did more for traffic than any code change.

Power users matter more than total users. A handful of people running thousands of jobs out earn hundreds of casual testers. So I optimize for runs per user, not signups.

And shipping beats polishing. A live actor that solves one real problem earns more than a perfect one that never goes out.

Where it is now

The full catalog lives on Apify at https://apify.com/scrapemint, and I just opened a Discord to keep the new drops and questions in one place at https://discord.gg/Ed2VNSHbr.

If you build things for your own problems and then wonder whether anyone else would pay for them, I would like to hear how you decided what was worth selling. What did you ship that surprised you?