Originally published on the Devil Scrapes blog.
Quick answer: Web scraper maintenance, not the first successful run, is what you're actually paying for. A scraper that returns clean rows today can return zero rows tomorrow because a target moved one field, tightened one rate limit, or started serving a different page to a different exit IP — and nothing in yesterday's green test tells you that happened. We run 202 public Actors on the Apify Store, and the daily discipline of catching that drift before a customer's run silently fails is most of what we do.
Shipping is the easy 10%
Every scraper we've ever built worked on day one. That's the least interesting fact about it. The target's HTML structure, its internal API shape, its rate-limit thresholds, and its bot-defense posture are all things the target's own team can change on a Tuesday without telling anyone — and often does. A scraper is a contract with a system you don't control, and the contract gets renegotiated constantly, silently, and without a changelog.
That's why we treat every Actor we publish as a standing commitment, not a finished artifact. Across our fleet, customers ran our Actors 16,077 times in the last 30 days, and 96.1% of those runs succeeded. That number isn't a launch-day snapshot — it's a rolling average we watch every day, because it's the only honest measure of whether a scraper is actually alive.
A "SUCCEEDED" run can still deliver nothing
The first lesson in web scraper maintenance is that the platform's own status field lies to you if you let it. A run can finish with a green SUCCEEDED status and zero rows in the dataset — the process didn't crash, it just quietly found nothing. We learned to stop trusting "it finished" as proof of "it worked," and instead check what actually landed in the dataset against what a healthy run for that target normally delivers.
The second lesson: a scraper's own recent runs aren't the fleet's health signal — the target's 30-day public run stats are, because a handful of our own QA runs passing tells you nothing about the last thousand customer runs. We watch the rolling window, not the latest tick.
Real breakages, not hypothetical ones
Three examples from our own fleet, because concrete beats abstract:
- One of our job-board Actors kept every retry inside its own bounded loop, but nothing capped the sum of all the retries across a full career-site crawl. On a large site, the retries stacked up past the platform's own run timeout, and the run died with nothing written — even though every individual request handler was working exactly as designed.
- A live-chat archiver opened a long-lived connection and read from it in a loop with no upper bound. Against a quiet channel with a stalled connection, that loop could hang indefinitely instead of timing out and flushing whatever it had already captured.
- On a geo-sensitive retail target, a residential exit in the wrong country didn't get blocked — it got a normal-looking
200 OKwith prices in the wrong currency. No error, no crash, just quietly wrong data next to correctly-labeled rows.
None of these are exotic. They're the ordinary failure modes of long-running network code, and every one of them looks identical to "everything is fine" until you go looking for it.
The loop: monitor, triage, fix, verify in the cloud
Our web scraper maintenance loop has four steps, and skipping the last one is how bugs come back:
- Monitor every public Actor's rolling 30-day success rate and dataset output, not just the latest run.
- Triage the failure to a specific cause — a fault that should have been isolated to one item instead of killing the whole run, a retry budget that needed a hard ceiling, or an exit IP that needs to be pinned to the right country.
- Fix the root cause: catch and skip per-item failures instead of letting one bad page abort a thousand good ones, cap the run's total wall-clock budget, add a guard that treats an unexpected currency or a country-selector page as a block rather than a result.
- Verify in the cloud, against a customer-sized input, before calling it done. A fix that only ran against a five-row test fixture hasn't been tested against the failure it was written for — the bug that took down a real crawl needed a real crawl's worth of pages to reproduce.
We also learned the hard way that "committed" isn't "deployed": a fix can sit correct and tested on our side while the live Actor is still running the build from before the fix. So the loop always ends by confirming the deployed version, not just the patched source, is what customers are actually running.
Price the maintenance, not the first run
This is why our pricing is pay-per-event instead of a flat license: a $0.20 actor-start fee, then a per-result price (typically $1–$5 per 1,000 rows depending on the target). If a run returns nothing, you pay the $0.20 warm-up and nothing else — you're never billed for a scraper that had a bad day. But the flip side matters too: the per-result price isn't just covering the compute for that one run. It's covering the fact that we're still watching this Actor next month, and the month after, catching the day the target changes something and fixing it before it becomes your outage.
Buyers who evaluate a scraper by running it once and checking the output are pricing the wrong thing. The right question isn't "did it work today" — it's "who's watching it tomorrow."
Two Actors from our fleet that illustrate the pattern directly: the Workday Jobs Scraper survived exactly the run-budget fix described above, and the Kick Chat Scraper & Real-Time Archive survived the unbounded-loop fix. Both are live, monitored, and priced the same way — pay only for rows that land.
Run it on Apify:
FAQ
What does "web scraper maintenance" actually mean?
It means someone is watching whether a published scraper keeps working after the target site changes — its HTML, its internal API shapes, its rate limits, or its bot defenses — and fixing it before those changes turn into failed customer runs. Without it, a scraper's success rate decays quietly over weeks.
How do you know an Actor is broken before a customer complains?
We watch each Actor's rolling 30-day public run stats — total runs, successes, failures, timeouts — rather than relying on our own last few test runs, which can look green while customer runs are failing.
Why can a run say "SUCCEEDED" but return no data?
Because the process finishing without crashing and the process actually finding data are two different things. A run can complete cleanly against a page that returned nothing useful — a geo-mismatched exit, an empty result set, a silently changed selector — and still report success unless something checks the output, not just the exit code.
Why do you price per result instead of a flat fee?
A flat fee prices the code as it exists on launch day. Pay-per-event pricing — a small start fee plus a per-result charge — means you only pay when data actually lands, and it keeps our incentives aligned with keeping the Actor working, since a broken Actor earns nothing.
Does this apply to every Actor you publish?
Yes. Every one of our 202 live Actors goes through the same monitor-triage-fix-verify loop; none of them are "launch and forget."
Devil Scrapes builds and maintains 200+ pay-per-result web scrapers on the Apify Store. Blocks, retries and proxies handled by us. Browse the full catalog or commission a custom Actor.
Actors mentioned in this post:
Top comments (0)