A scraper that works on your machine and an actor that survives in production are two different things. I found this out the hard way shipping ten of them on the Apify Store. Here's what actually breaks, and what fixes it.
Pinned dependencies, or the build silently breaks
Crawlee and Apify's SDK move fast enough that an unpinned apify or pydantic version can break the import at build time, with an error message that points nowhere useful. The fix is boring: pin everything (apify==2.7.3, pydantic==2.10.6, browserforge==1.2.3, httpx==0.27.0), and build on Python 3.11, not whatever your local machine runs. I lost an afternoon to this before pinning became a habit.
The default key-value store does not persist
If your actor needs to remember anything between runs, deduplicated items, a rolling window for trend detection, whatever, the default KV store gets wiped every run. You need Actor.open_key_value_store(name="something-persistent") explicitly. Easy to miss once, annoying to debug when your "new item" detector fires on the same item every single run.
Bare except blocks hide the real failure
Early on, a source going down looked identical to a source returning zero results: both produced an empty dataset, silently. The only fix is boring discipline: log the HTTP status on every request, and retry on 429/5xx with backoff instead of swallowing the exception. Once I added this everywhere, actors that used to fail invisibly started failing loudly and correctly, which sounds worse but is much better.
Datacenter IPs get blocked even when residential IPs don't
One actor scraping a big marketplace worked fine from my laptop and returned 429 on every single request from Apify's cloud IP range, on two different endpoints. Not a bug in the code, a block on the IP class. The fix was routing through Apify's own residential proxy (Actor.create_proxy_configuration()), which resolved it completely. If a scraper mysteriously works locally and dies in the cloud, check the IP class before you touch the code.
Sometimes the obvious source blocks you outright
I built an actor around Yahoo Finance's options data, then discovered its crumb-authentication endpoint flatly rejects datacenter IPs, no workaround, no proxy fix. The move was pivoting to a different public source entirely (CBOE's delayed-quotes endpoint, no auth required) rather than fighting a wall that wasn't going to move. Delayed data that actually loads beats real-time data that never does.
Resource defaults quietly eat your margin
Apify actors default to fairly generous RAM and timeout settings. On a pay-per-result actor, that default can mean you're burning more in compute than you're earning per run, silently, for weeks. Tuning RAM down (in one case from 4096MB to 512MB) cut the cost per run by 4-8x and turned a marginal actor profitable without changing a line of scraping logic. Worth checking before you assume a niche "doesn't work."
Every input field needs a description, or the build fails
Small one, but it costs real time the first time you hit it: Apify's input schema validation requires a description on every property. Skip one and the build fails with a message that doesn't always point at which field.
None of this is exotic. It's the difference between "I wrote a script that scrapes a thing" and "I shipped something that keeps working when the site changes, the IP gets flagged, or the traffic spikes." That gap is most of what people are actually paying for when they hire someone to build a scraper instead of writing one themselves.
If you'd rather not deal with any of the above yourself, I take on custom Apify actor builds. My existing actors are live on the Apify Store, not just a portfolio of screenshots: check my profile. Custom build requests go through my Fiverr gig.
Top comments (0)