Apify went down yesterday. Our monitoring business didn't notice.

Amine Mbarki — Wed, 22 Jul 2026 11:50:14 +0000

Yesterday afternoon, Apify had an incident. If your scraping pipeline lives entirely on their platform, you know exactly what that afternoon felt like: stalled runs, missing data windows, refreshing the status page.

Our price-monitoring pipelines didn't miss a single cycle. Not because we got lucky — because of an architecture decision we made months ago that I want to talk you into copying.

The architecture: self-hosted first, managed as insurance

We run Crawlee Cloud — a self-hosted, open-source platform that implements the Apify API — as our primary. Apify itself is our fallback.

Read that again, because the order is backwards from what most teams do. The open-source thing on our own droplets is production. The managed platform is the insurance policy.

This works for one reason: both platforms speak the same protocol. Our Actors use the official, unmodified Apify SDK. They genuinely don't know which platform they're running on. Same Actor.pushData(), same datasets, same request queues. Failover isn't a rewrite or an adapter layer — it's a base URL.

┌────────────────────────────────────────────┐
│              Your Actors                   │
│   (official Apify SDK, zero code changes)  │
└─────────┬──────────────────────┬───────────┘
          │ primary              │ fallback
          ▼                      ▼
   Crawlee Cloud            Apify platform
   (your droplets)          (their cloud)

Yesterday that architecture earned its keep: 100% uptime for pipelines whose entire value proposition is freshness, on a day the upstream platform was having a bad time.

It's also 3x cheaper

The failover story is the dramatic part, but the boring part is what made the decision easy in the first place: on real production runs, our costs dropped roughly 3x versus running the same workloads on Apify. Actual client workloads, compared invoice to invoice — I track runner costs daily from our DigitalOcean billing export.

The mechanics are simple. Metered platforms charge per compute unit, per proxy GB, per storage op. Great for prototypes, punishing at scale: your bill grows linearly with your success. Self-hosted runners are flat droplet-hours on infrastructure you already pay for. Scale up for a backfill, scale down after, using the same tooling as the rest of your stack. The heavier your usage, the wider the gap.

(Fair caveat before the comments write it for me: that's infrastructure cost only. You're trading dollars for ops ownership. Keep reading — that trade is exactly what the rest of this post is about.)

The part where I admit how this was built

A large share of Crawlee Cloud was written with heavy AI assistance. Vibe-coded, if you like. I'm not embarrassed by that, but I learned something important from it: AI writes plausible code, not audited code. During a deep review of my own codebase I found an insecure default, a container network isolation gap, and a Docker network name mismatch — all generated confidently, all wrong.

Building has become trivial. Any developer with an AI assistant can scaffold an actor orchestrator in a weekend — I've watched three separate teams in my own network each grow their own private, 80%-identical, half-maintained version. The hard part was never generation. It's verification, hardening, and the 3am pager. Those costs didn't go away when the first draft became free.

Which is the whole argument for sharing one codebase

If everyone can build it and nobody wants to maintain it, the worst outcome is a thousand private forks each carrying a full maintenance burden alone. The rational move is one shared, MIT-licensed codebase where the maintenance is split:

I fix sticky-session handling → your deployment gets it.
You harden runner isolation → mine gets it.
Someone adds a MinIO edge case for their compliance setup → everyone gets it.

Open source has always made this argument. AI made it sharper: when writing code costs nothing, all the value concentrates in the shared, battle-tested, collectively audited artifact.

Steal this setup

git clone https://github.com/crawlee-cloud/crawlee-cloud
cd crawlee-cloud
docker compose up

Point your existing Actors at it. Keep Apify configured as fallback if you want the same insurance we have — the protocol compatibility means you never have to pick a side, and either platform can cover for the other.

⭐ Star the repo — it's how other teams find this instead of vibe-coding their own.
🐛 Break it on your weird infra and file issues.
🔀 Send the PR you'd otherwise keep in a private fork.