It's midnight, someone huge shares your app, 100,000 people show up — and the page dies. You add three servers. It gets worse.
The rule that finally made this click for me: before you add capacity, find out where the request is actually waiting.
Follow one slow request and the whole "scaling" story falls out in order:
- It's usually the database, not the app — the app only looks busy holding unfinished work.
- More app servers = more waiters, one cook. They all hit the same DB and exhaust its connection pool.
- Index the hot query before buying hardware.
- Cache it — but respect the stampede when a hot key expires.
- Queue the slow work — and watch the backlog; async can be "silently hours behind."
- Retries make outages worse without backoff + jitter.
- Two buyers, one last ticket → one atomic conditional write.
Every fix moves the bottleneck. That's not failure — that's the job.
I animated the whole thing as one request hopping through each fix:
Where does your slow request wait? I'd genuinely like to hear the war stories.
Top comments (1)
The line I'd put first is "the app only looks busy holding unfinished work" - it's the reason so many scaling decisions get made against the wrong dashboard, because CPU on the app tier is the metric people can see while the DB's wait events are not.
One thing I'd add from running the cache-stampede case for real: the hot-key expiry problem is usually worse right after a traffic spike than during it, because the first generation of the cached value happens while everything is already contending. And "watch the backlog" for async work deserves the same emphasis - we found the only backlog metric worth having is age of the oldest pending job, not queue depth, since depth tells you about the queue and age tells you about the user. How did you handle the "silently hours behind" case, a stale alarm on lag or a hard ceiling?