"Just put it behind an autoscaling group" is one of those pieces of advice that's right so often it stops getting questioned. For a stateless API or a web server, it's genuinely close to free scaling add a replica, traffic balances across it, remove a replica, nobody notices. It's such a reliable default that it's easy to reach for it everywhere.
It doesn't fit every workload, though, and the difference matters enough that I think it's worth naming precisely not as "autoscaling is bad," but as: here's the specific property a workload needs to have before reactive autoscaling makes sense, and here's what to do instead when it doesn't.
The property that actually matters: interchangeability
A stateless replica is interchangeable. Any instance can serve any request, right now, with zero setup cost specific to that request. That's the entire reason autoscaling groups work as well as they do the thing you're adding is immediately, fully useful the moment it's up, and removing one costs nothing because whatever it was doing, another replica can pick up instantly.
A lot of backends quietly violate this assumption without anyone noticing, because the violation only shows up under real scale pressure, not in normal development. Two common ways this happens:
Session affinity. Some external systems only allow a single active session per account, or per some other resource key. Once a session is established on a specific instance, that instance not just "some instance in the pool" is now the only place that session lives. A second replica can't share it, take it over seamlessly, or load-balance it away.
Slow, non-trivial startup. A stateless container can typically start serving traffic in well under a second. Some backends anything wrapping a heavier external process, a licensed desktop application being automated, a protocol requiring a multi-step handshake before it's usable take real, measurable time (seconds to minutes) before a fresh instance is actually useful. If your scaling trigger is "traffic just spiked, react now," and your instances take two minutes to become ready, you've built a system that's structurally too slow for the exact problem it's meant to solve.
If either of these is true for your backend, a generic reactive autoscaling group isn't a slightly-worse fit it's actively the wrong shape of solution, and no amount of tuning the trigger thresholds fixes that.
Scaling out: the fix is timing, not tooling
The good news: scaling out for this class of system is solvable, just not with second-scale reactivity. Instead of "traffic spiked, add a replica now," the right pattern is a slower, trend-based trigger: watch a capacity metric (percentage of current instances' sessions in use, say), and cross a threshold 70-75% is a common choice before you're actually out of room, giving the new instance its full startup time to become ready ahead of when it's actually needed.
This is the same idea as reactive autoscaling, just running on a slower clock that respects your workload's real startup cost, instead of pretending it doesn't have one.
Scaling in: the part almost everyone skips
Scaling out gets most of the attention because it's the "we're growing" story. Scaling in safely is the harder half, and it's the part a lot of systems never actually build, quietly assuming it'll be fine.
It won't be fine, by default, for exactly the same reason scaling out needed rethinking: an instance in this class of system might have live, in-progress sessions on it. A naive scale-in policy "utilization dropped, terminate an instance" has no way to know that, and will happily kill active work.
This isn't a novel problem, and it's worth knowing the industry already has a real, named answer for it: AWS's Auto Scaling supports lifecycle hooks specifically for this a termination hook can hold an instance in a pending state, giving your own code time to drain in-progress work or migrate it elsewhere, rather than the platform just pulling the plug. There's an even more direct newer option built for exactly this situation: instance lifecycle policies that keep an instance retained rather than force-terminating it if a graceful shutdown doesn't complete cleanly. The pattern to copy, regardless of which cloud you're on: stop routing new work to an instance first, wait for its existing sessions to end naturally, only then actually remove it. That's not exotic engineering it's the same "drain, then remove" idea load balancers have used for connection draining for years, just applied one layer up, at the compute level instead of the request level.
This isn't a niche problem real, large-scale systems already do this
It's worth checking whether an actual production system at scale validates this shape, rather than trusting my own reasoning alone. Video conferencing platforms are a good real-world example a live meeting session is about as textbook session-affine as workloads get. Zoom's own published architecture describes exactly the pattern this post is arguing for: participants get routed to the least-loaded available server for their region, via a dedicated control-plane component that tracks real server load not round-robin, not a generic autoscaling group blindly adding and removing capacity. That's a capacity-aware assignment layer sitting in front of a pool of session-bound servers, which is precisely the shape you land on once you take "sessions can't just move between replicas" seriously.
The actual takeaway
Before reaching for an autoscaling group, ask one honest question: can any instance in this pool serve any unit of work right now, with no setup cost specific to that instance? If yes, standard autoscaling is a great, close-to-free default use it. If no because of session affinity, slow startup, or both you need two separate, deliberately different mechanisms: a slower, trend-based trigger for scaling out, and real drain logic for scaling in. Trying to force a session-affine, slow-starting workload into a fast, reactive, interchangeable-replica model doesn't just work worse it doesn't really work at all, and the failure mode is subtle enough that it's easy to only discover it once you're already depending on it.
Top comments (0)