DEV Community

Cover image for Autoscaling Adds Waiters, Not Ovens. Here's What Actually Breaks at 10x Load
yulyabrocoders for Brocoders

Posted on

Autoscaling Adds Waiters, Not Ovens. Here's What Actually Breaks at 10x Load

A signed enterprise deal or a viral campaign puts the same pressure on a platform: 10x the normal traffic, arriving faster than anyone budgeted for. The instinct is to trust autoscaling to absorb it, and autoscaling does its job exactly as designed. The problem is what it was designed to scale. It adds more stateless application instances, which were already the most elastic part of the system, and does nothing for the database connection pool, the cache origin, or the shared job queue underneath them. Manoj Yerrasani, VP of Global Platform Engineering at NBCUniversal, put it plainly after operating at Super Bowl scale: "auto-scaling is a lie." More app servers just means more processes competing for the same fixed set of database connections, and that competition is where the outage actually starts.

Four questions that predict where a platform breaks first

The Load Shape Audit is four diagnostic questions, not a tool you install:

  • Request shape: how many backend calls does a single user action trigger? A page load that fans out to six API endpoints multiplies your real request volume by six before traffic even grows.
  • Data shape: how many different access patterns are competing on one database? Write-heavy configuration writes and read-heavy analytics queries fighting over the same connection pool degrade both.
  • Tenant shape: does one large tenant's workload saturate a shared resource, connection pool, job queue, cache, search index, and degrade service for every smaller tenant on the same infrastructure?
  • Failure shape: when something has to give, is there a predetermined priority order, or does the system degrade randomly?

Signal 3 is the one most platforms discover the hard way. Neon's production analysis and the AWS whitepaper on SaaS tenant isolation both document the same pattern: one noisy tenant degrading everyone sharing its infrastructure.

The pattern that fixes the most common request-shape failure

When a cache misses under load, the naive implementation lets every concurrent request for that same uncached key hit the database independently, ten simultaneous cache misses become ten identical queries. The fix is request collapse, also called singleflight: the first request triggers the real query, and every other request for the same key waits on that one result instead of firing its own.

const inFlight = new Map();

async function getWithSingleflight(key, fetchFn) {
  if (inFlight.has(key)) {
    return inFlight.get(key); // piggyback on the request already running
  }
  const promise = fetchFn()
    .finally(() => inFlight.delete(key));
  inFlight.set(key, promise);
  return promise;
}
Enter fullscreen mode Exit fullscreen mode

That's a handful of lines, and it turns a thundering herd of duplicate queries into one query with N waiters. It's also exactly the kind of fix that autoscaling can't provide, because the problem was never a shortage of application instances.

Isolating the noisy neighbor, in order of cost

Tenant-shape problems get fixed in escalating stages, and jumping straight to the most expensive one wastes engineering time you don't have during an incident:

  • Row-level isolation at the database layer. PostgreSQL row-level security enforces tenant boundaries in the database itself, not in application code that a future developer might bypass.
  • Per-tenant connection pools and job queue lanes. A large tenant gets its own lane instead of competing with everyone else for the same shared pool.
  • Full tenant silo. Dedicated infrastructure per tenant, the most expensive option, reserved for the tenants whose load actually justifies it.

What this looked like on a real platform

Brocoders rebuilt Adact's platform on Node.js and Nest.js with a split database layer, PostgreSQL for write-heavy campaign configuration, MongoDB for read-heavy analytics, so the two access patterns stopped fighting over the same connection pool. Server-side rendering behind CloudFront collapsed what had been multiple API calls per page load into a single request. That architecture held through a campaign that generated 18,000 leads in 10 days and sustained over 2 million requests at peak across more than 3 million visitors in six months, without the app-server layer being the thing that mattered most. We run full-stack delivery with our own DevOps, no outsourced infra, so the team building the feature is the same team that has to answer when a connection pool saturates at 2am.

Pre-launch checklist before you trust autoscaling with a 10x event

  • Run expected load in staging at least twice before launch, not once, and not in production.
  • Freeze configuration changes the week of launch. Ben Maurer's 2015 analysis of Facebook outages found human error during change windows was a leading cause, not raw traffic.
  • Map every shared resource: connection pools, caches, job queues, search indexes, and who else is competing for each one.
  • Write explicit traffic-tier priorities before you need them: what's protected (checkout, auth), what degrades gracefully (recommendations), what gets dropped with a message.
  • Set numeric thresholds for architectural work, not vibes: "at X concurrent users the read path splits, at Y tenants we move to per-tenant pools."
  • Don't build the full silo model preemptively. Every extra service boundary adds deployment surface and on-call pages before the load that justifies it arrives.

If you're staring down a 10x traffic event and want a second technical opinion on which layer actually breaks first, brocoders.com is a reasonable place to start that conversation.

Top comments (0)