Scaling code is easy. Scaling state is where the real engineering happens.
There is a lie that we often tell junior engineers: “Scaling is just about adding more servers.”
And if you are building stateless web apps, that is mostly true. If your API traffic doubles, you auto-scale the compute. If a node fails, you kill it and spin up a new one.
But the moment you introduce State — databases, search indices, caches — that logic falls apart. You can’t just “spin up” a new database node in seconds. You can’t just “kill” a node that holds 5TB of customer data without consequences.
I learned this my own way while building the Search experience for Azure DevOps.
We were dealing with a massive multi-tenant system. We had millions of accounts. But the variance between those accounts was terrifying.
- The Mice: Some accounts were students with a “Hello World” project (3 files).
- The Elephants: Some accounts were massive enterprises with terabytes of source code and decades of history.
We realized quickly that we couldn’t build one architecture to serve them both.
If we put the “Elephants” on shared hardware, they would crush the “Mice” (the Noisy Neighbor problem). But if we gave every “Mouse” their own dedicated server, we would go bankrupt on infrastructure costs.
We needed a third option.
The Tenant Tiering Strategy
We didn’t solve this by choosing one architecture. We solved it by building a lifecycle.
We treated our accounts like passengers on a transit system. You start on the bus, but if you get big enough, we move you to a private car. We called this the Tenant Tiering Strategy.
Phase 1: The Shared Pool (The Commuter Train)
Every new account started here. This is a cost-efficient, multi-tenant cluster.
The Engineering Challenge: Isolation. Even though compute resources are shared, you cannot let one user hog the CPU. We implemented strict resource governance and throttling to ensure that a sudden spike from one user didn’t degrade the experience for the other 1,000 users on that node.
Phase 2: The Dedicated Shift (The Private Car)
This was the magic moment. We monitored every tenant in the shared pool.
- The Watchdogs: We didn’t use AI for this. We didn’t need a complex neural network to predict the future. We just had really good engineering. We set up watchdogs that monitored usage thresholds — index size, query volume, and latency.
- The Promotion: When a tenant hit a specific threshold (“The system is crying”), the automation kicked in. The system would automatically provision a dedicated cluster for that specific customer and migrate their data.
- Zero Downtime: The best part? The customer never knew. The migration happened in the background. One minute they were on a shared node; the next, they were on dedicated hardware. No maintenance windows. No 404s.
Phase 3: Sharding (The Fleet)
Eventually, even a dedicated node isn’t enough for the biggest customers (the whales). At that stage, we partitioned that single tenant across multiple nodes.
The Lesson: Logic Beats Hype
It has been almost 10 years since we designed this architecture.
In the tech world, 10 years is a century. Frameworks have died, languages have changed, and “AI” has taken over the conversation. But this core architecture — Shared → Dedicated → Sharded — is still running Azure DevOps Search today.
We didn’t need a complex AI model to predict capacity. We didn’t need to over-engineer a “perfect” system on Day 1. We simply needed to understand the shape of our data.
If you are a Senior Engineer or Architect looking at a scaling problem today, don’t ask “What is the fastest database?” Ask “What is the lifecycle of my customer?”
- How do they start? (Small/Shared)
- When do they grow? (The “Crying” Threshold)
- Where do they end up? (Dedicated/Sharded)
If you can answer those questions, you don’t need to rebuild your system every two years. You can build something that lasts a decade.
Top comments (0)