Muhammad Ahsan Farooq

Posted on Mar 3 Edited on Oct 10

Beyond Microservices: Surviving the Storm with Cell-Based Architecture

#architecture #distributedsystems #systemdesign #backenddevelopment

Containing failures before they become outages, with lessons from Slack, DoorDash, and AWS

For a decade, the answer to scaling was to break the monolith. Slice a giant codebase along domain lines, and let independent services scale, deploy and fail on their own. Microservices delivered on that promise, up to a point.

Then systems grew. Dozens of services became hundreds. They shared databases, caches and queues, and each of those became load-bearing. Teams added circuit breakers, retries and timeouts. Those tools help, but they treat symptoms. Slowly, teams had rebuilt the monolith with network hops in between: a networked monolith, where one sick component can slow down everyone.

Cell-based architecture goes after the root cause. Instead of one big shared pool that serves all customers, you run many small, identical, self-contained copies of your whole stack, called cells. Each cell serves a slice of your customers. When something breaks, only that slice is affected.

This post explains how cells work, why they contain failures, and how to adopt them, with examples from Slack, DoorDash and AWS.

In this post:

The problem: gray failures, and why shared infrastructure turns small faults into global outages
What a cell is, and the one rule that makes the pattern work
The router: partition keys, cell mapping and request flow
Failover and draining, and the arithmetic of blast radius
Case studies: Slack, DoorDash and AWS shuffle sharding
An adoption playbook, the costs, and a readiness checklist

The problem: from monolith to networked monolith

A classic microservice stack looks clean on a whiteboard. A pool of web servers feeds a pool of application servers, which share one database cluster and one global Redis cache. Traffic round-robins across everything, and scaling out is easy: add more pods.

The weakness is not scale. It is interconnection. Every request path depends on the same shared infrastructure, so a small fault in one corner can reach everyone.

Binary failures vs gray failures

Binary failures are the easy kind. A server crashes or a database goes offline. Health checks notice, the load balancer stops sending traffic, and automation takes over.

Gray failures are the dangerous kind. A component is degraded but not dead, so nothing triggers automated remediation. Slack describes them as failures where different components have different views of whether the system is available. A highway with one lane crawling at 10 mph does not stop traffic. It backs it up until everything gridlocks.

Common gray failures include:

Packet loss within a single Availability Zone
A Redis shard slowing down under memory pressure
A retry storm caused by a bad deployment config
Connection pool exhaustion behind a slow dependency
A noisy neighbor throttling CPU on a shared host
DNS resolution latency spikes

How one slow node takes everything down

gray failure cascade · illustrative, 4 stages

The numbers in the picture are illustrative, but the arithmetic behind the mechanism is not. Little's law says the number of requests in flight equals the arrival rate times the average time each one takes. Suppose an app tier has 100 worker threads, receives 1,000 requests per second, and normally answers in 50 ms. Now 10% of requests touch the slow node and take 2 seconds:

Scenario	Average latency	Threads needed	Threads available
Healthy	50 ms	50	100
10% of requests hit the slow node	0.9 × 50 ms + 0.1 × 2,000 ms = 245 ms	245	100

The pool is exhausted, even though 90% of requests are perfectly healthy. Once threads run out, the healthy requests queue up behind the slow ones, and everyone sees the slowdown. Retries make it worse by adding load to a tier that is already struggling.

The standard defenses are circuit breakers, bulkheaded thread pools, timeouts and retries with jitter. They help at the margins, but they all treat symptoms. The root cause is that all requests share all infrastructure, so the blast radius of any failure is effectively the whole platform.

Slack's own incident review asked the question this way after a 2021 network fault in a single Availability Zone: why was this outage visible to users at all? The answer led to the design in this post.

What a cell is

The name comes from marine engineering. Ships are built with watertight bulkheads, walls that divide the hull into compartments. If the hull is breached, only the damaged compartment floods. A common retelling of the Titanic's sinking is that its bulkheads did not reach high enough, so water spilled from one compartment into the next. The design assumption was violated, and the failure cascaded the way it does in a poorly isolated distributed system.

AWS's guidance defines it precisely: a cell-based architecture uses multiple isolated instances of a workload. Each instance, a cell, is independent, shares no state with other cells, and handles a subset of the overall requests. If a workload uses 10 cells to serve 100 requests, a failure in one cell leaves 90% of the requests unaffected.

cell-based architecture · router, 4 cells, control plane

The pattern has three parts:

Component	Its job	Design rule
Cell router	Maps each request to exactly one cell	The thinnest possible layer. It routes and does nothing else
Cell	A complete workload that can run on its own	No shared state, and no calls to other cells
Control plane	Provisions and removes cells, assigns and migrates tenants	Keep it thin, read-mostly, and cacheable at the cell edge

Building cells does not necessarily mean multiplying your infrastructure. AWS notes that an application with 30 hosts can keep the same 30 hosts, grouped into cells behind a router.

Anatomy of a cell

A cell is not a single service or a single database. It is a complete copy of your application stack, scoped to a bounded set of tenants:

inside one cell · compute, data, observability

Every cell owns its own:

Compute: API and worker containers
Storage: database shards, not a connection to a shared cluster
Cache: its own Redis or Memcached cluster
Queues: its own Kafka topics or SQS queues
Observability: its own metrics namespace, so a sick cell stands out on a dashboard instead of being hidden inside a global average

The golden rule: cells never talk to each other

A request that enters Cell 4 lives and dies in Cell 4. There are no cross-cell service calls, no shared databases and no shared caches.

This rule is not negotiable. The moment one cell depends on another, a slow cell can drag its neighbor down, and you have rebuilt the networked monolith with extra steps. If a request really needs data owned by another tenant, solve it at the routing layer by sending the request to the right cell, or by replicating data asynchronously. Never let cells call each other synchronously.

The control plane

Some concerns sit outside any one cell: assigning tenants to cells, provisioning new cells, migrating tenants, global feature flags and billing state. These belong to the control plane.

The discipline is to keep it read-mostly and to cache what cells need at the cell edge. If the control plane has a brief outage, each cell should keep serving traffic from its cached state. A control plane outage must never become a cell outage.

The routing layer

If cells are the body of this architecture, the router is the brain. AWS calls it the thinnest possible layer: it routes requests to the right cell, and only that. Every request passes through it, so it must be deterministic and extremely reliable. This is one of the few components worth over-engineering from the start.

Step 1: choose the partition key

The partition key decides how customers are sliced across cells. It is the most consequential design decision in the whole system. AWS says the key should align with the grain of the service, meaning the natural way its workload can be divided with minimal cross-cell interaction.

A good partition key has three properties:

It co-locates related data. Nearly everything a request needs belongs to the same key value.
It distributes reasonably evenly. No single value carries a disproportionate share of traffic.
It is present on every request. The router can read it without extra lookups.

Partition key	Typical fit	Example
Organization or tenant ID	B2B SaaS and multi-tenant platforms	A workspace, company or account
User ID	Consumer apps with isolated user data	Social, gaming and messaging apps
Region	Compliance and data residency needs	EU-only customer data
Order or delivery ID	Workflows scoped to one transaction	Food delivery, ride-hailing
Merchant or store ID	Marketplaces	One seller's catalog and orders

Avoid keys that are too coarse, such as country, where a single value could be larger than a cell can handle. Avoid keys that are too fine, such as session_id, which would need an unmanageable number of cells. The sweet spot is an entity that owns a meaningful, bounded set of data.

Step 2: map keys to cells

The router needs to know which cell owns which key. There are three common approaches:

Approach	How it works	Strength	Weakness
Lookup table	A small, fast store maps `org_id` to `cell_id`	Flexible. Moving a tenant is one row update	An extra lookup per request, so it must be cached aggressively
Deterministic hash	`cell = hash(org_id) % number_of_cells`	No lookups, stateless and fast	Adding cells reshuffles most tenants, and you cannot move one tenant alone
Hybrid	Hash by default, with a lookup table for exceptions	Speed on the common path and flexibility where it matters	Two mechanisms to maintain

Most production systems end up with the hybrid. Large enterprise tenants get pinned to specific cells for compliance, capacity or contractual reasons, and everyone else is hashed. A useful refinement is to hash into a fixed number of buckets, such as 1,024, and map buckets to cells. Adding a cell then means reassigning a few buckets, not rehashing everyone.

const BUCKETS = 1024;

function cellFor(orgId: string): string {
  // 1. Pinned tenants (compliance, very large customers) come from the lookup table
  const pinned = pinnedTenants.get(orgId);
  if (pinned) return pinned;

  // 2. Everyone else: a stable hash picks a bucket, and the bucket maps to a cell
  const bucket = stableHash(orgId) % BUCKETS;
  return bucketToCell[bucket];
}

Step 3: follow one request

Here is the full lifecycle of a request for the tenant acme-corp-7291:

request lifecycle through the cell router · 7 steps

Step 4 matters. The router tracks cell health, so it can steer traffic away from a sick cell. That capability is what makes failover possible, which is the next topic.

Failover and draining

Because the router tracks cell health, it can act when a cell goes bad. What it should do depends on what kind of cell you built.

Two kinds of cells, two kinds of failover

Tenant-partitioned cells own the data for specific tenants. If one fails, other cells cannot simply take over, because the tenant's data lives in the failed cell. The value here is containment: only that cell's tenants are affected, and the fix is scoped to one small environment. Recovery means repairing the cell, restoring it from backups, or failing over to a standby that holds replicated data.
Replica cells are interchangeable copies that can serve any customer, for example one per Availability Zone. Here the router can shift traffic from a sick cell to healthy ones. This is the model Slack uses.

Many systems combine both: cells aligned to Availability Zones, with tenants partitioned inside them.

Draining, as Slack does it

Slack's cells are Availability Zones. After a 2021 incident, its engineers set four goals for an AZ drain:

Move as much traffic as possible out of an AZ within 5 minutes.
Cause no user-visible errors. In-flight requests finish normally.
Make drains and undrains incremental, down to 1% of traffic, so a recovered AZ can be tested gently.
Never depend on resources inside the AZ being drained.

To meet them, Slack made each service talk only to services in its own AZ, a design it calls siloing. Draining an AZ then needs no change inside the services. It only requires steering new user requests away at the edge. Edge Envoy load balancers receive per-AZ weights from a control plane, and an AZ whose weight is set to zero stops getting new requests while existing ones complete. The change propagates in seconds. Slack also chose to keep a human in the loop: an operator decides when to drain.

The same idea works for any cell router. This picture shows the effect on three cells as Cell A is drained:

illustrative router weights · 3 cells, 3 states

Here is an illustrative control-plane routine. It lowers the weight in steps and stops if the receiving cells start to struggle:

async function drainCell(cellId: string, steps = [50, 20, 0]) {
  for (const weight of steps) {
    await controlPlane.setCellWeight(cellId, weight);   // the router applies it within seconds
    await sleep(60_000);                                // watch error rates before going further
    if (await errorRatesRising(otherCells(cellId))) {
      throw new Error('Drain paused: receiving cells are struggling');
    }
  }
}

Hard failover vs draining

	Hard failover	Draining
How it works	Remap all of a cell's traffic at once	Shift weight gradually while in-flight requests finish
Main risk	A sudden surge overwhelms the receiving cells	Slower to complete
Capacity needed	Spare capacity everywhere, immediately	Spare capacity, claimed in steps
Best for	A cell that is completely down	Gray failures, where you are still deciding

Either way, the receiving cells need spare capacity. The next section works out how much.

Blast radius: the math that sells it

With N equal cells, a failure in one cell affects about 1/N of your customers. AWS's example: with 10 cells serving 100 requests, a failure in one cell leaves 90% of the requests untouched.

computed: 1/N users affected, 1/(N-1) spare capacity, for N equal cells

The chart also shows the price of recovery. If a failed cell's traffic is spread evenly over the survivors, each one needs 1/(N-1) of spare capacity. With 10 cells that is about 11%. With 100 cells it is about 1%. If you instead pair cells so each fails over to a single neighbor, every cell must keep 100% spare capacity, which is why spreading the load matters. More cells mean a smaller blast radius and a smaller proportional safety margin, but also more environments to run.

Before and after

Failure	Without cells	With 100 cells
Bad deployment	Reaches everyone as the rollout proceeds	Rolled out cell by cell. The first cell exposes about 1% of customers
Corrupt data or a bad migration	Every tenant is at risk	Only the tenants in that cell
Infrastructure fault	Everyone sharing the infrastructure	Contained to the affected cell, and traffic can be drained from it
Noisy tenant	Degrades all other tenants	Degrades only its own cell
Poison pill request	Can take down the shared pool	Crashes one cell's workers, and the rest keep serving

Before cells, one engineer pushing a bad config at 2 a.m. could take down the whole platform. After cells, the same mistake reaches 1% of customers. It is still an incident, but it is a small one that is recoverable and debuggable by a single on-call engineer.

Deploy one cell at a time

Safe deployments get dramatically easier. In a pooled system, a canary sends 5 to 10% of global traffic to the new version, so a bug hits a random slice of all customers. With cells, you deploy to one cell. The affected customers are known, bounded and can even be notified. AWS recommends applying changes gradually, compartment by compartment.

A rollout plan for 100 cells might look like this:

Wave	Cells	Cumulative customers exposed
1	One canary cell, ideally with internal or low-risk tenants	1%
2	The next 4 cells	5%
3	The next 20 cells	25%
4	Everything else	100%

Pause between waves, watch per-cell error rates, and halt automatically if a wave looks worse than the baseline.

Real-world case studies

Three companies show three different ways to apply the idea. I checked each against the team's own write-up.

Slack: Availability Zones as cells

On June 30, 2021, a network link between Availability Zones in Slack's main region began to fail intermittently. The cloud provider removed it, put it back, and removed it permanently later that day. On paper it was a routine hardware failure, so Slack's engineers asked why users saw it at all.

The answer was a gray failure. One user request can fan out into hundreds of calls to backends, and every call must succeed. Failure detection needs some failures before it can exclude a bad server. Slack's main datastore also needs a single available primary for each write, so a shard whose primary is unreachable cannot accept writes. Different components held different views of what was healthy, and the whole system suffered.

Slack's response was to treat each Availability Zone as a cell and to silo services so they only talk within their own zone. That turns an AZ drain into an edge routing change, as described in the previous section. The design target was to drain most of a zone's traffic within 5 minutes, which matters because Slack's 99.99% availability SLA allows less than an hour of total unavailability per year.

Note what Slack did not do. Its cells are replicas aligned to Availability Zones, not partitions of customers by tenant. The pattern is flexible: what matters is that the isolation boundary is real, and that you can steer traffic around it.

DoorDash: cells for isolation, plus cheaper traffic

DoorDash's microservices run in multiple isolated cells. Every service has one Kubernetes deployment per cell, and traffic between cells is not permitted, which limits the blast radius of a single cell failure. Services not yet migrated live in a global cell, reached through internal Envoy routers. Each cell is made of several Kubernetes clusters, and each cluster spans multiple Availability Zones.

The cost problem came from success. As the number of microservices grew, so did cross-AZ data transfer charges, which are billed on both sending and receiving. DoorDash's answer was to enable Envoy's zone-aware routing in its service mesh, so callers prefer callees in their own zone, while still balancing load so no pod is overloaded. Even with pods spread evenly across zones, about 10% of traffic still crosses zones.

The investigation is instructive. Flow logs and metrics showed that the largest source of cross-AZ cost was non-storage traffic within each cell, and about 20% of that cost was traffic to a single load balancer in front of an event ingestion pipeline. That path had several extra hops. DoorDash moved the service into the same cell as its clients and let their sidecars route directly to its pods. The savings were large enough that the cloud provider reached out to ask whether DoorDash was having a production incident.

The lesson is that cells give you a bounded place to optimize. Because traffic stays inside a cell, you can count the hops and measure the cost. Note that the savings came from zone-aware routing, which the cell boundary made tractable. DoorDash did not publish a single overall percentage.

AWS: shuffle sharding, cells without hard partitions

Amazon uses a variant called shuffle sharding for workloads where pinning each customer to one cell is impractical, such as DNS. Instead of one cell per customer, each customer gets a random subset of workers, and different customers get different subsets.

shuffle sharding · 8 workers, 3 customers

The arithmetic is simple. With 8 workers and 2 per customer, there are 28 possible pairs. Only a customer assigned the very same pair as the troublemaker loses both workers, which is 1 in 28, or about 3.6%. Ordinary sharding into 4 shards would put 25% of customers on the failed shard. Customers who share a single worker still have the other one, as long as their clients retry.

Amazon Route 53 applies this at scale. It arranges capacity into 2,048 virtual name servers and gives every domain a shuffle shard of four, which allows about 730 billion distinct shards. No domain shares more than two virtual name servers with any other. If one domain is hit by a DDoS attack, its four name servers see a spike, and no other customer notices.

Shuffle sharding needs no partition key and works well for stateless workloads. It is a useful tool when you cannot commit to hard cells.

Implementation playbook

You do not need to boil the ocean. Most teams start with two or three cells for one critical service and expand from there. Five phases cover the work.

Phase 1: find your partition key

Look at your data model for the entity that naturally owns everything else. In B2B SaaS that is usually org_id or tenant_id. Draw the dependency graph and check that the vast majority of queries can be answered inside one tenant's scope.

If you cannot find a clean partition key, stop here. Cells without one produce a system that is complex without being resilient.

Phase 2: build the router

Stand up a gateway that does four things:

Extracts the partition key from the request, such as a JWT claim, a path parameter or a header
Looks up the cell assignment from a fast, cached store
Proxies the request to that cell's ingress
Applies a configurable fallback policy when a cell is unavailable

Keep it as thin as AWS recommends. Build on a battle-tested proxy such as Envoy or NGINX, and add cell-aware routing on top rather than writing a proxy from scratch.

Phase 3: provision cells as code

Every cell must be identical by definition. Use Terraform or Pulumi modules that take a cell_id and stamp out the whole stack. Manual drift is a ticking time bomb: if Cell 7's Redis configuration differs subtly from Cell 12's, you will find out at the worst possible moment.

module "cell_12" {
  source  = "./modules/cell"
  cell_id = "cell-12"
  region  = "us-east-1"
  az      = "us-east-1b"

  # Capacity is identical for every cell
  api_instance_count    = 3
  worker_instance_count = 2
  db_instance_class     = "db.r6g.large"
  redis_node_count      = 3
}

Tenant assignments belong in the control plane's data, not in the infrastructure code. Adding a cell should be a one-line change that triggers an automated pipeline. If it takes days of manual work, you will not spin up cells when you need them.

Phase 4: per-cell observability

This is not optional, because it is how the whole design pays off. Tag every metric and log line with cell_id. Build dashboards that show all cells side by side, so a degraded cell stands out like a red square in a grid of green. Alert on per-cell error rates, not just global averages, because global averages are exactly the metric that hides localized failures.

A useful test: if your on-call engineer has to query logs to find out which cell is sick, your observability is not good enough.

Phase 5: build tenant migration tooling

Build the tool that moves a tenant between cells before you need it in an emergency. You will need it when a cell becomes a hot spot, when an enterprise customer asks for data residency, and when you drain a degraded cell. One safe pattern is a dual-write with a delayed read flip:

tenant migration · 6 stages, where reads and writes go

Start dual-writing. All new writes go to both cells. Reads still come from the old cell, the source of truth.
Backfill. Copy the historical data to the new cell, then compare checksums until they match.
Flip reads to the new cell and watch error rates for a few minutes.
Cut over. Stop writing to the old cell and update the cell map so the tenant routes to the new cell.
Clean up the old cell's copy of the data.

The writes in step 1 must be idempotent and ordered, or the backfill can overwrite newer data with older data. Build that guarantee in from the start.

The cost: trade-offs you must accept

Cell-based architecture is not free. Anyone selling it without discussing the costs is doing you a disservice.

You now run N production environments. With 50 cells you have 50 databases to back up, 50 Redis clusters to monitor and 50 queue setups to watch. A schema migration becomes 50 coordinated operations. Infrastructure as code stops being a best practice and becomes a requirement for survival.
Not everything fits in a cell. Authentication, global feature flags, payment methods and anything shared between tenants must live in the control plane or be replicated to each cell. Keep the control plane as thin as possible, because it is a shared dependency and the more it does, the more its failures matter.
Cross-cell queries are gone. A query that used to join across all users now has to fan out to every cell and merge the results, which is slow and complex. The standard answer is a separate analytics pipeline: every cell streams its events to a warehouse, and analytics queries run there instead of against operational databases. That is a good pattern anyway, but cells make it mandatory.
You need spare capacity. As the previous chart showed, each surviving cell needs about 1/(N-1) of spare capacity to absorb a failed neighbor's traffic when load is spread evenly. That is a modest overhead for large N, but real.
Big tenants do not fit neatly. A tenant larger than a typical cell needs a cell of its own, or a larger one.

Challenge	Mitigation	Effort
N environments to manage	Strict infrastructure as code and automated cell provisioning	High
Global data needs	A thin control plane with caching at the cell edge	Medium
Cross-cell analytics	A separate asynchronous analytics pipeline	Medium
Moving tenants between cells	Dual-write migration tooling	High
Very large tenants	Dedicated or larger cells	Low
Spare capacity for failover	Spread load evenly, and use more, smaller cells	Low

Is cell-based architecture right for you?

The pattern is powerful, but it is not for everyone. Adopting it too early is premature optimization. Adopting it too late means living through incidents that cells would have contained.

It fits well when:

You run B2B SaaS with clear tenant boundaries
A single Availability Zone outage is unacceptable
Your volume is high enough that blast radius is a real concern
You have data residency or compliance requirements, such as EU-only data
Your team has strong infrastructure-as-code and platform engineering maturity

It fits poorly when:

You are an early-stage startup still finding product-market fit
Cross-user queries sit at the core of your product
You have a small team with no platform engineering capacity
Your workload is inherently global, such as a CDN or a search index
No natural partition key exists in your data model

Readiness checklist

Be honest with yourself before committing:

I can name a partition key that keeps more than 90% of related data together
All of my production infrastructure is covered by infrastructure as code
I can see per-service and per-dependency health, not only global averages
My organization will fund a platform investment of several months
I know which data is global and cannot live in a cell
I have sized the spare capacity needed to absorb a failed cell
I have, or can build, tooling to move a tenant between cells without downtime

As a rule of thumb, if you cannot tick more than a few of these, the architecture will fight you instead of helping you. Start with the gaps.

Conclusion

Microservices gave us deployment independence and team autonomy. Cells add something more fundamental: failure containment.

Traditional reliability engineering asks how to prevent failures. Cell-based architecture asks a different question: given that failures are inevitable, how far can we stop them from spreading? Cells do not reduce the probability of failure. They reduce its consequence, and at scale, reducing consequences is far more tractable than eliminating causes.

For a smaller company the economics differ, but the pattern scales down gracefully. Even three to five cells for your most critical service can cut your blast radius without a platform-wide overhaul.

The place to start is always the same: find your partition key. Identify the entity around which everything else pivots, and ask whether slicing along that line could save you from your next all-hands incident at 2 a.m.

DEV Community