TL;DR Architecture
At high level:
- Global Traffic Routing (Geo DNS / Anycast)
- Edge Layer (CDN + WAF + Rate Limiting)
- Load Balancers (L4 + L7)
- Stateless App Tier (autoscaled microservices)
- Multi layer Caching (edge, distributed, local)
- Data Layer (sharded DB + replicas + queue based writes)
- Async Processing (Kafka / stream workers)
- Observability + Auto healing
1) Capacity Planning First (Quick Math)
Never start with boxes and arrows. Start with numbers.
Assume:
- 100K RPS peak
- 95th percentile response target: < 150ms
- Read heavy workload: 90% reads, 10% writes
- Payload average: 2KB response
- Peak egress =
100,000 * 2KB = ~200MB/s(~1.6 Gbps, ignoring overhead)
Now read/write split:
- Reads: 90K RPS
- Writes: 10K RPS
Without caching, DB is dead.
So design target: serve 80–95% reads from cache.
If 90K reads and 90% cache hit:
- DB reads drop to 9K RPS (huge difference)
2) Request Flow (End-to-End)
Client
↓
Geo DNS / Anycast
↓
CDN + WAF + DDoS Protection
↓
L4 Load Balancer (TCP/connection distribution)
↓
L7 Load Balancer / API Gateway (routing, auth, throttling)
↓
Stateless Service Pods (Kubernetes / ECS)
↓
Cache Layer (Redis Cluster + local in memory cache)
↓
Database Layer (primary + read replicas + shards)
↓
Event Bus (Kafka/Pulsar) for async writes and side effects
3) Load Balancing Strategy (100K RPS Safe)
L4 + L7 combo
- L4 LB handles raw connection distribution efficiently.
-
L7 LB/API Gateway handles:
- path based routing
- JWT/auth checks
- per client rate limits
- request shaping
Algorithms
- Use least request or EWMA latency based routing over round robin.
- Enable slow start for new pods (avoid sending full traffic instantly).
- Circuit break unhealthy nodes quickly with active health checks.
Multi region failover
- Active in at least 2 regions for true HA.
- Traffic policy:
- nearest region by latency
- fail to secondary if SLA drops
4) Caching: Your Main Weapon Against DB Overload
At 100K RPS, caching is not optional.
Multi layer cache design
- CDN Cache (public GET responses, static + semi dynamic)
- Service side Distributed Cache (Redis Cluster)
- In process Local Cache (hot keys, tiny TTL)
Recommended pattern
- Cache aside for most reads.
- Key format namespaced:
user:v3:12345:profile
- TTL with jitter:
- e.g. 120s ± random(0–30s)
- prevents thundering herd at exact expiration moment.
Prevent cache stampede
Use:
- Request coalescing (single flight per key)
- Soft TTL + background refresh
- Distributed lock for ultra hot keys
Hot key protection
If one key gets massive traffic:
- Replicate hot key across multiple Redis slots
- Or keep a near cache in every app instance
- Add tiny randomized response delays for abusive clients
5) Database Design Under High Load
Primary principles
- Keep app tier stateless, keep DB tier protected.
- Reads should mostly hit cache or read replicas.
- Writes should be controlled and often asynchronous where possible.
Read scaling
- Primary + N read replicas
- Route stale tolerant reads to replicas
- Use follower reads with max replication lag guard (e.g. < 200ms)
Write scaling
For 10K writes/sec:
- Partition by tenant/user/hash key
- Use sharding for horizontal scaling
- Avoid cross shard joins in hot paths
Index strategy
- Index only what query patterns demand
- Watch write amplification from too many indexes
- Add covering indexes for hottest reads
Connection management
- Use DB proxy/pooler (PgBouncer, RDS Proxy, etc.)
- Enforce max connections per service
- Backpressure before DB collapse
6) Async Architecture for Spikes
Not every request must complete all work synchronously.
Use an event stream (Kafka/Pulsar):
- API writes critical data quickly
- Emits event
- Background workers process:
- emails
- analytics
- notifications
- search indexing
- cache warming
This decouples user latency from heavy downstream work.
Queue safety patterns
- Idempotency keys
- Retry with exponential backoff
- Dead letter queues (DLQ)
- Exactly once is hard; prefer effectively once with dedupe
7) Failure Handling & Resilience
At 100K RPS, partial failures are normal.
Must have patterns:
- Timeouts everywhere (client, service, DB, cache)
- Circuit breakers to isolate failures
- Bulkheads to isolate resource pools
- Rate limiting (per IP/token/tenant)
- Load shedding for non critical traffic
- Graceful degradation (return partial response instead of 500)
Example degradation:
- Personalized recommendations fail → show trending defaults from cache.
8) Observability (If You Can’t See It, You Can’t Scale It)
Track these golden signals per service:
- Latency (p50/p95/p99)
- Traffic (RPS)
- Errors (4xx/5xx)
- Saturation (CPU, memory, queue depth, DB connections)
Also monitor:
- Cache hit ratio (global + per endpoint)
- Redis latency + evictions
- DB replica lag
- Kafka consumer lag
- LB healthy target count
Use distributed tracing (OpenTelemetry) for cross service bottlenecks.
9) Security & Abuse Controls at Scale
High traffic attracts abuse.
Add:
- WAF managed rules
- Bot detection and challenge
- API keys/JWT scopes
- Per consumer quota tiers
- Payload size limits
- Schema validation at gateway
For sensitive operations:
- Idempotency tokens
- Signed requests
- Replay protection (nonce + timestamp)
10) Cost Aware Scaling
100K RPS can become very expensive very quickly.
Cost controls:
- Push cache hit ratio upward (biggest saver)
- Use autoscaling with smart cooldowns
- Rightsize pods and DB nodes continuously
- Use spot/preemptible nodes for stateless workers
- Tiered storage for old data
- Archive cold analytics out of primary DB
11) Example Technology Stack (Reference)
- Edge: Cloudflare / Fastly / Akamai
- LB/API: NGINX / Envoy / AWS ALB + API Gateway
- App: Go / Rust / Java (stateless containers)
- Cache: Redis Cluster (+ local LRU cache)
- DB: Postgres sharded / MySQL Vitess / Cassandra (depends on access patterns)
- Stream: Kafka
- Orchestration: Kubernetes
- Observability: Prometheus + Grafana + OpenTelemetry + Tempo/Jaeger
12) “Surprise Me” Section: Make It Sexy ✨
Here are advanced touches that make this architecture stand out:
A) Adaptive TTL Cache
Instead of fixed TTL, auto tune TTL based on key popularity + update frequency:
- hot and stable keys => longer TTL
- volatile keys => shorter TTL Result: higher hit ratio without stale data explosion.
B) Predictive Cache Warming
Use yesterday’s traffic patterns + upcoming events to pre warm top keys before peak hours.
C) Brownout Mode
When system stress is high, automatically disable non essential features:
- hide expensive widgets
- skip deep personalization
- defer non critical writes Users still get core functionality fast.
D) Cell Based Architecture
Split platform into isolated “cells” (mini stacks with their own app/cache/db shards).
If one cell fails, blast radius stays contained.
13) Minimal Launch Plan (Practical Roadmap)
Phase 1 (up to ~10K RPS)
- Single region
- L7 LB + autoscaled stateless services
- Redis + primary DB + replicas
- Basic metrics + alerts
Phase 2 (~10K to 50K RPS)
- Add CDN caching
- Add queue/event driven async workers
- Read/write splitting, stronger rate limits
- Better tracing and SLOs
Phase 3 (~50K to 100K+ RPS)
- Multi region active
- DB sharding + cell architecture
- Advanced cache protections
- Brownout + load shedding automation
- Game day chaos testing
Final Takeaway
At 100K RPS, success is less about one “super database” and more about traffic shaping + caching + graceful degradation.
If you remember one formula, make it this:
Scale reads with caches, scale writes with partitioning + async workflows, protect everything with backpressure.
That is how you survive high load without burning your database—or your team.
Top comments (0)