I once spent quite some time optimizing a database query that was bringing our API to its knees. We shaved 200ms off response time with careful indexing and query rewriting. Success, right?
Wrong.
Within a week, a different bottleneck surfaced. The network became the constraint. We optimized that. Then client-side rendering became slow. Then memory usage spiked. Each "fix" just moved the problem downstream.
Eventually, I learned something that changed how I approach performance work: bottlenecks aren't anomalies to eliminate, they're laws of nature baked into how systems work.
Why Bottlenecks Are Inevitable
The Theory of Constraints
Every system has a weakest link. Think of water flowing through a pipeline, the narrowest section determines your throughput, no matter how wide the rest of the pipe is.
Software systems work the same way. At any given moment, something is the binding constraint:
- Is it the CPU?
- The database?
- Network latency?
- Disk I/O?
- Memory?
When you optimize that constraint, you don't eliminate constraints altogether. The constraint simply moves to the next-slowest component. The system's total throughput increases, but a new bottleneck emerges.
Amdahl's Law: The Math That Proves It
Even with infinite parallelization, there's a mathematical ceiling. Amdahl's Law tells us:
If just 10% of your code runs sequentially, you can never speed up more than 10x—no matter if you have 1,000 CPUs. That sequential 10% becomes an eternal bottleneck.
For our social media app, some processes must happen in order:
- Authenticate user
- Fetch permissions
- Then fetch feed
You can't parallelize that without breaking safety. So authentication becomes a permanent bottleneck, though a small one.
The Great Trade-Off Game
Here's the dirty secret: optimizing for one thing almost always hurts something else. You're not solving problems; you're choosing which constraint to live with.
| Optimize For | What Improves | What Gets Worse |
|---|---|---|
| Latency | Fast responses | CPU/memory usage spikes |
| Throughput | More requests/sec | Resource exhaustion |
| Cost | Cheaper infrastructure | Performance degrades |
| Consistency | Strong guarantees | Write latency increases |
| Availability | Always online | Coordination overhead explodes |
Pick one. You're implicitly de-prioritizing others.
Example: The Classic API Scaling
Let's say I worked on an API that handled 10 requests/second. Bottleneck? The database.
We added caching.
Now: 100 requests/second. Bottleneck? Memory, our cache grew too big.
We sharded the cache.
Now: 500 requests/second. Bottleneck? Network bandwidth between nodes.
We optimized serialization.
Now: 1,000 requests/second. Bottleneck? Client-side parsing of responses.
We gzipped responses.
Now: 2,000 requests/second. Bottleneck? The upstream service that feeds our API.
At each step, we "solved" the problem. But we never eliminated bottlenecks—we choreographed a dance, moving them from place to place.
Distributed Systems Make It Worse
In a distributed system, bottlenecks hide in shadows and move in unpredictable ways:
Network RTT (round-trip time) is a physical law. You can't send data faster than light travels. For a global system, that's roughly 50-150ms of minimum latency per hop. Every extra network call multiplies this cost.
Leader election is expensive. When a leader fails in your distributed system, electing a new one takes time. During that window, writes are blocked. The leader election itself becomes the bottleneck.
Quorum writes create coordination overhead. To ensure consistency across replicas, you need a quorum (majority) to acknowledge a write. That coordination, waiting for responses from multiple nodes, is itself a bottleneck that throttles your write throughput.
Hot partitions kill performance. If all your traffic hits one partition (shard) of data, that partition becomes the bottleneck. You optimized everywhere else, but one hot partition brings the whole system down.
Cross-region latency compounds. Replicate data to multiple regions for resilience? Now reads and writes must cross regions. A single cross-region call might be 100ms. Add three of them together? You've hit a bottleneck that optimization can't touch.
Fan-out requests multiply latency. Your service calls 10 microservices in parallel. The slowest one determines your response time. You optimize 9 of them, but the 10th — now that's your bottleneck.
There is no way around these trade-offs. They're mathematical, physical. Bottlenecks in distributed systems aren't bugs, they're features of reality.
The Practical Mindset Shift
The question shouldn't be: "How do I eliminate bottlenecks?"
It should be: "Which bottleneck am I willing to live with, and how do I make sure it's the right one?"
Actionable practices:
1. Profile first, optimize later
Use a profiler. Find the actual bottleneck, not the one you think exists. 90% of optimization effort goes to wrong places because developers guess.
2. Optimize strategically
Focus on the binding constraint. Optimizing something that's 5% of your runtime is pointless if the database is 80% of your runtime.
3. Plan for the next bottleneck
When you fix something, ask: "What becomes slow now?" You can't prevent it, but you can prepare.
4. Make trade-offs consciously
Don't accidentally sacrifice something important. If you add aggressive caching, you're choosing performance over freshness. Make sure that's actually what you want.
5. Accept limits gracefully
Some bottlenecks are laws of physics (network latency, disk I/O). Design around them instead of fighting them. Use caching for latency. Use batching for throughput.
The Bottom Line
Some of the best engineers I know aren't the ones who "eliminate" bottlenecks. They're the ones who:
- Understand where bottlenecks live
- Move them strategically
- Stay ahead of the game by anticipating the next constraint
- Optimize for what actually matters (user experience, cost, reliability)
Your job isn't to break the laws of systems theory. It's to work within them intelligently.
The bottleneck never disappears. You just get better at dancing around it.
Further Reading
- Eliyahu Goldratt: The Theory of Constraints
- Gene Amdahl: Amdahl's Law (1967)
- Brendan Gregg: Systems Performance (book) + his incredible blog
- Eric Brewer: CAP Theorem
Top comments (2)
The constraint just moves is the right frame — we hit the literal infra version of this. Set a 30s
statement_timeoutthe obvious way (connect_args={"options": "-c statement_timeout=30000"}), checkedSHOW statement_timeouton prod and it said2min. We sit behind Supabase's Supavisor in transaction-pooling mode, which silently drops libpq startup options. Fix didn't error, it just did nothing. Ended up wiringSET LOCALthrough a SQLAlchemybeginevent, only form that survives a pooler. Same shape as your Amdahl's Law point — the bottleneck doesn't disappear, it moves somewhere your monitoring isn't looking yet. Wrote up the rest of that arc here if useful: three dumb ways our prod got slowSET LOCALworkaround via SQLAlchemy events is a clever way to force it through Supavisor, though. perfectly proves the point: you fix one thing, and the complexity just shifts into a blind spot you didn't know you had to monitor yet.Thanks for sharing this, I'm definitely gonna give it a read!