DEV Community

Cover image for System Bottlenecks Beyond Code: Why Rewriting in Rust Won't Save You
NullPointer
NullPointer

Posted on Originally published at thelp.me

System Bottlenecks Beyond Code: Why Rewriting in Rust Won't Save You

#architecture #devops #systemdesign #performance

When a production service starts choking under load, developers immediately attach profilers, tweak SQL queries, or debate rewriting the microservice in Go or Rust.

Yet in modern distributed systems, actual code execution accounts for less than 5% of total request latency.

Your code runs on Linux kernel schedulers, memory tables, network cards, and SSD controllers. When latency spikes, your application has almost certainly hit a physical hardware wall or an operating system boundary.

Here is a short guide to the real bottlenecks in modern systems.

1. The Latency Gap (If 1 CPU Cycle = 1 Human Second)

To see why optimizing for loops rarely moves the needle, look at physical hardware latency:

Operation Real Time Scaled (1 Cycle = 1 Sec) The Impact
L1 CPU Cache 0.5 – 1.0 ns 1 – 3 seconds On-chip SRAM, lightning fast
Main RAM (DRAM) 60 – 100 ns 3.5 – 5.5 minutes Memory bus & controller delays
NVMe SSD 4KB Read 10 – 50 μs 9.5 – 48 hours PCIe bus & NAND flash access
fsync / WAL Flush 0.5 – 2.0 ms 20 – 80 days Synchronous disk durability barrier
Cross-Region Network 80 ms 8.5 years Speed of light in optical fiber

💡 Takeaway: If an endpoint spends 2ms running business logic and 50ms waiting on network & disk I/O, optimizing your code by 50% only reduces total latency by a negligible 1.9%.

2. The Kubernetes CFS Throttling Trap

Ever seen a container hit massive latency spikes while Prometheus reports only 15% CPU usage?

You just met Linux CFS Quota Throttling.

Host: 8 CPU Cores | Container limit: 1 CPU (100ms quota per 100ms period)

Thread 1 (Core 0) ──► [12.5ms] ┐
Thread 2 (Core 1) ──► [12.5ms] │
Thread 3 (Core 2) ──► [12.5ms] ├─► 8 threads × 12.5ms = 100ms Quota Burnt!
Thread 4 (Core 3) ──► [12.5ms] │   (Entire quota gone in just 12.5ms!)
...                            │
Thread 8 (Core 7) ──► [12.5ms] ┘
                                 ▼
[ HOST KERNEL FREEZES CONTAINER FOR NEXT 87.5ms ]
Enter fullscreen mode Exit fullscreen mode
  • In Kubernetes, setting resources.limits.cpu: "1" gives the container a 100ms budget per 100ms window.
  • Runtimes like Go, Java, and Node run multiple threads. If 8 threads run simultaneously, they burn through the 100ms quota in just 12.5ms.
  • The Linux kernel freezes all container threads for the remaining 87.5ms.
  • The Monitoring Blindspot: Dashboards average CPU over 1-minute intervals and show a peaceful ~15% utilization, hiding the 80ms freeze on every burst.

🛠️ Fix: Avoid hard CPU limits on latency-critical services unless strict multi-tenancy requires it. Rely on resources.requests.cpu for scheduling.

3. Ephemeral Port Exhaustion (EADDRNOTAVAIL)

Linux has roughly 28,000 ephemeral outbound ports. When an unpooled client closes a TCP socket, the OS keeps that port locked in TIME_WAIT for 60 seconds.

1,000 unpooled requests/sec × 60s TIME_WAIT = 60,000 sockets locked in TIME_WAIT
28,000 available ports  ──►  Exhausted in under 30 seconds!
Enter fullscreen mode Exit fullscreen mode

Once exhausted, the OS throws EADDRNOTAVAIL (Cannot assign requested address), crashing all outbound calls while CPU and RAM sit near 0%.

🛠️ Fix: Always reuse connections with persistent connection pools (HTTP Keep-Alive, HTTP/2, or gRPC). 50 pooled sockets can easily handle 10,000+ RPS.

4. Bandwidth-Delay Product (BDP): Why 10 Gbps Links Crawl

Why does a 10 Gbps cross-region connection often stall at 400 Mbps?

Link Capacity = 10 Gbps | Round-Trip Latency = 80ms (US to EU)
Required in-flight buffer to saturate link: 10 Gbps × 0.08s = 100 MB

Linux default TCP receive buffer: 4 MB
Time to fill 4 MB buffer: 3.2 ms ──► Sender stalls for 76.8 ms waiting for ACK!
Enter fullscreen mode Exit fullscreen mode

The sender spends 96% of its time waiting for TCP ACKs, artificially capping speed at 4MB / 0.08s = 400 Mbps.

🛠️ Fix: Tune Linux socket buffers (net.ipv4.tcp_rmem and net.ipv4.tcp_wmem) to match the Bandwidth-Delay Product (BDP).

5. The Database Connection Pool Paradox

Setting database pool sizes (e.g., HikariCP, Django) to 500 connections hurts performance instead of helping it.

  • Databases spawn an OS process (PostgreSQL) or thread (MySQL) per connection.
  • When 500 connections hit a 16-core database, the CPU burns cycles on kernel context switching, cache invalidation, and row lock contention. 
Optimal Pool Size = (Physical CPU Cores × 2) + Effective Disk Spindles
Enter fullscreen mode Exit fullscreen mode

For a 16-core database on NVMe SSD, 32 to 40 connections consistently yields 300% higher throughput and 80% lower p99 latency than 500 connections.

6. The Keep-Alive Mismatch (Sporadic 502s) 

Upstream Load Balancer (ALB) idle timeout : 60 seconds
Downstream App Server idle timeout        : 15 seconds
Enter fullscreen mode Exit fullscreen mode

If the app server closes an idle socket at second 15 just as the load balancer routes an incoming request, the request hits a closed TCP socket, producing intermittent 502 Bad Gateway errors.

🛠️ Fix: Always ensure App Server Keep-Alive timeout > Load Balancer idle timeout (e.g., ALB = 60s, App Server = 65s).

Quick Production Architecture Checklist

System Layer Trap Production Practice
K8s CPU CFS Quota Throttling Use CPU requests; monitor cfs_throttled_periods
Outbound HTTP Port Exhaustion (TIME_WAIT) Reuse persistent connection pools (Keep-Alive)
Database Massive Connection Pools Size pools to (2 × Cores) + Disks (30–40 conns)
Edge / ALB Keep-Alive Race (502s) Downstream timeout > Upstream timeout
WAN Transfers Socket Buffer Starvation Scale TCP window / buffer sizes to match BDP

📖 Full Engineering Deep Dive:

For complete architecture diagrams, Goetz's thread sizing formulas, Write Amplification breakdowns, and cache stampede (XFetch) algorithms, check out the full article on:

👉 System Bottlenecks: Beyond Application Code

Top comments (0)