Latency is often treated as if it can be simply "optimized". But it's much more complex than simply adding a CDN or buying a faster link. After reading (and re-reading) an assortment of books over the last few months, such as The Datacenter as a Computer, and, High-Performance Browser Networking, I've come to see latency not as a performance metric, but as a visible expression of physics.
And if we want to create better networks, or fix the ones that we already have, we need to treat latency as a physical constant and not as a tunable parameter.
What Latency Truly Is (Beyond Ping Time)
Latency is the sum of several delays, including:
Propagation Delay (the time it takes for a signal to physically travel through fiber or air. This is bounded by the speed of light.)
Transmission Delay (the amount of time required to push bits onto the wire.)
Queuing Delay (time spent waiting in buffers during congestion.)
Processing Delay (time spent in protocol stacks, encryption, and application logic.)
These concepts are foundational for understanding modern distributed systems, and latency is a distribution shaped by physics and queuing theory.
Why We Cannot Route Around the Speed of Light
You cannot route around the speed of light. This is the conclusion reached across the systems literature I've studied and by professionals such as Jeff Dean, Luiz Andre Barroso, Martin Kleppmann, and others whose work consistently shows that latency is bound by physics and not just engineering optimism.
In The Tail at Scale, Jeff Dean demonstrates how latency floors dominate distributed systems behavior, especially at the tail; hence, the book's title. Barroso and Hölzle’s The Datacenter as a Computer makes the physical layout of compute clusters a first‑class design constraint, explicitly tying propagation delay to datacenter geography. Ilya Grigorik’s High Performance Browser Networking goes even further, showing that even “quick” networks are limited by the speed of light through fiber. All these works together summarize that distance is latency, and latency is correctness.
You can optimize things like queuing, processing, and protocol overhead, but you cannot shorten the distance between two points without changing the topology.
What Is Latency Floor?
If two systems are, say, 3,000 kilometers apart, there is a hard lower bound on latency, even in fiber, where light travels at nearly two-thirds of its vacuum speed. This is known as the latency floor. This means that you can shorten the path and reduce the processing overhead, but geography will remain an obstacle.
What We Can Really Do About Latency
You obviously can't change physics. But you can change how systems are designed around it. And after researching the above topics, I've come to believe that the industry has had a more backwards approach to latency: we've focused on optimizing everything except for the part that physics controls. The result of this approach is a system with a normal-looking dashboard and unpredictable reality.
But the realistic solutions are not tricks or optimizations, but architectural corrections. The methods work because they all align with the physical and mathematical realities that shape latency. Below is an explanation of each possible solution, why it works, and how it changes the future of network design.
1: A Topology-Aware System Design
Most of the systems out there are designed as if the network is flat, and every service is reachable from every other service. But since, as I explained before, the distance between nodes determines the latency floor, no amount of optimization can actually push below that.
Topology-aware design reduces the physical distance that data must travel. This directly lowers propagation delay. What this may look like in practice is co-locating tightly coupled services, regionalizing deployments, and designing microservices around locality boundaries. And if you use areas, or "zones" that reflect physical layout, not just logical grouping, this could eliminate the flat-network assumption that causes many latency and coordination problems.
2: A Latency-First Protocol
Protocols such as QUIC and HTTP/3 are considered modern, but they are also designed around the reality that latency is dominated by handshake overhead and retransmission behavior. Latency-first protocols reduce the protocol overhead portion, the portion caused by round trips and blocking semantics. Key improvements could include: QUIC eliminating TCP's head-of-line blocking, 0-RTT reducing handshake latency, and modern congestion control adapting to jitter and queuing.
3: Tail-Latency Engineering
Optimizing for average latency is what most systems do. But average latency is irrelevant to correctness. Systems will perish at the tail (P99, P99.9, and beyond). Tail-Latency engineering reduces the variance in latency, not just the median. This stabilizes distributed systems, consensus protocols, and even automation inference pipelines.
4: Optical Bypass Paths
Optical bypass paths reduce latency by minimizing the number of intermediate hops between two points. Rather than routing traffic through several routers and/or switches, a bypass creates a direct optical path, often using something called DWDM (dense wavelength division multiplexing) between distant regions or datacenters. Every hop adds processing or queuing delay and potential jitter. Optical bypass paths, however, eliminate these intermediate steps, reducing both propagation and processing delay. This creates a sort of "express lanes" between major regions.
5: Hardware-Accelerated Transport Paths
A lot of modern networks increasingly rely on SmartNICs, DPUs, and hardware-accelerated transport stacks to reduce processing delays. These devices offload packet processing, encryption, congestion control, and even parts of the transport protocol from the CPU. Processing delay is a significant side of end-to-end latency, especially under load. Hardware acceleration reduces this delay dramatically and also reduces jitter by eliminating CPU scheduling variability.
(This is a strategy used by many high-performance datacenter networks)
6: Latency-Explicit APIs
Most API's treat latency as an implementation detail. But latency-explicit API's expose latency expectations directly to clients. For example, an API might specify that a particular operation has a 20–30 ms latency floor due to geographic constraints, or that a write operation may incur tail latency under certain conditions. By making latency explicit, APIs allow clients to make better decisions about retries, timeouts, caching, and locality.
In Conclusion
Latency won't vanish, but the way we design around it can. The future of networking isn't simply "speedier" networks or clever optimizations; the future is systems that truly acknowledge physics as a design constraint, and behave predictably and efficiently because of it.
Sources and Further Reading
Jeff Dean & Luiz André Barroso, The Tail at Scale: https://research.google/pubs/pub40801/
Luiz A. Barroso, Jimmy Clidaras, Urs Hölzle, The Datacenter as a Computer: https://cs.yale.edu/homes/yu-minlan/teach/csci599-fall12/papers/dccomputer.pdf
Ilya Grigorik, High Performance Browser Networking: https://hpbn.co/
Brendan Gregg, Systems Performance: https://www.oreilly.com/library/view/systems-performance-2nd/9780136821694/
Martin Kleppmann, *Designing Data‑Intensive Applications *: https://www.oreilly.com/library/view/designing-data-intensive-applications/9781491903063/
Andrew S. Tanenbaum, Computer Networks: https://www.pearson.com/en-us/subject-catalog/p/computer-networks/P200000003507/9780132126958
W. Richard Stevens, TCP/IP Illustrated: https://www.informit.com/store/tcp-ip-illustrated-volume-1-the-protocols-9780321336316
Top comments (1)
I don't typically write in this academic style, but I've been digging into a lot of systems books for the past couple of months to try and make sense of how all these ideas fit together. This post is my best attempt so far to piece together and share what I've been studying, so please excuse any possible mistakes or oversights; I'm still learning!
I'd really like to hear other perspectives about latency and network design. I'm curious how this lines up with what you've seen in practice, and I'd love to learn from your experiences!