TL;DR: Physics puts a hard 101 ms floor on a Mumbai to Sydney round trip. I measured 31 ms. Working out how that was possible taught me more about CDNs than the chapter of High Performance Browser Networking that explains them. Every number here is from ping and traceroute on my own connection.
The measurement that did not make sense
I was reading the chapter on latency and bandwidth, and it makes a claim you can check yourself: light in fiber travels at roughly 200,000 km per second, so the round trip between any two points has a hard floor set by the distance between them.
Mumbai to Sydney is 10,157 km on the great circle. That gives a round-trip floor of about 101 ms in fiber, and about 68 ms even in a vacuum. Nothing you deploy, buy, or optimize gets under that.
So I pinged a university in Sydney.
$ ping -n 4 www.sydney.edu.au
Ping statistics for 2600:9000:237b:7a00:12:109e:7d40:93a1:
Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 31ms, Maximum = 117ms, Average = 62ms
31 ms. That is a third of the physical floor.
Either the chapter was wrong, or the thing answering me was not in Sydney.
Latency is four things, not one
Before getting to the answer, the chapter's actual model is worth having, because "latency" as a single number hides everything interesting. Every hop between me and a server charges four separate delays:
- Propagation delay. Distance divided by the speed of the signal in the medium. This is the one governed by physics.
- Transmission delay. Time to push the bits onto the wire, set by the link's data rate and the packet size. This has nothing to do with distance.
- Processing delay. Reading the header, checking for errors, deciding where the packet goes next.
- Queuing delay. Time the packet sits in a buffer waiting its turn.
The split matters because the four scale with completely different things. Propagation scales with distance, and you cannot buy your way out of it. Transmission scales with link speed, and that is genuinely what more Mbps buys you. Processing and queuing scale with how busy the path is right now.
My route to google.com was 12 hops. Every router added processing and queuing, and every link between them added propagation and transmission. All four, twelve times over.
What the numbers actually looked like
I pinged a spread of hosts and checked what each one resolved to. The last column is the theoretical fiber floor for the city I believed the service was in.
| Target | Resolved to | Min RTT | Avg RTT | Fiber floor |
|---|---|---|---|---|
| www.sydney.edu.au | d1puhpgqrhw451.cloudfront.net |
31 ms | 62 ms | 101 ms (Sydney) |
| dev.to |
151.101.130.217 (Fastly) |
26 ms | 65 ms | n/a |
| github.com |
20.207.73.82 (Azure) |
26 ms | 38 ms | n/a |
| google.com | 37 ms | 121 ms | n/a | |
| ping.online.net | Scaleway, Paris | 135 ms | 150 ms | 70 ms |
The Paris host is the control. It is a real machine in a known city with no CDN in front of it. Mumbai to Paris is 7,009 km, so the floor is 70 ms, and I measured 135 ms. Roughly twice the theoretical minimum, which is exactly what the chapter predicts once you add real routing, real queues, and the fact that cables do not follow great circles.
That line held up everywhere except Sydney.
The Sydney answer
Look at the resolved column. www.sydney.edu.au is a CNAME to d1puhpgqrhw451.cloudfront.net.
My packets never went to Australia. They went to a CloudFront edge somewhere near me, which had the page already. The origin in Sydney is only involved on a cache miss, and I never caused one.
This is the whole argument for a CDN, and I understood it much better after watching physics appear to break than I did from reading the paragraph that explains it. You cannot make light faster. The only remaining move is to make the distance shorter, which means putting the bytes closer to the person asking for them.
It also means a latency number is meaningless without knowing what answered. "31 ms to Sydney" was never true. It was 31 ms to a cache.
The part I did not expect
I assumed a ping to a fixed IP would be roughly stable. It is not. Twenty back-to-back pings to google.com:
Packets: Sent = 20, Received = 20, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
Minimum = 37ms, Maximum = 294ms, Average = 121ms
Same destination, same route, same distance, and nearly an 8x spread between best and worst.
None of that variance is propagation. The distance was identical on every one of those twenty packets. It is queuing delay, which is the chapter's point about buffers, and it is the same mechanism behind bufferbloat: routers with oversized buffers that would rather delay your packet for 200 ms than drop it.
The useful thing I took from this:
The minimum tells you about geography. The spread tells you about load.
My 37 ms minimum to google.com says there is a Google edge close to me. The 294 ms maximum says something on the path was congested. Those are two different facts, and reporting a single average of 121 ms throws both of them away. Latency is a distribution. Measuring it once tells you close to nothing.
Traceroute lies about the middle
The chapter suggests traceroute, so I ran it. My first hop, the home router, came back at 3 ms. The second hop, my ISP's first aggregation point, returned 28 ms, 65 ms, and 259 ms on three consecutive probes. That is the last-mile problem the chapter describes, and it was the least stable part of the entire path.
One caveat I would add to the book: intermediate hop times in traceroute are not very trustworthy. Routers treat the ICMP replies that traceroute depends on as low-priority work, so a hop can look slow while forwarding real traffic perfectly fine. A hop showing 607 ms is not proof of a problem there. I trust the end-to-end ping numbers a lot more, and I use traceroute for the shape of the path rather than its timings.
Why more bandwidth would not have helped
This is the part I had backwards. I had assumed a faster connection makes pages faster, roughly linearly.
Bandwidth is a supply problem, and supply is fixable. You can light up more fiber, lay more cable, or improve the multiplexing on cable you already have. The chapter notes that between 2007 and 2011 more than half the added trans-Pacific capacity came from better equipment on the ends of existing cables.
Latency has no such move. We are already running at about two thirds of the speed of light in fiber. Perfect materials and instant routers would buy something like 30 percent, once, forever. After that the only lever left is distance.
So if a page needs six sequential round trips before it renders, and each round trip costs 60 ms, that is 360 ms you cannot buy your way out of. Doubling the bandwidth changes none of it. Cutting the round trips, or moving them closer, changes all of it.
What I am doing with this
Three things I will actually carry forward:
- Check what a hostname resolves to before drawing conclusions from a latency number. Half the time you are measuring a CDN edge, not the service.
- Look at the minimum and the spread separately. An average hides the two facts you wanted.
- Count round trips, not kilobytes, when something feels slow.
I am reading this book chapter by chapter and writing up what survives testing on a real connection. Chapter 2 is TCP, which is where the round-trip cost I keep hand-waving at actually gets explained.
If you want to try this yourself it is three commands and about two minutes. Pick a host you believe is far away, run nslookup on it, then ping -n 20. I would like to know what your minimum RTT looks like against a CDN-fronted host versus a bare server in a city you can name, and where you are pinging from. My guess is that a lot of people are going to find their "distant" server is sitting a few kilometres away.
Book: High Performance Browser Networking by Ilya Grigorik, free to read online.
Top comments (0)