DEV Community: Productivity Hacks & Software Performance Analysis

Turing Technique: Trick I learned from Taleb to get better at chess and chess variants extremely quickly— Speed Test Demon

Productivity Hacks & Software Performance Analysis — Mon, 06 Sep 2021 08:14:25 +0000

(NOTE: If you disagree with me, I’ll be willing to listen on Twitter 😊: @SpeedTestDemon)

This technique was life-changing because it helped end a gaming addiction consuming nearly every moment of my day. And I got it from the famous author Nassim Taleb (thank you, eternally grateful). Other mental techniques I used to solve the addiction was affirmations and gradually replacing addiction with a meaningful hobby.

The Turing Technique is summarized as follows: the fastest way to learn a game that a computer has mastered… is to play against the computer. And memorize as many positions, moves, and patterns as you can. To be clear, this is **not just** about memorizing the beginning moves (also called opening theory by chess players), but also about improving anything in the middlegame and endgame.

I’m calling this the Turing Technique because this learning hack makes heavy use of the computer as a shortcut to mastery, and Turing is famous for pioneering computer science.

The Turing Technique is a rather bold claim. However, I speak from deep personal experience. This increased my crazyhouse (a popular chess-like game) rating by 217 Elo points (from 2233 to 2450) in only 4 hours of work for 5 days, or 20 hours total. And it finally helped end my crazyhouse addiction that I was desperate to end.

I’m certain the Turing Technique is the best way to learn chess — and I am certain beyond the shadow of a doubt that the Turing Technique is highly effective for simple chess-like games like crazyhouse due to my personal experience I write about in this essay.

Some people are immediately going to question whether I actually know what I’m talking about. Before crazyhouse, I’ve played chess for over 10 years and quit at National Master as I personally no longer cared about “chasing rating points.” Way back in the day when we were growing up, I’ve played International Master (IM) Eric Rosen and IM Kassa Korley and won both games (to be fair, it was a LONG time ago now) — I mention because it seems they’re both popular streamers now (I sometimes wonder if I went down that path…). And I’ve played Marc Esserman at Harvard Square, where he gave me an extra Queen and I still lost horribly (I was already a couple years retired from chess though 😅) — and then later he tried to sell me his Smith-Morra book. I know way more about the game than I probably should.

Skeptics are going to say the Turing Technique won’t work with chess, as the game is too large “to be memorized”. At the end, I have a full section of rebuttals addressing concerns such as these.

First, Thank you Nicholas Nassim Taleb

First, let me give credit where credit is due: Nassim Taleb deserves 80–90% of the credit for the Turing Technique. He might not remember these two tweets, but it totally changed my life completely (by putting an end to my gaming addiction).

My point is that if you want to practice skiing, you learn disproportionally more from hard runs (tail events) than from slopes for a beginner. Just as you gain more from lifting extremes (from convexity) than regular weights.

You learn disproportionally more from tails of distributions than you do from their center.

https://twitter.com/nntaleb/status/1060384417024217088

The Epiphany

You see it’s not a far jump from Taleb’s tweets to the Turing Technique. My first thought was: “What is the chess equivalent of a hard ski slope? Is it playing against Grandmasters?”. A moment’s pause. No…it’s playing against computers! What could be tougher than playing against computers? And it struck me that this must be correct. It must be the fastest learning method.

It’s like solving Rubik’s cube, which has thousands of combinations, but you only need to memorize a few key patterns to take you from start to finish.

How did this change my life? At the time I was extremely addicted to a chess variant called crazyhouse, spending 13+ hours a day playing the game. I had just successfully paused playing the game using mindset hacks, which I write about here and here. Then…I read Taleb’s tweet…commencing a funny “Angel vs Demon” dialogue inside my head:

Devil: “I know I want to stop playing this game, but…….what if I tried one more thing…”.
Angel: “That’s literally what every addict says. Do you really want to end up playing 13+ hours a day again? You just managed to stop playing, and now you want to jump back in?”
Devil: “But I think this time it’s different!”
Angel: “Aren’t those the most dangerous words in finance? This time it’s different?”
Devil: “But this time I think it’s really different!”
Angel: “Oh really? How is it different?”
Devil: “It really is different! I’ve never tried learning by playing against computers, and only computers…”
Angel: “OK, I guess you have a point. And if this learning technique doesn’t work? What’s the worst case scenario?”
Devil: “Then I guess it’s going to be 40+ hours of my life gone while trying to knock the addiction off again…”
Angel: “Sounds like you know the devil’s bargain. Your choice.”
Devil: “I just have to try it…Taleb’s tweet makes too much sense. I think this could really work…”

Taleb’s idea just made too much sense-the signal-to-noise ratio of learning from the computer must be very high…

The Turing Technique in Action

…so of course, I went ahead and tried the Turing Technique.

The plan was simple.

The “test procedure”: How to test my knowledge

Go to lichess.org and play crazyhouse against the lichess Stockfish computer
Allow myself as many take backs as I wanted.
Lose horribly despite all the take backs.
Resign the game and open the analysis board, which shows computer analysis. Find all the mistakes I made. Use curiosity to explore anything interesting.
Go to Step 1. Repeat.
Occasionally switch it up by playing taking the other side: play against the line I was trying to learn.

The “learn procedure”: how to learn

Open lichess analysis board, turn on Stockfish computer.
Play out opening and all moves I was interested in analyzing.
Then to test the new opening knowledge, run the “test procedure”. Then finally, the “real world test procedure”: play a few rated games against humans-but only a few…majority of time should be spent with the computer. This is to get real-world feedback on what to learn next. If a particular line was weak, then you know where to spend more time with the computer.

Results? End of the week report.

It was very consistent progress. Every single day I would put in 4 hours of training against the computer, then a few games against humans, and I would gain 50 points. By the end of the week, only 5 days later, my rating went from 2233 to 2450. That’s when I knew, this time in my gut, this visceral feeling…that I was done with crazyhouse. I was really done. My addiction was finally over. Because…if mastery means playing as close to a computer as possible, then I don’t want to play the game. Let me do something that computers cannot master. Let me be human.

Incidentally, my “Let me be human” desire resonates extremely similar to this clip I found about Bobby Fischer hating chess…

Why Bobby Fischer Hated Chess:

This section was a last-minute add to the essay, but I just had to add it because it fit so perfectly. I just watched this great clip of Bobby Fischer hating chess, and it ties in so perfectly with this essay. Why does he hate chess? Because of computers. Good quotes from the clip:

“I hate chess because I know what chess is all about. It’s about memorization, it’s all about pre-arrangement, creativity is lower down on the list!”.
NOTE on “pre-arrangement”: a terrible word choice by Fischer that implies conspiracy and collusion by players. This is NOT what he means. He’s referring to the grandmasters analyzing and memorizing the crap out of the first 20–30 moves in chess. His complaint is players aren’t finding moves over the board, but just playing from memory. A better word choice that chess people will understand would have been “extremely excessive opening theory”. Here’s another guy’s explanation of the same thing.
“Chess just the last few years has changed dramatically with all this computer stuff”
“But why would you want to get involved with something that is mainly rote and pre-arrangement?”
Nakamura comments: “When he talks about chess and how it’s getting harder and harder, that’s true of every game…he should be angry at computers, not angry at the game”.
I think Nakamura misses the point — I don’t think Fischer is angry about “chess getting harder”, but that it’s no fun playing a game that’s largely based on rote memorization and massive opening theory.
Tangential point: this is why he spent a lot of time trying to revive chess with FischerRandom, which throws away all the opening theory by randomizing the starting position.
And lastly, there’s this great Youtube comment that resonates so much with me that I’ll quote it here:
“I quit chess at 2200 for this reason. I “looked ahead” at FMs IMs GMs and saw what they did. They just memorize. They know all the endgame principles, are very good at calculation and objective evaluation, but ultimately… all that separates a 2200 from a 2500 is how many hours he spent on his engine memorizing a line in the Sicilian. Chess at the beginner level is its most pure form of the game. Wild sacs, fun mates and silly opening traps, kids smiling and old people sipping their coffee. Thats what chess is about, friends and community.. not the hyper competitive memorization and brute force analysis of tournaments today.”

There IS one thing I’d like to comment on Fischer Random since Bobby Fischer thinks it’ll bring back chess: the optimal way to learn Fischer Random is still ……playing against the computer! This time it’s not about memorizing a lot of the beginning moves but about recognizing patterns really well. Still, it feels so lame that the best learning strategy is to bang your head against the computer. Why not do something that only humans can do?

Rebuttals to potential arguments by skeptics

But how are you supposed to learn basic chess concepts by playing the computer?

Fair point. I’d say you still need to know the “big ideas” or “mental models” of the chess game in order to hang all the learnings on those hooks. However, the middlegame concepts can easily be covered in a book, while your specific opening patterns can be covered in a book as well. The rest of the time…?

How would you learn chess tactics? By learning how the computer crushes you with those tactics
How would you learn chess openings? By learning how the computer crushes you in those openings
How would you learn positional play? By learning how the computer crushes you in positional play.

Seeing a pattern?

You can still learn those good moves by learning from your losses against the computer.

If you really believe this is the best way to learn chess, why don’t you become a chess grandmaster (GM)?

For the same reason Bobby Fischer hated chess, dear reader. Why get involved in something that’s largely based on rote and memorization and opening theory? Scroll up to see the quotes from Bobby Fischer on chess.

And besides, I already proved for myself that the Turing Technique drastically speeds up learning in crazyhouse. That’s good enough for me. And I really don’t want to be like a computer. Let me be human.

Secondly, I’m sure out of the thousands of people who’ve read this blog post, there’ll be one person who takes it to heart. That person will succeed, and write a blog post going viral. You can let me know and I’ll write the blog post publicizing it for you.

But chess is much more complex than crazyhouse. You can’t learn chess just by playing a computer

Readers who are chess fans will have disbelief that the same rapid growth of 200 Elo points in such a short time (~20 hours over a week) could be achieved in a game as big and complex as chess.

Let me preface this by saying I’ve spent many years of my childhood playing chess. I was only a few points away from the National Master title, when I purposely quit when I realized the futility and pointlessness of these chess titles and rating points. I quit in protest and rebellion of the whole darned system.

Let me also say that I agree chess is a much bigger, complex game than crazyhouse. Chess has much more to memorize. Crazyhouse barely has an opening, and no possibility of an endgame.

However, I firmly believe the Turing Technique applies to chess as well. Just wait a moment: you realize grandmasters are famed for having insane recall of literally thousands of games? They can recall games played decades ago. There was one video that went mega-viral testing Magnus Carlsen’s memory.

Now why wouldn’t you expect the GMs to just memorize against the computer? It’s a miracle of the computing revolution that even cheap pocket phone computers have more processing power than the famous Deep Blue computer that beat Kasparov.

I would advise an aspiring chess player to learn from a book the “mental models” of chess such as pawn structure. But all the ideas can be easily covered in a single book (How to Reassess Your Chess by Silman was one book I read often in my childhood). Then, spend a ton of time with the computer, tying specific examples to the mental models.

It’s not going to be as quick as crazyhouse…it won’t take only 20 hours and 2 weeks of work. It might take 6–12 months to gain 200 Elo points. But the Turing Technique is without a doubt the fastest possible path.

Think about it. Do you really think Magnus Carlsen (or whoever the World Champ is) is not using a computer? Do you really think none of the grandmasters are using computers to analyze the crap out of everything? I remember Marc Esserman spending his entire Harvard winter break analyzing the hell out of the Smith-Morra, which is how he became the expert on that variation (By the way, I have actually met and played Marc Esserman at Harvard Square! Pretty chill dude 😊). Why has the age of “world’s youngest GM” been steadily declining, now to only 9 years old?

Rhetorical questions. The answers are obvious.

A related point: I believe one of the best things an aspiring chess player can do is to take a couple well known opening books and just memorize the crap out of it. One of the most common (bad) advice is advising beginners to focus on the middle game principles instead of studying openings. I had the most rapid growth (200 Elo points easily) in my “chess career” simply by studying the heck out of 3 chess opening books, every single line and position. Usually middle game performance improves from understanding the opening so well.

Why not play a human then analyze that game with a computer?

Some people might say the Turing Technique doesn’t sound that different from playing a game with a human and then analyzing it with a computer.

Au contraire. Couldn’t be more different.

Forget humans. You should literally play against computers, and mainly computers. Why? The signal-to-noise ratio is far higher. Even the slightest mistake is punished, and highly visible. And it is easy to explore many variations simply out of curiosity, with the computer showing you the best moves every step of the way.

The main benefit of playing against computers is being able to focus on exactly the problem area. It’s focused training. Playing against random humans on the internet…well you can easily go a whole hour until you get challenged in that one line you have trouble with. Playing against computers, you get heavily challenged, 10 times in a row, in a very short time.

Why not study Grandmaster games?

This is sort of the same idea as the Turing Technique, in that the goal is to study what the good moves and patterns are. However, the major missing weakness is the Grandmaster is not there to answer what would happen if a different move was played.

If you’re going to study GM games, better have computer analysis nearby so that you are able to explore questions easily. It’s a mistake to study GM games in isolation otherwise.

There is no crazyhouse book, hence no one has any idea what the “best moves” are. You can try copying the high-rated players, but what happens if you get taken off-variation…?

How To Learn Chess Openings?

A combination of an opening book and Turing Technique. One book per opening is probably fine. The opening book is mainly to learn the “mental models”. Then test and expand your opening knowledge fast and heavy with the Turing Technique. Play against the computer with both colors on the same opening.

Final Thoughts: Dear Younger Self…

I’m writing this as if I were writing a guide to my younger self.

Several years ago, I was severely addicted to crazyhouse and desperately wanted to shake off the addiction.

Here’s what I’d say to my younger self of several years ago: “Son. It’s simple. Just use the Turing Technique. Just play against the computer and memorize everything. That’s all there is to it. You can go do something else now. Don’t be addicted.” Followed by a wholesome hug 🥰

Don’t believe me that this Turing Technique works? Why don’t you try it out for 10–20 hours and come discuss it with me on Twitter about it.

I also have a theory on how to master first-person shooter games like Halo rapidly. However, I haven’t tested it yet. If it works I’ll write about it 😊. Subscribe if you’re interested in getting that update.

Random Thoughts I Couldn’t Fit Into Essay

The corresponding “hard ski slope” for language learning would be going to a bar in France and attempting to converse using French. This thought is also inspired by something Nassim Taleb said in a tweet, which I am too lazy to dig up at the moment (readers, help?).
Computers are significantly cheaper and more widespread, leading to ever-younger chess GMs from even countries with lower median income.

Python CDN Speed Test Script — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Sat, 21 Aug 2021 11:03:21 +0000

Simple python script I use to speed test CDNs. It’s open sourced under MIT license. The repository is: https://github.com/speedtestdemon/speed-tests and the python script is “test.py”.

The script takes a single input “URL”. Example usage: python3 test.py [https://d20zaq59cm4c4j.cloudfront.net/wp-content/uploads/2021/05/cf.png](https://d20zaq59cm4c4j.cloudfront.net/wp-content/uploads/2021/05/cf.png)\.

The script outputs these things:

Cold cache, hot cache, and warm cache tests (in that order).
The response headers for cold cache and warm cache tests. This is important to see the header that reports “Hit”, “Miss”, or “RefreshHit”. It ensures that you’re running the cold cache and warm cache tests correctly.
The hot cache times are averages of 10 curls.

Unique notes about the CDN Speed Test Script:

Here is an example output of the python script:

\-------------------------------------------------------------  
Testing "Cold cache speed"  
\-------------------------------------------------------------  
Got headers:  
HTTP/2 200  
content-type: image/png  
content-length: 719983  
date: Fri, 25 Jun 2021 15:01:17 GMT  
last-modified: Mon, 07 Jun 2021 00:16:21 GMT  
etag: "52ae2ff2354d4a68e680b77b4da58985"  
accept-ranges: bytes  
server: AmazonS3  
x-cache: Miss from cloudfront  
via: 1.1 9e8d1ff1eee770054f1561535ab320b2.cloudfront.net (CloudFront)  
x-amz-cf-pop: DFW55-C3  
x-amz-cf-id: lQPqV7PnAuwl\_52kDXS2rsC20kSrloThTOG8cN1fK-MHUce0NNIDIA==time\_namelookup: 0.12467000000000000304  
time\_connect: 0.02141099999999998560  
time\_appconnect: 0.06681200000000001027  
time\_pretransfer: 0.00020100000000000673  
time\_redirect: 0.00000000000000000000  
time\_starttransfer: 1.02790999999999987935  
time to download: 1.06810300000000002463  
time\_total: 2.30910700000000002063  
\-------------------------------------------------------------  
Testing "Hot cache speed"  
\-------------------------------------------------------------  
10 requests done. Average:  
time\_namelookup: 0.00173870000000000036  
time\_connect: 0.02068409999999999693  
time\_appconnect: 0.06124410000000000270  
time\_pretransfer: 0.00024639999999999802  
time\_redirect: 0.00000000000000000000  
time\_starttransfer: 0.03418949999999999767  
time to download: 0.21361620000000000608  
time\_total: 0.33171899999999998609  
\-------------------------------------------------------------  
Testing "Warm cache speed"  
\-------------------------------------------------------------  
Sleeping for 0.5 hr to move cache from hot to warm  
Got headers:  
HTTP/2 200  
content-type: image/png  
content-length: 719983  
date: Fri, 25 Jun 2021 15:01:17 GMT  
last-modified: Mon, 07 Jun 2021 00:16:21 GMT  
etag: "52ae2ff2354d4a68e680b77b4da58985"  
accept-ranges: bytes  
server: AmazonS3  
x-cache: Hit from cloudfront  
via: 1.1 7c556a1545ad08a01c9e86ad41ac73a1.cloudfront.net (CloudFront)  
x-amz-cf-pop: DFW55-C3  
x-amz-cf-id: ACpfQ14wbO\_vjjAisApBAJkRphZQ7IbD4Lawc77eo\_K5UM\_3-yWPZw==  
age: 1804time\_namelookup: 0.05657399999999999929  
time\_connect: 0.02411100000000000743  
time\_appconnect: 0.07374400000000000399  
time\_pretransfer: 0.00145299999999998208  
time\_redirect: 0.00000000000000000000  
time\_starttransfer: 0.02921799999999999398  
time to download: 0.22752000000000000002  
time\_total: 0.41261999999999998678

Test Methodology: how I test CDNs. I use this python script to test CDNs.

Cheat Sheet on Curl Performance Metrics: how to benchmark server latency with curl: explains the curl metrics outputted by the python script.

Jetpack CDN vs Amazon Web Services CloudFront: Jetpack surprisingly faster!, where I analyze Jetpack CDN using the python script.

Cloudinary vs CloudFront: Cloudinary crushes Amazon Web Services!, where I analyze Cloudinary using the python script.

Prediction: YC startup Mighty will be new default infrastructure as web VIP pass — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Sat, 21 Aug 2021 10:51:15 +0000

(Note: people, this is just my opinion, possibly overexcited and incorrect. Discuss cordially with me if you think I’m wrong 😊. I’m on Twitter @SpeedTestDemon or email list at blog. And by the way, I’m not affiliated with Mighty App whatsoever. I don’t know anybody who works there).

There was this big commotion on Twitter and the news media that Mighty App was just a glorified Desktop-in-the-cloud, and “why would anyone in the world pay for $30/month for this”. And noticeably Paul Graham on twitter tweeting many supporting tweets, including a “ There’s a good chance it’s the new default infrastructure “

For those who don’t know Mighty App, it’s a service where a datacenter runs Chrome (Chromium?), and is streamed to the user’s computer. Advertised benefits are: multi-day battery life, better browser experience.

While I personally was not 100% skeptic, I wasn’t 100% convinced either.

But then I had this epiphany while testing CDN speeds. I finally realized what could make Mighty App work.

It’s like getting a VIP pass in Disneyland with short lines. It’s like a toll road with almost no traffic. It’s like TSA pre-check line with few people.

What am I talking about?

It seems the main focus of all the Mighty App skeptics is they think Mighty is simply a faster computer hosted in the datacenter. You know: a faster CPU, more RAM, faster disk (NVMe?). And, the skeptics say, why would you pay $30/month for a faster computer when you could just shell out $1000 for a computer that lasts a few years?

The thing is, Mighty App’s performance gains are NOT just about “a faster computer”. If it were really just about “faster hardware”, I’d agree with the skeptics.

However……while those things are nice, and certainly help, what everybody is missing is the internet speed in a datacenter is blazingly faster than you could ever hope to buy at home.

The thing is that you simply cannot buy a “faster internet”. The fastest internet you could possibly buy, apparently, is Google Fiber. They advertise “max speed of 2000 Mbps”, but according to “ highspeedinternet.com”, they only have slightly higher average download speed of 160.5 Mbps, and average upload speed of 173.6 Mbps. Seems fast, right? And it’s fiber optic too!

Except if you run an internet speed test on a DigitalOcean VM, which I did while speed testing CDNs, it’s a whopping 1740 Mbps! Upload speed was 1420 Mbps.

What the hell! 1740 Mbps? You simply cannot buy that kind of internet unless you’re Tony Stark on a buying binge with no regard for cost. It is simply not commercially available.

Do you guys realize how crazy that 1740 Mbps number is? It’s 10x faster than the Google Fiber number of 160 Mbps. 10x faster.

Wait a second. Does this mean files will download 10x faster? Really? Let me check.

I test downloading a file from CloudFront…and the DigitalOcean VM downloaded the file in just 22 milliseconds. Seriously! While my home computer downloaded it usually around 300 milliseconds.

That’s right. 300 milliseconds vs 22 milliseconds. More than 10x speedup. Holy cow.

And you can easily reproduce the results yourself. Here’s a CloudFront URL you can use from your home computer vs DigitalOcean VM: https://d20zaq59cm4c4j.cloudfront.net/wp-content/uploads/2021/05/cf.png and here’s a tutorial on curl performance testing.

When I saw that 10x speedup show up in the terminal, was the moment I had the epiphany that Mighty App is really on to something.

How did DigitalOcean do that? Turns out it’s not just fiber optic cables. Cloud providers like DigitalOcean can have something called “peering”, which is much faster than usual internet connections. As a very simplified summary: “peering” is like running a cable between your computer and the server, while normal internet connections rely on many intermediate routers. In other words, it’s like getting in the special VIP line at Disneyland. They have a special line so that the data travels faster.

Do you think you can buy that at home? How are you going to get peering connections with the popular CDNs or services like Netflix? It is not simple to replicate that internet access.

So, that’s what I mean when I say the Mighty App is like a VIP pass to the internet. They literally have special access to the most popular CDNs and data centers.

And that’s just the beginning. There’s so many mind-boggling performance hacks that Mighty App can do.

Cache common internet pages so every user’s browser simply reads from disk instead of downloading from internet?
Buy ASICs (custom hardware) for TLS encryption. This is probably a 10% speedup by itself, and is extremely expensive hardware.
Heck, design a custom ASIC for JS? LOL.

Rebuttals

But SpeedTestDemon, you still have to account for the time it takes to stream a frame over! Answer: the good news is no matter how large the web app is, you’re only streaming that frame, while the web app is speedily downloaded (or cached!).

But SpeedTestDemon, 1 Gbps is going to be more and more available! Answer: I already talked about this. Even if you had Google Fiber, which advertised 2 Gbps max speed, you still only had average download 160 Mbps. Even if everybody in the world got Google Fiber, chances are pretty good you’re never going to get the “peering” agreements that data centers have, unless you’re Tony Stark and decide to buy your own data center.

But SpeedTestDemon, mightyapp.com tech specs advertise only “over 1 Gbps”, not 1740 Mbps! Answer: that may be true for now. My bet is if DigitalOcean could make it work, then so can Mighty.

But SpeedTestDemon, I want to say something you didn’t point out here! Fine, drop me an angry comment on this twitter thread I wrote for this article. I promise to at least read it 😊.

Cloudinary CDN totally smokes Amazon AWS CDN CloudFront — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Sat, 21 Aug 2021 06:22:04 +0000

(Note: after seeing stern feedback on Hacker News and reddit, I’m changing my test approach to hire 10+ contractors in popular cities around the globe to compare CDN download times. And if you disagree with my analysis in any way, please cordially do so 😊. I’m on twitter at @SpeedTestDemon)

I’ve learned Cloudinary basically does image and video transformations on top of the Fastly CDN. As such, I feel this is basically a review of the Fastly CDN platform. Fastly costs minimum $50/month, so it’s not free CDN I’m interested in analyzing. However, Cloudinary has a free tier going up to 25 GB of storage or bandwidth, so here’s “Analysis #2 of a Free CDN”.

Cloudinary was SUPER impressive. Cloudinary was way better than AWS CloudFront. They had a secret sauce: dramatic image transformation. Cloudinary somehow compressed a 703 KB image down to 37 KB.

Before I present the simplified summary, here are some links relevant to this article.

My testing approach to CDNs
Python script for measuring CDNs
Analysis #1 of a Free CDN: Jetpack CDN vs CloudFront
I tweet analysis about software performance or email list

As a very simplified summary, based only on hot cache results

Cloudinary was faster than CloudFront by 3x (0.133 vs 0.397 seconds, or difference of 264 milliseconds)
Even after controlling for image size, Cloudinary was still faster by 13% (0.11 vs 0.126)

Cloudinary had better performance for several reasons:

How did Cloudinary compress that image 95% smaller? (~700 KB image to ~36 KB).
Much more common DNS path (res.cloudinary.com vs d20zaq59cm4c4j.cloudfront.net), which allows for significantly faster DNS lookup. In one case, 0.11 vs 0.008 seconds.
Uses Fastly CDN underneath. A lot of big companies (including Amazon) have adopted Fastly, probably for a good reason.
CloudFront typically degrades a file from hot cache to warm cache very rapidly, whereas Cloudinary (i.e. Fastly) doesn’t do this until…weeks? (it’s a long time whatever it is).

What follows is a much more detailed technical analysis.

I had to do 2 speed tests. The 1st speed test was not apples-to-apples because Cloudinary had compressed the image by nearly 95%. The 2nd speed test, I controlled for file size, and this time Cloudinary came out ahead very slightly by 6% (although the main credit should be due to Fastly).

In the first speed test, Cloudinary really crushed CloudFront.

At first sight, I had to double-check for errors. Cloudinary was nearly 3x faster — on the hot cache! — and it didn’t make sense. Turns out the test was correct. Cloudinary simply compressed the image 95% lower (700 KB to 36 KB).

I had to do a second speed test to make it more apples-to-apples between CloudFront and Cloudinary (masquerading as Fastly). CloudFront did a lot better in the hot cache. Still sucked at the other two metrics.

There were several reasons for Cloudinary performing so well:

OK Cloudinary’s free tier is pretty impressive. The only limitation I can think of is you can’t host your whole site on it like you can with Cloudfront.

Now one moment…you do “get what you pay for”…while Cloudinary has great performance, it is limited in many other ways. You can’t update a file on Cloudinary easily, unless you wholesale rename the file (very annoying). The amazing performance applies mainly to images. JS and CSS won’t have the same amazing image compression, although you will see Cloudinary perform on par with CloudFront. See the conclusion for my thoughts on the Cloudinary vs CloudFront decision.

It should be noted that not every free CDN performs as well as Cloudinary. I found a couple free CDNs that performed really poorly.

Why is it interesting to test Cloudinary?

Honestly it was just because they had a free tier CDN and my goal here was just to test every free CDN.

I didn’t know anything about Cloudinary before writing this article.

But wow, the results have blown my mind. Good job Cloudinary guys! These guys have a good product.

After doing some more research on Cloudinary, I found out these guys have won some interesting accolades

Inc. Magazine called Cloudinary the “gold standard” of image management on the web (according to Wikipedia at least).
Actually has rather large usage (unusual that I’ve never heard of them even though I’ve been in the industry for many years). Usage stats are 1 million developers and 7500 companies (Wikipedia).

OK, so it’s got a reputation for great image management. I can see that actually. They did insanely well in this test.

Why is it interesting to test CloudFront as the gold standard in CDN speed tests?

Because Amazon is Amazon and they’re the biggest cloud provider out there (they are #1 at 32% of the market). But Cloudinary’s impressive speed test results here have me really questioning why I’m using CloudFront. Well Cloudinary’s premium tier is very expensive at $100/month, so it’s not an obvious switch. CloudFront is only at $2/month for the equivalent network out.

Brief Summary Of The Metrics Being Measured

We are interested in the hot, cold, and warm cache speeds because CDNs are essentially a cache layer. If you’re interested in reading more, see the analysis here: metrics measured in Jetpack vs CloudFront and my CDN testing methodology.

Couple findings on why Cloudinary was so much better than Amazon AWS Cloudfront.

Finding #1: Cloudinary’s DNS lookup is significantly faster than CloudFront’s DNS lookup.

This was similar to what I found in Jetpack CDN vs Cloudfront. Cloudinary is able to achieve a significantly faster lookup time because it’s DNS path is more likely to be cached by the intermediate routers. Cloudinary DNS lookup path is “res.cloudinary.com” which seems to be the shared DNS path between most Cloudinary users, so that DNS lookup path is receiving a lot of traffic. While CloudFront’s DNS lookup path is a very customized “d20zaq59cm4c4j.cloudfront.net”.

What’s very interesting is Jetpack CDN’s DNS lookup times were always nearly as fast as the hot cache’s DNS lookup time (even the cold cache 0.00223 seconds was so close to hot cache 0.0016 seconds), while Cloudinary’s DNS lookup times took a small hit (hot cache 0.00175 seconds to warm cache 0.0236 seconds). I think this can be attributed to Jetpack CDN’s immense popularity which means its DNS lookup path “i0.wp.com” is going to be cached much more frequently than Cloudinary.

Finding #2: Cloudinary had a mind blowingly good image compression algorithm

I’m not exactly sure Cloudinary got it so good. The same 703 KB image was somehow compressed down to 36 KB. That’s just really good and explains how Cloudinary was able to get much faster Data Transfer times than CloudFront.

Other interesting findings

I already talked about these in previous essays ( Jetpack CDN vs CloudFront) so I didn’t want to write more paragraphs for these.

Generally, the TCP handshake and SSL handshake times are the same regardless of file size or file type. Most CDNs and servers are going to be equally fast on these metrics, unless they are using a slower CPU to do SSL computations.
CloudFront’s warm cache (a request 30 minute after the most recent) is about 2x to 4x slower than hot cache. I found this a lot when testing Jetpack CDN. CloudFront likes to degrade a file from RAM to disk (NVMe or SSD) if it’s not seen active requests recently.

The first time I ran this comparison test, I was blown away by how good Cloudinary was. I thought Jetpack CDN was a good Image CDN. I was wrong. Cloudinary was way better.

Cold cache: OK so Cloudinary was 5x faster than CloudFront….because Cloudinary doesn’t really have a cold cache? Because there is no expiration on the files (unless you get Cloudinary’s premium tier).
Hot cache: I guess this is the metric that really matters, since it’s the most common case that a CDN is going to contain the file. And boy did Cloudinary impress. By compressing the file from ~700 KB down to 36 KB, Cloudinary beat CloudFront by 3x.
Warm cache: Cloudinary did even better, beating CloudFront by 4x. Cloudinary hot cache didn’t degrade even after 30 minute wait time, while CloudFront once again degraded the file so it performed 33% worse (same thing I found in Jetpack CDN vs CloudFront).

Cold cache measurement and analysis (speed test #1)

CloudFront really lagged horribly here at 2.355 vs 0.395 seconds for Cloudinary.

Sharp readers will notice that CloudFront showed a much worse time (2.355 seconds) in this speed test comparison than in the Jetpack CDN vs CloudFront analysis (1.199 seconds). It’s because I just ran this CloudFront download at a different time than when I wrote the Jetpack CDN analysis.

Wait a second. CloudFront took 2.35 seconds while Cloudinary was only 0.39 seconds? Holy moly. Where did all that time go?

The two big time consumers were: 1) Request Latency, and 2) Data Transfer.

Regarding Request Latency: CloudFront took 1.022 seconds while Cloudinary took 0.258 seconds. What’s impressive is Cloudinary must have done two things during the cold cache lookup: it must have had the file still cached on a server despite my asking for the file only several days after getting that CDN setup. Cloudinary certainly did not fetch the file again, or it wouldn’t have gotten such a fast time. That’s really really good.

Cloudinary’s cold cache is not as fast as its hot cache. Most of the time went in Request Latency. For cold cache, it’s likely they had the file stored somewhere on disk, while for hot cache, it’s probably in RAM. Actually, since Cloudinary is built on top of Fastly, I would place a strong bet that Fastly has the file in RAM or else why are people buying Fastly?

Regarding Data Transfer: yeah this was really incredible. Cloudinary only needed 0.0069 seconds while CloudFront took 1.098 seconds. There’s only one main factor for this and it’s Cloudinary’s insane image compression algorithm which means Cloudinary only has to send 36 KB while CloudFront has to send 703 KB. It’s just not even close. See more analysis in Finding #2.

Regarding SSL handshake: honestly not sure why CloudFront took much longer at 0.091 seconds vs 0.056 seconds but the absolute difference is only 0.035 seconds, so it’s a small contributor to the overall delta of nearly 2 seconds.

Regarding TCP handshake: this is just the same for any connection. So there’s not a big absolute time delta.

Regarding DNS lookup: not the biggest factor for the large overall time delta but it’s still worth mentioning because on a relative basis, Cloudinary outperformed by over 100% (0.0494 seconds vs 0.124 seconds). See Finding #1 analysis for why Cloudinary’s DNS lookup is generally much faster than CloudFront.

Hot cache measurement and analysis (speed test #1)

This hot cache measurement was the one that really blew my mind.

Cloudinary smokes Cloudfront, nearly 3x faster (0.133 vs 0.397 seconds).

Wow! How did Cloudinary do that?

(Reminder: these times are an average of 10 curl calls).

Looks like all the times are pretty similar except for Data Transfer. In fact, Data Transfer accounts for 0.234 seconds of the overall time difference of 0.263 seconds. Literally about 90%.

It’s really shocking that CloudFront took so much longer to transfer the file, even in a hot cache state.

Reason is Cloudinary compressed the original 700 KB file down to 36 KB. That’s nearly a 20x reduction in file size. So no wonder Cloudinary ended up smoking the pants off CloudFront.

Notes:

Warm cache measurement and analysis (speed test #1)

Ouch. CloudFront nearly 4x slower than Cloudinary: 0.465 vs 0.126 seconds.

What happened??

(reminder: warm cache is download taken exactly 30 minutes after hot cache)

First off, a monstrous Data Transfer difference: Cloudfront 0.198 seconds vs Cloudinary 0.009 seconds. Absolute difference of 0.19 seconds, contributing to the overall time difference of 0.34 seconds.

Second, a monstrous DNS lookup difference: Cloudfront 0.115 seconds vs Cloudinary 0.0236 seconds. Absolute difference of 0.09 seconds, contributing to the overall time difference of 0.34 seconds. If you thought DNS lookup speeds don’t matter — well think again. See “Finding #1” for why CloudFront’s DNS lookup is so much slower than Cloudinary.

Data Transfer and DNS lookup combine for 0.28 seconds out of the 0.34 seconds difference.

Other notes:

As said in Finding #2, CloudFront’s Data Transfer was so much slower because it was transferring ~700 KB of data while Cloudinary was transferring ~36 KB. Cloudinary compressed the image really well.
Dang, Cloudinary’s warm cache faster than the hot cache time (0.126 vs 0.133 seconds). That’s impressive.
I don’t know why CloudFront’s SSL handshake is always slower than Fastly’s SSL handshake. I’m still thinking about it. (Cloudinary is built on Fastly’s CDN).
Yeah, CloudFront’s “Request Latency” was 2x slower. I’ve noticed CloudFront tends to degrade the file from RAM to NVMe or SSD.

Speed Test #1 conclusion and analysis

Cloudinary beat CloudFront in hot cache, cold cache, and warm cache, due mainly to the image automatically compressed to 36 KB and CloudFront still trying to transfer a 720 KB PNG.

Impressively, the hot cache was a staggering 3x faster: 0.133 vs 0.397 seconds.

What if we even the odds and make CloudFront and Cloudinary transfer the same file? Surely CloudFront should beat Cloudinary now? It’s not that simple. See the next section for Speed Test #2.

I re-ran the speed test to make sure CloudFront and Cloudinary were downloading the exact same file. To do this, I just uploaded the Cloudinary 36 KB file to CloudFront CDN.

Notes:

Cold cache: I guess Cloudinary keeps the file in hot cache for an absurdly long time (this is a good thing for the end-consumer. It does not cost extra money for this feature). it takes a long time for Cloudinary’s cache to turn cold…so I’m skipping this test.
Hot cache: Cloudinary still slightly faster (12%). Absolute difference is only 0.016 seconds. Not a big deal I guess.
Warm cache: after waiting 30 minutes, Cloudinary didn’t degrade the cache at all, while CloudFront faced a 2x slow down.

Cold cache measurement and analysis (speed test #2)

So….I found out Cloudinary doesn’t really have a cold cache? Even if I try running the test a few days after my last fetch…it is still pretty much the exact same as the hot cache result….

For example, I ran the Cloudinary test again, and I noticed the cold cache was only 0.02 seconds slower than the hot cache…nearly all of which was in the DNS lookup.

The coldest I’ve ever seen Cloudinary was when I waited a couple weeks to test Cloudinary. If you test only a few days after, it is pretty much the same as hot cache.

This is actually a good thing for the consumer. It means Cloudinary doesn’t degrade the file down to cold cache until a couple weeks. It is kind of incredible.

I’m not going to bother writing much of an analysis for this.

Cloudinary’s cold cache is pretty much as fast as the hot cache, with the only difference of the DNS lookup (was not cached by intermediate routers).

CloudFront’s cold cache is just the same as last time except for significantly faster Data Transfer phase since we are making CloudFront download 36 KB.

Hot cache measurement and analysis (speed test #2)

Much closer where we can say that perhaps CloudFront is as fast as Cloudinary. Cloudinary holds a slight 12% faster time.

Cloudinary still 12% faster than CloudFront. Although the absolute difference is only 0.016 seconds so perhaps not a big deal.

DNS lookup, TCP handshake, SSL handshake, Data Transfer were all pretty close (these are the same operations for any file size or type).

The main difference of 0.010 seconds comes from Request Latency, where CloudFront takes 50% long (0.030 vs 0.20 seconds). Honestly not sure but perhaps Cloudinary has a more efficient file cache read than CloudFront. Cloudinary is built on Fastly, which has significant market share among enterprise companies.

So Cloudinary is faster, but it’s not really worth stressing about it at this point. The difference is small enough.

Warm cache measurement and analysis (speed test #2)

Same thing that happened in Jetpack CDN vs AWS CloudFront. CloudFront warm cache degraded significantly (relative to hot cache) — 2x slower (0.25 seconds vs 0.126 seconds). Cloudinary warm cache pretty much the same as hot cache.

Regarding Request Latency: the big rock was Request Latency: 0.076 vs 0.0168 seconds. Cloudinary’s Request Latency was the same in both hot and warm cache, so Cloudinary must be keeping the file in RAM. CloudFront definitely downgraded the file from RAM down to disk. This is very consistent behavior that I’ve seen in the Jetpack CDN vs Cloudfront comparison as well.

(reminder: warm cache is download taken exactly 30 minutes after hot cache)

CloudFront’s speed performance is once again significantly slower as found when I was doing Jetpack CDN vs CloudFront. Cloudfront’s slowdown came in 3 areas: DNS lookup, SSL handshake, and Request Latency.

Regarding DNS lookup: Cloudfront is slower here because the DNS name d20zaq59cm4c4j.cloudfront.net is entirely unique to my account so it is even more unlikely to be cached in routers than the Cloudinary DNS name res.cloudinary.com (which appears to be shared with multiple users).

Interestingly, Cloudinary’s DNS lookup was significantly slower when going from the hot cache to the warm cache (0.035 seconds vs 0.0016 seconds).

Another interesting point is Jetpack CDN’s DNS lookup was only 0.0092 seconds. This happened because the Jetpack CDN DNS name i0.wp.com is heavily used across millions of wordpress installations and Cloudinary has significantly smaller install base, so the Jetpack’s DNS name is way more likely than Cloudinary to be cached in routers.

Regarding SSL handshake: it is really unusual that Cloudfront’s SSL handshake takes nearly 2x as long as Cloudinary (0.084 vs 0.045 seconds), because the TCP handshake occurred with equal speed on both CDNs, which indicates the problem is not the internet speed. There’s 2 possibilities: 1) for some reason the certificate authority is taking a long time to respond, and 2) the CloudFront is serving the traffic from a cheap CPU server so the SSL calculations are taking much longer?

I guess possibility #2 is more likely. Still weird.

My thoughts on this are extremely mixed. I would say it highly depends on your circumstances whether you should use Cloudinary.

Reasons to use Cloudinary

Cloudinary is a highly performant CDN built on top of Fastly CDN, very widely used among tech companies such as Reddit and even Amazon itself (which is weird because Amazon has its own CDN called CloudFront…).

Cloudinary has some very impressive image compression capabilities. The 700 KB test image was compressed down to 36 KB. It’s possible to do the same thing in CloudFront…but it’s more involved. You’d need a WordPress plugin that was as good at compressing images. Actually Jetpack CDN was already compressing, but only down to 216 KB. Again, really impressive stuff from Cloudinary. I guess nobody is compressing the image that well…(…yet. I will probably build this WordPress plugin).

Cloudinary keeps files in a hot cache state for weeks (much longer than CloudFront which degrades the file in less than 30 minutes).

Cloudinary also boasts video CDN delivery. I have not tested using videos so I can’t make an evaluation here. Although…it’s likely the video CDN performs as well as the image CDN, although we still need to test it.

Even in the scenario that CloudFront is delivery the same compressed file as Cloudinary, I’m seeing Cloudinary (i.e. Fastly) is performing better on cold, hot, and warm cache.

12% faster on hot cache
2x faster on warm cache
Cloudinary’s cold cache is a couple weeks, not 30 minutes.

So…Cloudinary has been extremely impressive indeed.

Reasons not to use Cloudinary

However, the free tier is extremely limited to only 25 GB/month of data transfer, while the premium tier is very expensive (starting at $99/month). Cloudinary is by far the most expensive CDN I’ve tested so far. To put the $99/month in perspective: CloudFront only charges $0.08/GB, and Cloudinary has a free tier of 25 GB and premium tier of 225 GB. Run the numbers: 1) for the free tier, CloudFront would cost only 25 GB * $0.08/GB = $2.00. 2) for the premium tier, it’s 225 GB * 0.08/GB = $18. And…Cloudinary’s charging $99/month? Wow.

On the plus side, if you’ve got a low or medium traffic website and you’re sure you’re transferring less than 25 GB a month…by all means go for Cloudinary. Since you’re staying in the free tier.

Another limitation is you can’t host a static HTML page on Cloudinary like you can with CloudFront and CNAME Aliases. You can host static assets like CSS, JS, images, and videos…but not HTML. For example, this entire blog is hosted off CloudFront, even the initial HTML request. I can’t do that with Cloudinary…just offload the images/videos/CSS/JS, that’s it.

As an aside: there’s a good reason why Cloudinary is so expensive…it is built on top of Fastly CDN. Cloudinary’s primary value-add are image and video transformations. And Fastly is already fairly expensive (minimum $50/month).

Final Analysis: to use or not use Cloudinary?

In the final analysis, what you’re paying Cloudinary for are the “on-the-fly” image and video transformations. Cloudinary contains some bells and whistles that makes it a powerful image & video CDN. Is this worth $99/month to you? i.e. $1188/year.

Also remember Cloudinary is basically a reseller of the Fastly CDN. Cloudinary’s primary value-add is in the image and video transformations. Are you the kind of person or business that needs Cloudinary’s advanced features? It costs $99/month…

What kind of person or business are you?

Hobby blog: since you’re probably a low-traffic site, you could easily stay within Cloudinary’s free tier of 25 GB.
High traffic blog: in my opinion, you can easily compress the images yourself to get it as good as Cloudinary. Then for fast delivery, either use Fastly directly or use CloudFront. Since using Cloudinary would simply be using Fastly indirectly.
Your website earns $1000/month: well $99/month is 10% of your profits. Not really worth it.

Software engineer building complex app: Cloudinary has some complex image transformations like cartoonify, brightness, rotate. That’s neat. Using Cloudinary as a REST API could save you some time (and developer time is always the most expensive).

Python CDN Speed Test Script: I use this to automate the collection of metrics about CDNs.

APPENDIX: Wanna reproduce our test results? Here’s how to setup Cloudinary

Login or Sign up in Cloudinary.
Install and Activate Cloudinary Plugin in WordPress

CDN experiment shows Free CDN surprisingly faster than Amazon CloudFront in many cases (case study…

Productivity Hacks & Software Performance Analysis — Thu, 01 Jul 2021 06:20:24 +0000

(Note: I’m not affiliated with Jetpack CDN in anyway. And if you disagree with my analysis in any way, please cordially do so 😊. I’m on twitter at @SpeedTestDemon)

Ohh…so premium CDNs are NOT always faster than free CDNs. While testing many free CDNs and whether they performed better than a premium CDN like CloudFront, I discovered many free CDNs performing faster than Amazon CloudFront. One was Cloudinary, the other is WordPress’s Jetpack CDN. Cloudflare is starting to look good too. In this article, I go over my CDN experiment showing how Jetpack CDN was faster than Amazon CloudFront in nearly every case except for one.

Before I present the simplified summary, here are some links relevant to this article.

My testing approach to CDNs
Python script for measuring CDNs
On twitter, I tweet analysis about software performance or email list

As a very simplified summary , based only on hot cache results

Jetpack CDN was faster than CloudFront by 25% (0.332 vs 0.245 seconds, or difference of 87 milliseconds)
Even after controlling for image size, CloudFront was only faster by 6% (0.199 vs 0.214 seconds, or difference of only 15 milliseconds). Basically within margin of error.

Jetpack CDN had better performance for several reasons:

Nice trick of compressing images 2–3x smaller (~700 KB image to ~236 KB).
Much more common DNS (i0.wp.com vs d20zaq59cm4c4j.cloudfront.net), which allows for significantly faster DNS lookup. In one case, 0.11 vs 0.008 seconds.
CloudFront typically degrades a file from hot cache to warm cache very rapidly, whereas Jetpack CDN doesn’t do this until at least several days.

What follows is a much more detailed technical analysis.

I had to do 2 speed tests. The 1st speed test was not apples-to-apples because Jetpack CDN surprisingly compressed the image, leading to a crushing result. The 2nd speed test, I controlled for file size, and CloudFront came out ahead very slightly by 6%.

In the first speed test, Jetpack CDN surprisingly crushed CloudFront.

I was shocked. I investigated and found out Jetpack CDN magically compressed the ~700 KB image down to ~219 KB. Well…that explains a lot.

I had to do a second speed test making sure CloudFront was delivering the same image as Jetpack CDN. CloudFront did a lot better in the hot cache. Still sucked at the other two metrics.

There were several reasons for Jetpack CDN performing so well:

Now one moment…you do “get what you pay for”…while this free Jetpack CDN has great performance, it is limited in many other ways. You can’t host videos. You can’t host a whole website off Jetpack CDN (while this blog itself is entirely CloudFront). You can’t update a file on Jetpack CDN easily, unless you wholesale rename the file (very annoying). The amazing performance applies mainly to images. JS and CSS won’t have the same amazing image compression, although you will see Jetpack CDN perform on par with CloudFront. See the conclusion for my thoughts on the Jetpack CDN vs CloudFront decision.

It should be noted that not every free CDN performs as well as Jetpack CDN. I found 2 free CDNs that performed really poorly.

By the way, I’ve been blogging a series of experiments on free CDNs and comparing them against CloudFront. You can get updates by following me on twitter SpeedTestDemon, or via email list.

Why is it interesting to test Jetpack CDN (previously known as Photon CDN)?

Because it’s an extremely popular CDN with millions of users. It was able to get so many users due to it being a first-class feature in WordPress’s Jetpack, which itself has 5 million users. If you’ve got Jetpack, it is basically 1 click to enable Jetpack CDN, and all your blog’s images get served from the Jetpack CDN.

And not to mention free! Jetpack CDN is a free CDN. So for that reason, it’s popular.

It used to be called Photon CDN before they did a name change.

But how “good” is free? Is it as good as a CDN you have to pay for, like Amazon’s CloudFront?

Why use CloudFront as the gold standard in CDN speed tests?

Well Amazon AWS is in general the gold standard for anything in the cloud. Amazon AWS has the largest market share for cloud providers (32% with Azure at 19%). They were the first to market with cloud services, and they’ve had many years to develop CloudFront into a top-of-the-line service.

AWS CloudFront has a generous free tier as well, along with only $0.08 / GB network-out. If you’re a normal website, you might incur a total aggregate cost of $0.50/month (which is nothing).

Brief Summary Of The Metrics Being Measured

It’s not as simple as running a curl on the same file from the two CDNs. Most simplistically: CDNs are just a caching layer. And there are 3 states that a cache can be in: cold cache, hot cache, and warm cache. We are going to measure all 3 (because we have it automated so why not).

If you are interested, you can read this much more detailed article about my CDN speed testing methodology.

If you’re interested in trying to replicate our results, see the Appendix where we discuss the exact steps we took.

I didn’t expect any of this when I first started. There is actually quite a bit of nuance when analyzing the curl measurements between Jetpack CDN and CloudFront.

Finding #1: Jetpack CDN’s DNS lookup is significantly faster than CloudFront’s DNS lookup.

To those new to measuring request latency, DNS is actually a VERY important detail that can seem minor. While DNS is normally very fast because it’s usually cached…if your DNS lookup is on a low-trafficked website, it’s highly unlikely to be cached in most routers.

This analysis was done off of speed test #2’s warm cache test.

I can only speculate why CloudFront took a whole 0.111 seconds compared to Jetpack CDN’s almost imperceptible 0.005 seconds. Compare these two DNS lookups: CloudFront “d20zaq59cm4c4j.cloudfront.net” and Jetpack CDN “i0.wp.com”. Jetpack CDN’s i0.wp.com looks like a shared domain that many other people are also using (there’s millions of JetPack users). Which means that it’s likely that nearby routers will have that DNS lookup cached. Unlike Jetpack CDN, CloudFront’s domain is highly unique and customized to this test instance. It’s highly unlikely nearby routers will have the DNS lookup cached, resulting in a significantly longer DNS lookup.

As you can see, it’s fairly interesting that there are indeed cases where DNS lookups can take a very significant part of the time (0.111 seconds out of 0.445 seconds is nearly 25%!). So don’t discount DNS lookups in your measurement times.

Finding #2: Generally, the Data Transfer, TCP handshake and SSL handshake times were the same between CloudFront and Jetpack CDN

In general, I found the “generic metrics” speeds were usually almost the same between CloudFront and Jetpack CDN. Probably because these operations “Data Transfer”, “SSL handshake”, “TCP handshake” are the same for every file regardless of file size or file types, no matter if you’re requesting a 10 GB video or just a 10 KB HTML file. I have an essay explaining what each of these metrics mean if you’re interested in learning more: guide to curl metrics.

The only time the “Data Transfer” was different was when Jetpack CDN compressed the image to a 2x lower size than CloudFront (see Finding #3).

Finding #3: Jetpack CDN had a surprisingly strong image compression algorithm

I found that uploading a 703 KB image was compressed to 219 KB on WordPress’s image CDN, while CloudFront compressed it to only around 300–400 KB.

The bottom line is Jetpack CDN is able to have 2x to 3x faster Data Transfer times than CloudFront. Remember Data Transfer time is specifically regarding how long the download takes (once TCP, SSL, and server is prepared to start sending bytes).

This is quite impressive. You’ll see Finding #3 referenced several times during analysis of Speed Test #1 when I didn’t account for this factor.

Finding #4: Jetpack CDN’s warm cache times were significantly stronger than CloudFront’s warm cache.

This analysis was done off of speed test #2’s warm cache test.

Impressively, Jetpack CDN’s warm cache time was nearly the same as its hot cache time. This indicates that Jetpack CDN keeps the image in RAM for quite a long time. That’s good! For the user, that is, because it’s a faster response time. This is going to be more expensive for Jetpack CDN because RAM is the most expensive hardware resource by far (way more than SSD).

CloudFront on the other hand…was nearly 2x slower than its hot cache time. Taking a closer look at where exactly CloudFront lost so much time, it was in two metrics: “Request Latency” and “DNS lookup”. With regards to “Request Latency” (0.135 seconds compared to 0.020 seconds), it seems clear that CloudFront degrades its hot cache rather quickly, where the image file probably got shunted over to NVMe or SSD. Well Amazon has a reputation among software engineers for being cheap ;), we really should not be surprised they are trying to save on hardware costs where they think they can get away with it.

CloudFront beats out Jetpack CDN when it comes to the cold cache test. However for the 2nd download, Jetpack CDN is slightly faster (5%) than CloudFront.

When we did the measurement again a couple other times, Jetpack CDN was beating out CloudFront handily on cached downloads. Sometimes it was 2x, sometimes 3x.

Cold cache measurement and analysis (speed test #1)

CloudFront is 10% slower than Jetpack CDN for a new file.

Note: I am aware there is no “Data Transfer” row. I had a mistake with this test run where I accidentally made curl not download the file. CloudFront’s Data Transfer is probably going to be slower than Jetpack CDN’s Data Transfer (see later sections). I don’t need to belabor this point.

Yeah, CloudFront is 8% slower than Jetpack CDN for cold caches. I’m not really entirely certain why. The main deltas were in DNS lookup and SSL handshake.

The DNS lookup delta (0.048 vs 0.002 seconds) might surprise some people who think DNS lookups are normally always very fast. No they are not always fast. The big exceptions are low-trafficked DNS names ( see more analysis in “finding #1”)

I’m really not certain why CloudFront took so much longer in SSL handshake. It’s kind of out of left-field, and perhaps is just randomness.

Request Latency was the same for both CloudFront and Jetpack CDN. That’s good news because I would expect both CDNs to take the same amount of time for a file that’s not in cache at all. For some reason, re-running the cold cache test in “speed test #2” showed CloudFront much slower than Jetpack CDN in “Request Latency”.

Hot cache measurement and analysis (speed test #1)

Jetpack CDN is 25% faster than CloudFront! (0.332 vs 0.256 seconds)

(Reminder: these times are an average of 10 curl calls).

Where did Jetpack CDN save so much time?

We can see CloudFront is neck-and-neck with Jetpack CDN for most of the metrics except for “Data Transfer” which is the download time. This makes sense as CloudFront is trying to send a 703 KB file while Jetpack CDN is only around ~250 KB. See analysis in section “Finding #3”.

And Data Transfer was literally the only difference.

What’s even more impressive is you could argue the “hot cache” times is one of the most important metrics for a CDN. If your site has a moderate amount of traffic, chances are pretty good the CDN will be in a hot cache state.

So Jetpack CDN’s hot cache time is 25% better than CloudFront. Very impressive work from Automattic, the company behind WordPress!

To even out the odds between Jetpack CDN and CloudFront, we will be conducting a second speed test. You can jump down to the second speed test section later on in the article.

Warm cache measurement (speed test #1)

Jetpack CDN is nearly 2x faster than CloudFront.

Reminder: the warm cache is a measurement time taken 30 minutes after the hot cache time.

You should also see the 2nd speed test’s warm cache analysis, which differs slightly from the 1st speed test’s warm cache analysis.

TCP handshake, SSL handshake were the same. See analysis in section “Finding #2”.

Request Latency was also nearly the same. This differs from the 2nd speed test where I’m seeing CloudFront’s Request Latency spike for some reason. At any rate, it makes sense Request Latency should be the same as both CDNs probably have the file in RAM and that’s pretty much the best you can do, no matter who you are.

The big difference in time went to “Data Transfer” where Jetpack CDN was…nearly 3x faster than CloudFront? 0.089 seconds vs 0.228 seconds. Already touched on this earlier. Jetpack CDN did a great job of compressing the image down to around 219 KB while CloudFront is still trying to transfer ~700 KB image. See analysis in section “Finding #3”.

Speed Test #1 conclusion and analysis

Jetpack CDN beat out CloudFront in hot cache, cold cache, and warm cache, due mainly to the image automatically compressed to 219 KB and CloudFront still trying to transfer a 720 KB PNG.

What if we even the odds and make CloudFront and Jetpack CDN transfer the same file? Surely CloudFront should beat out Jetpack CDN now? It’s not that simple. See the next section for Speed Test #2.

In brief: CloudFront’s Data Transfer metric improved to be on par with Jetpack CDN. This was expected because CloudFront and Jetpack CDN are now transferring the same file (previously, was transferring ~700 KB compared to ~200 KB image).

CloudFront’s Hot Cache times also improved from -25% to +7% speed over Jetpack CDN (due to improved Data Transfer).

However, CloudFront’s Cold Cache and Warm Cache still lags behind Jetpack CDN. Impressively, Jetpack CDN is still 2x faster in warm cache time, due to DNS lookup and Request Latency.

Cold cache measurement (speed test #2)

Seems Cloudfront is still slower than Jetpack CDN when downloading a new file.

Note: I am aware there is no “Data Transfer” row. I had a mistake with this test run where I accidentally made curl not download the file. In my opinion, not worth re-running the test since I don’t expect Data Transfer numbers to be extremely different.

It’s interesting…I had run the cold cache test a few times and CloudFront is still slower than Jetpack CDN for new files. Even after ensuring the images downloaded the same size. That’s unfortunate.

The main reason for the large time discrepancy was in Request Latency — all the other metrics were close enough (except for DNS lookup — see finding #2 section). I honestly have no clue why CloudFront should take much longer (about 50%) than Jetpack CDN for fetching the image from the origin server — the images are the same size in both cases.

Hot cache measurement and analysis (speed test #2)

CloudFront and Jetpack CDN neck-to-neck in almost all the individual metrics.

(Reminder: these times are an average of 10 curl calls). CloudFront beats out Jetpack CDN by 7%. Oh good, so the premium CDN is finally outperforming the free CDN. But in absolute difference, only about 16 milliseconds.

Every single metric here shows CloudFront and Jetpack CDN spending approximately the same time. Some metrics CloudFront was faster, and vice versa for some other metrics. CloudFront just ended up edging out overall. The relative difference was only 7%, not much of a big deal. And possibly within margin of error.

See “ Finding #2 “ explaining why “Data Transfer”, “TCP handshake”, and “SSL handshake” times were similar between CloudFront and Jetpack CDN.

Warm cache measurement and analysis (speed test #2)

Reminder: the warm cache is a measurement time taken 30 minutes after the hot cache time.

This was an extremely interesting test. Jetpack CDN is 2x faster than CloudFront. That’s not a small difference. How? Where did the time go?

The first interesting bit was that Jetpack CDN DNS lookup was way faster than CloudFront. In fact, CloudFront’s 0.111 seconds is nearly 25% of the overall time of 0.445 seconds. There’s a good reason why Jetpack CDN’s DNS lookup was way faster: see analysis in “Finding #1”.

Secondly, CloudFront’s Request Latency was way longer at 0.135 seconds compared to 0.020 seconds. I write about this in the analysis for “Finding #4”.

The rest of the metrics did not make much difference in absolute time. Yes CloudFront was “2x faster” TCP handshake (0.025 vs 0.052 seconds) but the absolute difference is only 0.027 seconds. The bigger rocks were in DNS lookup and Request Latency.

TL;DR — Jetpack CDN holds files in hot cache for longer than CloudFront. Evidence is Jetpack CDN’s warm cache is nearly the same as its hot cache time. CloudFront degraded about 2x due to slow non-cached DNS lookup and likely non-cached at the point of presence.

My initial stance going into this was CloudFront was going to be the outright winner (even after accounting for cost). As it turns out, Jetpack CDN has acquitted itself remarkably well. Based purely on the performance benchmark, I’ve come to the conclusion that Jetpack CDN is better than CloudFront (and yes, Jetpack CDN is free).

There two main scenarios for evaluation:

Jetpack CDN compresses image better (i.e. smaller) than Cloudfront,
Cloudfront and Jetpack CDN deliver the same image.

In scenario 1: Jetpack CDN is significantly faster than CloudFront for cold, hot, and warm cache. Even for the hot cache case, Jetpack CDN was 25% faster.

In scenario 2: Jetpack CDN maintains a significant edge over CloudFront in cold and warm cache times (50–100%), while CloudFront has a slight edge in hot cache times (6%).

Summary of Jetpack CDN’s superiority over CloudFront:

image_downsizing feature works well. It shrunk a 721 KB image to 225 KB. This basically doubles or triples Jetpack CDN’s download speed. This is a big advantage.
Jetpack CDN keeps images in hot cache for significantly longer than CloudFront. At one point I even tested Jetpack CDN a full day after the hot cache test, and it was still nearly as quick as hot cache. I know it’s still cached because I’m getting back this header “x-nc: HIT dfw 6” from Jetpack CDN. The warm cache is some fairly impressive stuff from Jetpack CDN.

Extraordinarily simple to set up on a wordpress instance. Basically 1-click install from the Jetpack plugin (which itself has millions of installs).

If you’ve ever used CloudFront, you’d know it is very confusing. Take a look at this screenshot, it’s dizzying. Even for a software engineer like me, I had a moment of confusion and minor agony when staring down the options (I thought: Why isn’t it simpler?).

There are good reasons to use another CDN besides Jetpack CDN despite its great performance, cost, and ease of use. The main thing is that Jetpack CDN is fairly limited. For one thing, Jetpack CDN can only do images, JS, and CSS…yeah that’s fairly limited lol. If your website is totally static, you should put HTML on Jetpack CDN too. And also videos and audio if you’ve got them.

Here’s a full list of (important) Jetpack CDN limitations. There are other minor limitations but I don’t deem them important.

Doesn’t serve video or audio
Doesn’t serve HTML.
No cache invalidations — this means it’s much more annoying to update an image. You need to rename the image in order to force CDN to load the new image.
Limited to WordPress instances. Need Jetpack installed.

So at the time of writing I would recommend Jetpack CDN if: 1) you’re using wordpress, 2) for images only. You still need to get a CDN like CloudFront to handle all the other stuff like HTML, video, and audio. Things may change in the future that may change my mind, such as Jetpack CDN’s warm cache times.

In general though I would recommend getting a premium CDN. See my analysis here on why a premium CDN is worth it.

This section is specifically for Jetpack CDN and CloudFront. Here is the article about my general testing methodology.

We used a python script that used cURL. We used cURL because of its ability to be flexible and complete complex tasks. You can use curl for things like user authentication, HTTP post, SSL connections, proxy support, and FTP uploads. You can also do simple things with curl, such as download web pages and web images, which we used cURL for in this scenario.

Tests are conducted on a Macbook Air 13" 2020 using macOS Catalina v. 10.15.7. This uses an Dual-Core Intel Core i3 processor clocked at 1.1 GHz alongside 8 GB 3733 MHz LPDDR4X RAM and an SSD.

Creating python file “test.py”

Clone this repository. Free speed test! git clone https://github.com/speedtestdemon/speed-tests.git

cd speed-tests

Install pycurl pip3 install pycurl

If you encountered an error, try first sudo apt install libcurl4-openssl-dev libssl-dev
Then run pip3 install pycurl again

Now that installation is complete, try to run speed test

python3 test.py "https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024x576.png"

Setup and Testing — Cloudfront

Create a distribution in CloudFront

Create a behavior with a Path pattern “/wp-content/”

Install CDN Enabler plugin on WordPress
Update the Settings of CDN Enabler, input the Domain Name provided by CloudFront from the setup above

Add new Post in wordpress, upload the image provided above
View Post and right click on the Image, Right click and Inspect

Copy the link from the Image Source

Download the image using the command, open Terminal, go to the same folder as test.py and paste the command below

8.a For the different size but same image — due to downsizing capabilities of each CDN

python3 test.py “https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024×576.png"

8.b For the same size, same image

python3 test.py “http://d20zaq59cm4c4j.cloudfront.net/wp-content/uploads/2021/05/shoeb-jetpack.png"

Setup and Testing — Jetpack CDN

1 . Install Jetpack plugin on WordPress site

Go to Settings of Jetpack, enable the Site Accelerator feature under Performance tab

Add new Post in wordpress, upload the image above
View Post and right click on the Image, Right click and Inspect

Copy the link from the Image Source

Go to the same folder as test.py and download the image using cURL with the command below python3 test.py “ https://i0.wp.com/comparecdns.com/wp-content/uploads/2021/05/shoeb.png?resize=1024%2C576&ssl=1 "

Related Essays

That’s the end of the Appendix. Here are related essays:

Is a premium CDN like CloudFront worth it, instead of a free CDN?

Cheat Sheet on Curl Performance Metrics: how to benchmark server latency with curl

CDN Speed Tests (Testing Methodology)

Python CDN Speed Test Script: I use this to automate the collection of metrics about CDNs.

Is a premium CDN like CloudFront worth it, instead of a free CDN? — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Thu, 01 Jul 2021 05:54:45 +0000

Short answer : yes you should use a premium CDN, because 1) premium CDNs are easily 3 times faster than free CDNs — 1+ seconds vs 0.3 seconds, a huge difference for traffic as valuable as ecommerce, 2) it doesn’t even cost you that much. Extensive cost analysis later in this essay. It’s only a few dollars unless you’re running a very large traffic website.

I’m on Twitter @SpeedTestDemon and email list is here.

EDIT: I just read on this random tech website (techradar) “no doubt prices can be high”. Ugh it’s just trash content. It’s written by someone who just wants to fill up word count. It’s not written by someone who’s actually operated a website. Come on and actually do the math. If you’re the typical small-time website, you know CDNs cost you close to nothing. If you actually have a lot of traffic, it might end up getting to $100/month — hardly more expensive than whatever you’re already paying for hosting. Apologies for the mini-rant.

EDIT 2: just looked up techradar’s monthly traffic and holy crap they have 52 million visitors a month for writing that kind of trash content? Unbelievable. Makes me feel like ranting. Why do people put up with trash content. I will write a rant about this later.

Longer answer : rest of this essay.

The main focus of the rest of this essay is going to be discussing the pricing advantage of free CDNs over premium CDNs.

Regarding the other dimension of performance — well I’m already going to be discussing that in 20+ articles analyzing 20+ free CDNs. A brief summary thus far: Premium CDNs are obviously way faster than free CDNs — easily 3 times better. It’s not even close. And it makes sense, right? A fast CDN like CloudFront is hugely expensive to maintain high performance across terabytes of data, thousands of servers. This is really hard. Imagine debugging and maintaining thousands of computers. And then making sure the servers have high availability. Really hard.

So the only advantage that free CDNs have over premium CDNs is pricing. It’s true: a premium CDN will cost you money. But how much is it really? Let’s explore the cost question for the rest of this essay.

To find out how much a CDN will cost you, all we need to find out is how much data your website transfers. This is because the metric that CDNs charge on is the amount of data transferred by the CDN. Cloudfront, for example, is around USD $0.11 per GB (BunnyCDN is a cheaper alternative at $0.04 / GB. Not sure whether it’s actually faster, I will be speed testing Bunny CDN shortly!).

Let’s estimate how much data you’re going to be using if you’re in either one of these two buckets:

You already have a working website
You’re still building out your website

How to find how much a CDN would cost you if you already have a working website

If you’ve already got a website setup, there’s a simple test you can do. Go to your website. Open up Chrome Developer Console. Click the Network tab. Ctrl+R or Cmd+R in order to reload the website. Then you see a number where it says how much data got transferred over the network. This is how much data the CDN will charge you.

Here’s an example screenshot of where to find the data transferred in the Chrome Developer Console tab. Look for the red circle. This example is for facebook by the way.

Please note you can safely ignore the “X MB resources” number. That number is usually far larger than the “X MB transferred” number because “X transferred” is highly compressed. For example, a 1 MB photo is usually first compressed (probably gzip, although brotli compression recently released by Google is starting to gain adoption) before your browser downloads a 500 KB photo, which gets decompressed back to 1 MB. Web servers do this in order to speed up how long it takes for the web page to render for you.

OK once you’ve got the number, jump down to the section analyzing the cost of the premium CDNs.

How to find how much a CDN would cost you if you are planning to build a website

If you’re still planning out your website, we can still get a good estimate of how much data you might transfer by looking at a few representative examples.

The first best thing you could do is to do the “data transfer” test (written about in the previous section for a working website) on somebody else’s website already out there that is most similar to what you want to create.

Another good rule of thumb is to assume your average web page has a size of 2.07 MB. Pingdom has conducted a study on the average web page size. They have all this data since millions of people use Pingdom to test how long their website takes to load. They found the average page size is 2.07 MB.

Lastly, you could go through a bunch of case studies below where we analyze how much data was transferred for each example. You can pick out the example that represents your goal most closely and use that number to calculate your estimated CDN cost.

Note: case studies are conducted with an adblocker, which means data transferred could be much smaller than without an adblocker.

Case Study: How much data does a Shopify store use?

Example: https://seloolive.com/ (they have awesome olive oil, you should buy from them)

Data transferred: 1.2 MB

Case Study: How much data does a minimal wordpress website use?

Example: https://speedtestdemon.com/ (my own website, haha, which runs on WordPress)

Data transferred: 203 KB (wow that’s small!)

Case Study: How much data does a complicated custom ecommerce solution on WooCommerce use?

Example: the product page of an ecommerce store doing 7 figures in revenue per year.

Link: https://www.solioswatches.com/product/solar-black-black-mesh?caseColor=matte-black&dialSize=36mm&gender=men

Data transferred: 4.6 MB

Case Study: How much data does a simple sports betting web app use?

Example: this is a simple web app that does optimizing.

Link: https://www.topflightfantasysports.com/pga/pga-fd-optimizer/

Data transferred: 4.4 MB

Case Study: How much data does a big complicated web app like Facebook use?

Example: facebook.com

Data transferred: 3.8 MB

Case Study: How much data does a simple landing page use?

Example: https://sizzy.co/

Data transferred: 516 KB

Case Study: How much data does a portfolio website use?

Example: http://www.baileyritchiephoto.com/

Data transferred: 1.8 MB

Case Study: How much data does a magazine website use?

Example 1: typical techcrunch.com article

Link: https://techcrunch.com/2018/09/11/comma-ais-george-hotz-ousts-george-hotz-as-ceo/

Data transferred: 2.4 MB

Example 2: oil price news site

Link: https://oilprice.com/Geopolitics/Middle-East/Could-Energy-Ties-Bring-Saudi-Arabias-Conflict-With-Iran-To-An-End.html

Data transferred: 650 KB (oh interesting, MUCH smaller than techcrunch)

Example 3: bitcoin news site (because who doesn’t love bitcoin??)

Link: https://news.bitcoin.com/latin-american-countries-bitcoin-adoption-el-salvador-bitcoin-law/

Data transferred: 3.6 MB

So how much does a premium CDN really cost you?

Once you’ve got the number of how much data you’re transferring per pageview, now find how many pageviews you’re getting per month (or per day, your preference). The math is just a simple multiply: (number of pageviews * amount of data per pageview) = total data per month (or day).

OK now what is the cost of transferring that data? In the case of Cloudfront, you can just use the highend number of $0.11 / GB (they vary their pricing per region).

If you rather use other premium CDNs, here’s the full list of the most popular ones (it’s a crowded market, there’s probably many premium CDNs I missed. Drop me a comment on twitter or email if you want me to include your favorite CDN).

CloudFront: $0.11 / GB
Cloudflare (the premium version, not their free tier which also has bad performance like every other free CDN): cheapest premium plan is $20/month.
Fastly: minimum charge $50/month.
BunnyCDN: $0.01 / GB for North America & Europe, $0.06 / GB for Africa.
KeyCDN: minimum charge $0.04/GB to $0.11 depending on your location.
StackPath (MaxCDN): Starting from $15
Akamai: flexible pricing model starting from $0.087 / GB
Microsoft Azure CDN: Starting from $0.081 to $0.233 depending on location.
CDN77: starting at $199/month ($0.033 / GB).
Leaseweb: minimum of $49/month ($0.06 / GB).

Have you finished calculating how much that premium CDN costs you? If you’re a small-time website, you should probably go with CloudFront (or KeyCDN or Azure) which allows you to pay a much smaller upfront cost than some of these other premium CDNs like Fastly. $50/month is ridiculous to pay for a CDN if you’ve got barely a few thousand traffic.

Let’s say you’re a website doing 10,000 pageviews/month, which is an amount most websites don’t even get to. The math is 10,000 pageviews/month * 2 MB/pageview = 20 GB/month of data. Then 20 GB/month * $0.11 /GB = $2.20 / month. Basically a pittance. Chances are pretty good you’re already paying $5/month for hosting along with many other widgets and plugins.

In that example we just quickly analyzed, you’re getting 3 times faster page loading times for 10,000 people for just $2.20/month. It’s an incredible deal.

Are there exceptions where a Free CDN is as good as a premium CDN?

Actually, yes. I’m currently doing an investigation on free CDNs and I’ve been impressed with some Free CDNs so far.

The best one I’ve seen so far is Jetpack CDN (a.k.a. Photon CDN), where Jetpack’s image CDN performed very admirably (as well as CloudFront) while also having a great image compression algorithm. However, very limited to images, when you probably would also like to put HTML, JS, CSS, video, and audio on the CDN.

Another good one is Cloudinary but they have a fairly limited free tier of 25 GB in a month, after which you need to pay $99/month. I’m currently writing an article on Cloudinary right now.

So the answer is yes, premium CDNs are worth it

In my humble opinion, the premium CDNs don’t cost that much. The only problem is if you’re the kind of person who has trouble understanding how to set up a CDN. Am I wrong? Feel free to argue with me on Twitter or in email (hello at speedtestdemon dot com). I’m willing to change my mind if your argument’s good.

Cheat Sheet on Curl Performance Metrics: how to benchmark server latency with curl — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Thu, 01 Jul 2021 05:48:08 +0000

Cheat Sheet on Curl Performance Metrics: how to benchmark server latency with curl — Speed Test Demon

I wrote this cheatsheet mainly because I was mildly frustrated googling for 10+ minutes trying to figure out what in the world does time_pretransfer mean??? Try googling it. I'll wait. It's abstruse! Ugh!

The top Google result for “ curl time_pretransfer meaning" is this StackOverflow post, where they quote the curl man page and are still very confused (as was I): What is the meaning of time_pretransfer, I checked man page of curl, it says "it took from the start until the file transfer is just about to begin"，I don't understand very well

That’s right-even if you “RTFM” (Read the Fine Manual), the curl man page itself is totally unhelpful about the meaning of time_pretransfer.

You’d think somebody would’ve written a good article on the curl metrics by now!

I finally figured out what time_pretransfer meant...after using my brrrr-rrrrain and the speed of light (0.186 miles/microsecond). But nobody had the right answer for it. And I do. Hence this article.

While here’s the full cheatsheet, there is also an explainer section I wrote on each of the metrics. If you’re interested in learning how I solve the meaning of time_pretransfer, jump down to the section " What does time_pretransfer mean?"

(by the way, you can follow me on twitter, Analyzing Software Performance for new posts like these, or email subscribe)

time_namelookup : how long DNS lookup took (translating domain name to IP address)
time_connect : how long TCP handshake took (done by both HTTP and HTTPS)
time_appconnect : how long SSL handshake took (only for HTTPS)
time_redirect : how long the redirect took. It is 0 if there is no redirect. Need "-L" flag in curl to redirect.
time_pretransfer : This is essentially an alias for either time_appconnect or time_connect (depending on HTTP or HTTPS). It is only useful as a delineator of when the specific request to the server has begun.
time_starttransfer : This is when the server is ready to deliver bytes. Same as TTFB (Time To First Byte). It includes time_pretransfer, just subtract time_pretransfer from the time_starttransfer to get the amount of time spent in this phase
: The total time of the entire curl call. Calculate time_total - time_starttransfer to get the data transfer (a.k.a. download) time.

These are also all the available metrics to be printed out, according to curl’s Linux man page.

I also have a section for each metric with more in-depth analysis. See sections with title “What does…”.

Curl speed test example

Here is a typical curl speed test output. Important to note: it is to an HTTPS URL. That factor is relevant to time_appconnect and time_pretransfer.

Get the curl-format.txt from github and then run this curl command in order to get the output

$ curl -L -w "@curl-format.txt" -o tmp -s $YOUR_URL

time_namelookup : 0.001654s

time_connect : 0.054957s

time_appconnect : 0.156267s

time_pretransfer : 0.156494s

time_redirect : 0.000000s

time_starttransfer : 0.219013s

All time numbers are cumulative since the start of the curl. They are all in seconds.

Couple important notes about the curl command:

The -o tmp is actually really important in order to actually download the file. If you tried to be smart and pass -o /dev/null , what curl does is it skips the download of the file because it interprets /dev/null to mean you're not download the file. And it would mean you get a download time of nearly 0 seconds (the delta between time_total and time_starttransfer.
The -L flag is actually important in order to make curl follow redirections.

What does curl’s time_namelookup mean?

Thankfully this is an easy one to analyze. It is just DNS lookup time. The end result of this is just an IP address of the destination server that the curl program should talk to. No connection to the destination server has been made yet.

Note that this metric is actually important to factor in a speed test! You might be surprised if you think this is something as banal as DNS lookup.

Usually the DNS lookup goes by extremely quickly because the computer has probably cached the result. The computer caches because the results rarely change. For example, if there’s an A record for a domain that resolves to an IP, the computer can easily cache it — it’s maximum 32 bytes for the whole entry (and most computers have gigabytes of memory).

However, the DNS lookup time can take a while if the computer hasn’t cached it, especially if the destination website doesn’t have a lot of traffic, because it means the routers in-between doesn’t have the DNS lookup cached so they can’t return the result quickly. I’ve had one speed test where it was as much as 56% of the total download time (0.122108 seconds / 0.217073 seconds). Actually, wow, that’s an incredibly high percentage. I’m surprised it got that high. That was on a newbie website of mine, so it was a very low traffic website. The next DNS lookup took only 0.001408s seconds. It definitely got cached.

It is crucially important for CDN speed tests that the DNS operation is performed by curl. DNS lookups are how the CDN finds the IP address of the server closest to you. So for the same domain path, the IP address is going to be wildly different depending on your location. Although there is a way to exclude DNS lookup from the curl operation by giving curl an IP address as part of the input, you shouldn’t do it. DNS lookups are very important to how CDNs function!

That was rather interesting to write about. I am probably going to be doing DNS performance testing at some point, although this seems to be much harder to test than just plain CDN performance tests.

What does curl’s time_connect mean?

Time_connect is how long it takes to establish a TCP connection to the destination server. In the previous step, the DNS lookup, all we got back was an IP address. Now, at this step, there are actual sockets created, and data is ready to transfer between the client and server.

Note that this procedure is done for every request, for both HTTP and HTTPS and no matter which server you’re getting your request from (Google, Facebook, Microsoft, or any random guy’s WordPress server).

What does curl’s time_appconnect mean?

This is how long the SSL handshake takes, only measured for HTTPS requests. If you make a curl to an HTTP URL, curl will report a 0.000… for time_appconnect.

Note that this procedure is done for every HTTPS request, regardless of purpose or which server is responding, even if it’s Google.

Also, no idea why it’s called time_appconnect. That seems like a misnomer. Maybe better is “time_sslconnect”.

What does curl’s time_pretransfer mean?

time_pretransfer is essentially equal to either time_appconnect (HTTPS) or time_connect (HTTP) - depending on whether the request is HTTPS or HTTP. That's all it is. Nothing happens between time_pretransfer and the previous stage.

This was the one that really confused me. I was having a hard time understanding, until I realized something…the time lapse from the previous stage is far too short for there to be any network activity.

Why am I so confident that there is only CPU involved? Because the time lapse between time_pretransfer and time_appconnect is so small, even if data was traveling in a straight line between two points at the speed of light, there could not possibly be round-trip network transfer.

Here’s the math breakdown. First note that in the example curl output, the time difference between time_pretransfer and time_appconnect is only 227 microseconds. If you try it yourself, it could even be as low as 27 microseconds (as seen on a curl output on a curl’s github issue).

Second thing to note, what is the speed of light per microsecond? The speed of light is 186,000 miles per . Converting that to microseconds….there’s 1 million microseconds in a second, so 186,000 miles/sec * 1 sec / 1,000,000 microseconds = 0.186 miles per microsecond.

Finally, multiply the numbers we get 227 microseconds * 0.186 miles/microsecond = 42 miles.

Wait, 42 miles??

And that time is roundtrip, so one-way is only 21 miles.

Yeah…I seriously doubt there’s a server or datacenter that close to where I live. I’m in a major US city but not the usual suspects SF, NYC, LA, Chicago or Dallas.

What is probably going on here is curl is getting itself ready to receive bytes — and that’s it. There can’t possibly be any bytes exchanged as part of the time between time_pretransfer and time_appconnect.

So, time_pretransfer is really just the same as either time_connect or time_appconnect depending on whether the call was HTTP or HTTPS. As such, it seems the only purpose of time_pretransfer is to make it easier to clearly analyze how long it took for the server to begin transferring bytes. This is calculated by “time_starttransfer — time_pretransfer”….which brings us to the next metric, “time_starttransfer”.

What does curl’s time_starttransfer mean?

The time duration “time_starttransfer — time_pretransfer” is basically latency. It specifically measures how quickly the server can respond to this particular curl request. If you’re a backend engineer, this is the one metric you’ll pay a lot of attention to. All the other metrics are just normal TCP or SSL stuff for every server, and as a backend engineer, there’s not a lot you can do to affect the performance of those aside from renting more expensive servers.

Time_starttransfer is essentially the same as Time To First Byte (TTFB), which means how long it took for the server to prepare the response. If you’re a backend engineer, you’ll be familiar with that term too. Look, it’s just latency OK. And very important to get latency down.

OK lets walk through an example of what time_starttransfer metric would mean for CDNs. If you were requesting a file from a CDN, then this phase is how long it took for the CDN to prepare that file for you. Either the CDN has cached the file in memory so it can return it quickly to you. Or it has cached it in the SSD layer so it needs a little more time. Or maybe it doesn’t have it cached at all, so it has to make a HTTP/HTTPS request of its own to the origin server where the file is supposed to be located. So, this step is very crucial to analyze the difference between CDNs — everything else is going to be more or less the same.

One more example. Let’s say you were making an API request to Google Translate API to get back the French translation of an English sentence. Then this particular step (time_starttransfer — time_pretransfer) is how long it took for Google to translate. All previous steps are just normal TCP connection setup common to every other request.

Mkay? Time_starttransfer is just the latency of the server. We’re almost done going through every metric. Last two are time_redirect and time_total which are straightforward.

What does curl’s time_redirect mean?

If there was no redirect, this value is 0.000.

If there was a redirect, then you will see the amount of time that redirecting took.

Here’s a simple input to curl you can use to see how long a redirect might take: “google.com”, which gets redirected to www.google.com

For our purpose of speed testing CDNs, it’s important that time_redirect is 0 because redirecting is not really a relevant factor for a CDN’s download speed.

An important thing to note: if you’re interested in measuring redirect time, you want to make sure the redirect is happening at the right step. When I ran curl on google.com, time_redirect should've been on step 2 (right after time_namelookup). Here was the output I got:

curl -L -w "@curl-format.txt" -o /dev/null -s google.com

time_namelookup : 0.006038s

time_connect : 0.069728s

time_appconnect : 0.000000s

time_pretransfer : 0.069885s

time_redirect : 0.069052s

time_starttransfer : 0.161169s

Another important thing — you need -L flag to curl in order for it to follow redirects, or else it won't redirect. It's an easy mistake to make, because curl won't complain loudly it got a redirect but didn't follow. Your only clue is if 0.000 is in time_redirect but you expected it to be non-zero.

What does curl’s time_total mean?

Thankfully this one’s a really simple one to understand. This is just how long the entire download took — starting from the DNS lookup request all the way until the client has sent a FIN connection tear down since all the bytes have been downloaded.

By the way, if you are using curl to test CDN speeds, you should know that doing just one curl is by far not enough to measure a CDN’s delivery speed. The two metrics you should measure are “1st download speed” vs “cached downloads speed”, which means doing about 11 curl calls to the same URL. I have a more in-depth article explaining my testing methodology.

Final Thoughts

If you want a quick summary of each metric, refer to the cheat sheet at the top of the page.

If you have any questions, send me an email (hello /at/ speedtestdemon \dot/ com )or hit me up on Twitter (I love Twitter and I use it actively):

If you wanna get email notifications of new posts, you can subscribe

Cloudinary vs CloudFront: Cloudinary crushes Amazon Web Services!

Python CDN Speed Test Script: I use this to automate the collection of metrics about CDNs.

How To Speed Test a CDN (Testing Methodology) — Speed Test Demon

Productivity Hacks & Software Performance Analysis — Tue, 15 Jun 2021 05:02:59 +0000

How To Speed Test a CDN (Testing Methodology) — Speed Test Demon

Since I’m going to be writing a whole bunch of articles about CDN speeds, I wanna consolidate my testing methodology into a single article. I’m going to be linking back to this article a lot.

Questions to be answered in this article:

What numbers are measured for CDN download speed? (it is not just one number)
Why were those numbers selected?
How are the numbers measured?
Where is the measurement done? Cloud server or home computer?

I’m on Twitter @SpeedTestDemon and email list is here.

The three important measurements to make

At the end of the day, the CDN is really just a caching layer. All it is, is just a massively distributed caching layer. That’s it. That’s all it is.

And if you’re familiar at all with how caching layers work, there are 3 all-important metrics to measure:

Cold cache latency — item is not in cache at all
Hot cache latency — item is in fastest response layer of cache
Warm cache latency — item is at a slower response layer of cache.

To illustrate this in hardware terms: server responding from RAM is the fastest possible latency. That’s hot cache. Getting it from disk (SSD) is the second fastest possible latency. And you can add more layers (HDD). Finally, cold cache is if the server doesn’t have the item at all.

With regards to a CDN, how do we measure hot, cold, and warm cache?

Cold cache latency : is simple, you just fetch a file that the CDN has never seen before. This is going to take the longest time. On CloudFront for example, I’m seeing a drop from 1.2 seconds to 0.232 seconds in the hot cache.

Hot cache latency : also simple, you can fetch the same file 10 times in a row and take the average.

Warm cache latency : trickier. Much trickier. We must wait some amount of time to wait for the file to leave the hot cache. For now we are sticking with 30 minutes but we may adjust this as we learn how CDNs are moving items from hot cache to warm cache. The result of the warm cache is also much more ambiguous as it must be fetched at some time away from the cold and hot cache tests.

You can see the cold cache is much slower than hot cache by running this curl: curl -w “@curl-format.txt” -o tmp -s “ [https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024×576.png](https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024x576.png) “. This example uses CloudFront. There is a massive jump from 1+ seconds to 0.1–0.2 seconds.

How to measure the 3 measurements: my open sourced Python script (check github)

I used a python script to automate the measurements. This is important so that the measurements are very consistent between CDNs. Also important to remove human error.

It is open sourced here: https://github.com/speedtestdemon/speed-tests/blob/master/test.py

There is a massive drop in CDN download time from cold cache to hot cache. CloudFront gave me 1.09 seconds for cold cache, then 0.112 seconds for hot cache, then 0.226 for warm cache.

Here is the full python output. The cool thing is that it prints out the headers it got for the cold cache and for warm cache. The headers are used to sanity check that the test is doing the correct thing regarding cold cache and warm cache test.

------------------------------------------------------------------ Testing " **Cold cache speed**" ------------------------------------------------------------- Got headers: HTTP/2 200 **content-type** : image/png **content-length** : 719983 **date** : Sun, 20 Jun 2021 21:47:06 GMT **last-modified** : Mon, 07 Jun 2021 00:16:21 GMT **etag** : "52ae2ff2354d4a68e680b77b4da58985" **accept-ranges** : bytes **server** : AmazonS3 **x-cache** : Miss from cloudfront **via** : 1.1 c39432c353feb02b03735f3850e19107.cloudfront.net (CloudFront) **x-amz-cf-pop** : IAH50-C1 **x-amz-cf-id** : NgkCqcrwb3K65LeGu7uhebFNODrNI9s8wVeHZ93lq2XKrE3q9PMm-A== **time\_namelookup** : 0.06327399999999999691 **time\_connect** : 0.01762099999999999778 **time\_appconnect** : 0.05910300000000001663 **time\_pretransfer** : 0.00010199999999999099 **time\_redirect** : 0.00000000000000000000 **time\_starttransfer** : 0.95064099999999995827 **time to download** : 0.00008300000000005525 **time\_total** : 1.09082400000000001583 ------------------------------------------------------------- Testing " **Hot cache speed**" ------------------------------------------------------------- 10 requests done. Average: **time\_namelookup** : 0.00190460000000000000 **time\_connect** : 0.02373660000000000006 **time\_appconnect** : 0.06306589999999999419 **time\_pretransfer** : 0.00019590000000000024 **time\_redirect** : 0.00000000000000000000 **time\_starttransfer** : 0.02317680000000000434 **time to download** : 0.00007860000000000089 **time\_total** : 0.11215840000000001919 ------------------------------------------------------------- Testing " **Warm cache speed**" ------------------------------------------------------------- Sleeping for 0.5 hr to move cache from hot to warm Got headers: HTTP/2 200 **content-type** : image/png **content-length** : 719983 **last-modified** : Mon, 07 Jun 2021 00:16:21 GMT a **ccept-ranges** : bytes **server** : AmazonS3 **date** : Mon, 21 Jun 2021 00:09:46 GMT **etag** : "52ae2ff2354d4a68e680b77b4da58985" **x-cache** : Hit from cloudfront **via** : 1.1 9b59bfec44582f64d3d8dac9fb7d27b7.cloudfront.net (CloudFront) **x-amz-cf-pop** : DFW50-C1 **x-amz-cf-id** : hzwVoHfaHen2TR3cCNRsnwniXMc3_BaOWk7oa2DiQaWkioXqwSGRrg== **time\_namelookup** : 0.11359600000000000253 **time\_connect** : 0.01640300000000000091 **time\_appconnect** : 0.07292699999999999183 **time\_pretransfer** : 0.00030900000000000372 **time\_redirect** : 0.00000000000000000000 **time\_starttransfer** : 0.02220300000000000051 **time to download** : 0.00011099999999999999 **time\_total** : 0.22554899999999999949

Please note these metrics like time_namelookup do correspond to the same meaning curl's time_namelookup. 'Curl' shows a cumulative time measure, so it is always increasing. However I want to look at the time each stage took separately from each other, so curl's cumulative increasing timestamp was not helpful to me.

If you’re curious about what each of those metrics mean, such as time_appconnect (which is a complete misnomer and doesn't make sense), read this writeup that explains each curl metric.

Common pitfalls of speed testing CDNs

These are some mistakes I made while speed testing CDNs. These are easy mistakes to make so I decided to write about it.

Mistake #1: cold cache tests should be done where the CDN does not have the file cached.

Why it’s an easy mistake to make : most people think files are cached for only 24 hours. That’s not necessarily true. Some CDNs like Jetpack CDN evidently cache it for more than 1 day (based on my tests). Additionally, most people’s curl calls do not include the headers which have the warning sign that the file could be cached.

How to fix : check the headers returned. You should see the keyword “MISS” somewhere in there. If you see a “HIT”, that’s a warning sign. This is also why the python script prints out the headers for cold cache test and warm cache test.

Mistake #2: you need to actually download the file.

Why it’s an easy mistake to make : for some reason, it is extremely easy to accidentally make curl skip the download of the actual file. There are at least 3 ways of doing this.

If you set the “NOBODY” option to curl via pycurl, it does not download the file. I made this mistake in my python script.
If you set the “-O /dev/null” flag via the curl in the command line, it does not download the file. You think it would just download the file and dump it to /dev/null. No. curl is “smarter” than that and just skips the download altogether.
If you set the “-I” flag via curl in the command line, it does not download the file.

Why this is an important mistake to avoid : CloudFront does not cache the file if you do not download the file! And since CloudFront is an industry-standard CDN, it is likely other CDNs have the same behavior.

How you can detect this mistake : look at the download time (time_total — time_starttransfer). If it is in the hundreds of microseconds, that’s a problem. The speed of light is only 0.184 mile/microsecond. If you got only 200 microseconds, then that is 37 miles…roundtrip. Or 18 miles one way. It is highly unlikely there’s a datacenter that close to where you are. It means that there was no network transfer.

How to fix : in pycurl, write the downloaded contents to a BytesIO or StringsIO variable. To see how it’s done, see my python CDN speed test script. Or if you’re using curl from command line, avoid “-O /dev/null” and “-I” flag and make sure download time is at least several milliseconds.

Mistake #3: The input URL is invalid.

Why it’s an easy mistake to make : your curl speed test doesn’t report any 404 error. So you get back the speed test results without realizing there’s an error.

How to catch mistake : one red flag you wanna look at is the “download” time. If it’s less than 1 millisecond, that’s way too fast. Do the math. Speed of light is .184 per microsecond. There are 1000 microseconds in a millisecond. If you’re getting, lets say 200 microseconds, that’s only 36 miles…. roundtrip. One way, it’s only 18 miles. So 200 microseconds is way too fast, and indicates there’s probably no network transfer (i.e. no download) happening at all.

Of course, the other thing you could do is actually download the file…use wget. You’ll see an error message pretty easily.

How to fix the mistake : use the right URL.

Mistake #4: The input URL is HTTP not HTTPS.

Why it’s an easy mistake to make : people are careless or perhaps not technical enough. This is actually an important thing to ensure as I’m noticing the SSL exchange normally takes 50 milliseconds. Given a fast curl time is only around 120 milliseconds (from a home computer not cloud provider).

How to fix the mistake : be consistent in your testing. Use HTTPS for all or HTTP for all. And you should probably use HTTPS since that’s the standard (everybody uses HTTPS).

Why I chose not to test on a normal web server like DigitalOcean

Turns out it’s not a good idea to use DigitalOcean to run these CDN speed tests.

I initially thought it would be a good idea to run these CDN speed tests on a standardized server location like DigitalOcean. Since I travel a lot, I can’t guarantee my internet is always going to be the same quality, or the location could be drastically different, such as different countries.

So that’s what I did. I tried it out…and noticed DigitalOcean’s internet speed is fast — REALLY fast. Like, way faster than my home internet speed. I was a little shocked. I didn’t think download speeds could ever get that fast. Makes me wonder what I need to do to make my home internet as good as that, because I really thought I had the best possible internet that money could pay for (I basically use my computer 100 hours a week, OK. It’s important to me).

If you wanna see an example of the drastic difference in CDN speed…well here’s one. It’s using the CloudFront example that we’re using in the “Free CDN” blog series. When I ran my speed test python script with: python3 test.py [https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024×576.png](https://d3va53q3li7xt1.cloudfront.net/wp-content/uploads/2021/05/shoeb-1024x576.png), I got back a download time of a blazing fast 0.0225 seconds, or 23 milliseconds. Holy cow. And then doing the same thing on my home internet, I only get 0.2315 seconds, or 232 milliseconds. What the hell.

Out of curiosity, I ran an internet speed test on the DigitalOcean server. Like how the hell is it that much faster. That’s gotta get anybody’s interest up. And would you believe the numbers???

# speedtest-cli Retrieving speedtest.net configuration... Testing from DigitalOcean (128.199.187.118)... Retrieving speedtest.net server list... Selecting best server based on ping... Hosted by NewMedia Express (Singapore) [13.17 km]: 2.16 ms Testing download speed................................................................................ Download: 1740.52 Mbit/s Testing upload speed...................................................................................................... Upload: 1420.29 Mbit/s

LOL holy crap Download speeds of 1740.52 Mbit/s and Upload speeds of 1420.29 Mbit/s? How much money did DigitalOcean pay for that? I wanna buy it.

That is literally 10x as fast as my internet, which tops out at 160 Mbit/s (using fast.com).

So there are several reasons why I decided not to run these speed tests on DigitalOcean after all:

It’s hard to tell the difference between good or bad CDNs.
It’s not a real-world usage of how CDNs actually get used. Almost nobody is going to have internet speeds that fast, where it’s nearly 10x faster than the most premium internet you can get.

BTW: relevant article that also talks about the distorted internet speeds of cloud providers: how accurate are CDNPerf’s numbers? Answer: not very accurate. TODO: I will link to this essay (rant) once I’ve written it!

Why the CDN speed tests need to be done almost at the same time

This is probably obvious to most internet users but because I’m trying to measure CDNs as methodically as possible, I might as well explain it briefly.

Internet speeds usually vary greatly depending on the time of the day. For example, during the evenings, there is typically massive internet congestion as people get off work and get leisure time. Video bandwidth is particularly notorious for hogging a lot of bandwidth. For example, did you know Netflix alone hogs 40% of the internet traffic during the evenings? Google it. And that’s just Netflix. Imagine if you added Youtube too. The evenings just get really congested with internet traffic.

And since we are focused on CDNs as the variable, we should make internet conditions an invariant. This means running the CDN speed test for 2 CDNs at basically the same time (let’s say within 1 minute is fine).

Summary (TL;DR)

I measure three CDN download speeds: cold, hot, and warm cache.
The measurement is automated with a python script (it’s open sourced, see it here: https://github.com/speedtestdemon/speed-tests/blob/master/test.py)
CDN speed measurements on 2 or more CDNs are to be conducted within the space of a few minutes.
CDN speed measurement is conducted on a home computer with home internet, not on the cloud where internet speeds are 10x home internet speeds (easily).

Questions? You can ask me on social media https://twitter.com/SpeedTestDemon