(Note: after seeing stern feedback on Hacker News and reddit, I’m changing my test approach to hire 10+ contractors in popular cities around the globe to compare CDN download times. And if you disagree with my analysis in any way, please cordially do so 😊. I’m on twitter at @SpeedTestDemon)
I’ve learned Cloudinary basically does image and video transformations on top of the Fastly CDN. As such, I feel this is basically a review of the Fastly CDN platform. Fastly costs minimum $50/month, so it’s not free CDN I’m interested in analyzing. However, Cloudinary has a free tier going up to 25 GB of storage or bandwidth, so here’s “Analysis #2 of a Free CDN”.
Cloudinary was SUPER impressive. Cloudinary was way better than AWS CloudFront. They had a secret sauce: dramatic image transformation. Cloudinary somehow compressed a 703 KB image down to 37 KB.
Before I present the simplified summary, here are some links relevant to this article.
- My testing approach to CDNs
- Python script for measuring CDNs
- Analysis #1 of a Free CDN: Jetpack CDN vs CloudFront
- I tweet analysis about software performance or email list
As a very simplified summary, based only on hot cache results
- Cloudinary was faster than CloudFront by 3x (0.133 vs 0.397 seconds, or difference of 264 milliseconds)
- Even after controlling for image size, Cloudinary was still faster by 13% (0.11 vs 0.126)
Cloudinary had better performance for several reasons:
- How did Cloudinary compress that image 95% smaller? (~700 KB image to ~36 KB).
- Much more common DNS path (res.cloudinary.com vs d20zaq59cm4c4j.cloudfront.net), which allows for significantly faster DNS lookup. In one case, 0.11 vs 0.008 seconds.
- Uses Fastly CDN underneath. A lot of big companies (including Amazon) have adopted Fastly, probably for a good reason.
- CloudFront typically degrades a file from hot cache to warm cache very rapidly, whereas Cloudinary (i.e. Fastly) doesn’t do this until…weeks? (it’s a long time whatever it is).
What follows is a much more detailed technical analysis.
I had to do 2 speed tests. The 1st speed test was not apples-to-apples because Cloudinary had compressed the image by nearly 95%. The 2nd speed test, I controlled for file size, and this time Cloudinary came out ahead very slightly by 6% (although the main credit should be due to Fastly).
In the first speed test, Cloudinary really crushed CloudFront.
At first sight, I had to double-check for errors. Cloudinary was nearly 3x faster — on the hot cache! — and it didn’t make sense. Turns out the test was correct. Cloudinary simply compressed the image 95% lower (700 KB to 36 KB).
I had to do a second speed test to make it more apples-to-apples between CloudFront and Cloudinary (masquerading as Fastly). CloudFront did a lot better in the hot cache. Still sucked at the other two metrics.
There were several reasons for Cloudinary performing so well:
OK Cloudinary’s free tier is pretty impressive. The only limitation I can think of is you can’t host your whole site on it like you can with Cloudfront.
Now one moment…you do “get what you pay for”…while Cloudinary has great performance, it is limited in many other ways. You can’t update a file on Cloudinary easily, unless you wholesale rename the file (very annoying). The amazing performance applies mainly to images. JS and CSS won’t have the same amazing image compression, although you will see Cloudinary perform on par with CloudFront. See the conclusion for my thoughts on the Cloudinary vs CloudFront decision.
It should be noted that not every free CDN performs as well as Cloudinary. I found a couple free CDNs that performed really poorly.
Why is it interesting to test Cloudinary?
Honestly it was just because they had a free tier CDN and my goal here was just to test every free CDN.
I didn’t know anything about Cloudinary before writing this article.
But wow, the results have blown my mind. Good job Cloudinary guys! These guys have a good product.
After doing some more research on Cloudinary, I found out these guys have won some interesting accolades
- Inc. Magazine called Cloudinary the “gold standard” of image management on the web (according to Wikipedia at least).
- Actually has rather large usage (unusual that I’ve never heard of them even though I’ve been in the industry for many years). Usage stats are 1 million developers and 7500 companies (Wikipedia).
OK, so it’s got a reputation for great image management. I can see that actually. They did insanely well in this test.
Why is it interesting to test CloudFront as the gold standard in CDN speed tests?
Because Amazon is Amazon and they’re the biggest cloud provider out there (they are #1 at 32% of the market). But Cloudinary’s impressive speed test results here have me really questioning why I’m using CloudFront. Well Cloudinary’s premium tier is very expensive at $100/month, so it’s not an obvious switch. CloudFront is only at $2/month for the equivalent network out.
Brief Summary Of The Metrics Being Measured
We are interested in the hot, cold, and warm cache speeds because CDNs are essentially a cache layer. If you’re interested in reading more, see the analysis here: metrics measured in Jetpack vs CloudFront and my CDN testing methodology.
Couple findings on why Cloudinary was so much better than Amazon AWS Cloudfront.
Finding #1: Cloudinary’s DNS lookup is significantly faster than CloudFront’s DNS lookup.
This was similar to what I found in Jetpack CDN vs Cloudfront. Cloudinary is able to achieve a significantly faster lookup time because it’s DNS path is more likely to be cached by the intermediate routers. Cloudinary DNS lookup path is “res.cloudinary.com” which seems to be the shared DNS path between most Cloudinary users, so that DNS lookup path is receiving a lot of traffic. While CloudFront’s DNS lookup path is a very customized “d20zaq59cm4c4j.cloudfront.net”.
What’s very interesting is Jetpack CDN’s DNS lookup times were always nearly as fast as the hot cache’s DNS lookup time (even the cold cache 0.00223 seconds was so close to hot cache 0.0016 seconds), while Cloudinary’s DNS lookup times took a small hit (hot cache 0.00175 seconds to warm cache 0.0236 seconds). I think this can be attributed to Jetpack CDN’s immense popularity which means its DNS lookup path “i0.wp.com” is going to be cached much more frequently than Cloudinary.
Finding #2: Cloudinary had a mind blowingly good image compression algorithm
I’m not exactly sure Cloudinary got it so good. The same 703 KB image was somehow compressed down to 36 KB. That’s just really good and explains how Cloudinary was able to get much faster Data Transfer times than CloudFront.
Other interesting findings
I already talked about these in previous essays ( Jetpack CDN vs CloudFront) so I didn’t want to write more paragraphs for these.
- Generally, the TCP handshake and SSL handshake times are the same regardless of file size or file type. Most CDNs and servers are going to be equally fast on these metrics, unless they are using a slower CPU to do SSL computations.
- CloudFront’s warm cache (a request 30 minute after the most recent) is about 2x to 4x slower than hot cache. I found this a lot when testing Jetpack CDN. CloudFront likes to degrade a file from RAM to disk (NVMe or SSD) if it’s not seen active requests recently.
The first time I ran this comparison test, I was blown away by how good Cloudinary was. I thought Jetpack CDN was a good Image CDN. I was wrong. Cloudinary was way better.
- Cold cache: OK so Cloudinary was 5x faster than CloudFront….because Cloudinary doesn’t really have a cold cache? Because there is no expiration on the files (unless you get Cloudinary’s premium tier).
- Hot cache: I guess this is the metric that really matters, since it’s the most common case that a CDN is going to contain the file. And boy did Cloudinary impress. By compressing the file from ~700 KB down to 36 KB, Cloudinary beat CloudFront by 3x.
- Warm cache: Cloudinary did even better, beating CloudFront by 4x. Cloudinary hot cache didn’t degrade even after 30 minute wait time, while CloudFront once again degraded the file so it performed 33% worse (same thing I found in Jetpack CDN vs CloudFront).
Cold cache measurement and analysis (speed test #1)
CloudFront really lagged horribly here at 2.355 vs 0.395 seconds for Cloudinary.
Sharp readers will notice that CloudFront showed a much worse time (2.355 seconds) in this speed test comparison than in the Jetpack CDN vs CloudFront analysis (1.199 seconds). It’s because I just ran this CloudFront download at a different time than when I wrote the Jetpack CDN analysis.
Wait a second. CloudFront took 2.35 seconds while Cloudinary was only 0.39 seconds? Holy moly. Where did all that time go?
The two big time consumers were: 1) Request Latency, and 2) Data Transfer.
Regarding Request Latency: CloudFront took 1.022 seconds while Cloudinary took 0.258 seconds. What’s impressive is Cloudinary must have done two things during the cold cache lookup: it must have had the file still cached on a server despite my asking for the file only several days after getting that CDN setup. Cloudinary certainly did not fetch the file again, or it wouldn’t have gotten such a fast time. That’s really really good.
Cloudinary’s cold cache is not as fast as its hot cache. Most of the time went in Request Latency. For cold cache, it’s likely they had the file stored somewhere on disk, while for hot cache, it’s probably in RAM. Actually, since Cloudinary is built on top of Fastly, I would place a strong bet that Fastly has the file in RAM or else why are people buying Fastly?
Regarding Data Transfer: yeah this was really incredible. Cloudinary only needed 0.0069 seconds while CloudFront took 1.098 seconds. There’s only one main factor for this and it’s Cloudinary’s insane image compression algorithm which means Cloudinary only has to send 36 KB while CloudFront has to send 703 KB. It’s just not even close. See more analysis in Finding #2.
Regarding SSL handshake: honestly not sure why CloudFront took much longer at 0.091 seconds vs 0.056 seconds but the absolute difference is only 0.035 seconds, so it’s a small contributor to the overall delta of nearly 2 seconds.
Regarding TCP handshake: this is just the same for any connection. So there’s not a big absolute time delta.
Regarding DNS lookup: not the biggest factor for the large overall time delta but it’s still worth mentioning because on a relative basis, Cloudinary outperformed by over 100% (0.0494 seconds vs 0.124 seconds). See Finding #1 analysis for why Cloudinary’s DNS lookup is generally much faster than CloudFront.
Hot cache measurement and analysis (speed test #1)
This hot cache measurement was the one that really blew my mind.
Cloudinary smokes Cloudfront, nearly 3x faster (0.133 vs 0.397 seconds).
Wow! How did Cloudinary do that?
(Reminder: these times are an average of 10 curl calls).
Looks like all the times are pretty similar except for Data Transfer. In fact, Data Transfer accounts for 0.234 seconds of the overall time difference of 0.263 seconds. Literally about 90%.
It’s really shocking that CloudFront took so much longer to transfer the file, even in a hot cache state.
Reason is Cloudinary compressed the original 700 KB file down to 36 KB. That’s nearly a 20x reduction in file size. So no wonder Cloudinary ended up smoking the pants off CloudFront.
Notes:
Warm cache measurement and analysis (speed test #1)
Ouch. CloudFront nearly 4x slower than Cloudinary: 0.465 vs 0.126 seconds.
What happened??
(reminder: warm cache is download taken exactly 30 minutes after hot cache)
First off, a monstrous Data Transfer difference: Cloudfront 0.198 seconds vs Cloudinary 0.009 seconds. Absolute difference of 0.19 seconds, contributing to the overall time difference of 0.34 seconds.
Second, a monstrous DNS lookup difference: Cloudfront 0.115 seconds vs Cloudinary 0.0236 seconds. Absolute difference of 0.09 seconds, contributing to the overall time difference of 0.34 seconds. If you thought DNS lookup speeds don’t matter — well think again. See “Finding #1” for why CloudFront’s DNS lookup is so much slower than Cloudinary.
Data Transfer and DNS lookup combine for 0.28 seconds out of the 0.34 seconds difference.
Other notes:
- As said in Finding #2, CloudFront’s Data Transfer was so much slower because it was transferring ~700 KB of data while Cloudinary was transferring ~36 KB. Cloudinary compressed the image really well.
- Dang, Cloudinary’s warm cache faster than the hot cache time (0.126 vs 0.133 seconds). That’s impressive.
- I don’t know why CloudFront’s SSL handshake is always slower than Fastly’s SSL handshake. I’m still thinking about it. (Cloudinary is built on Fastly’s CDN).
- Yeah, CloudFront’s “Request Latency” was 2x slower. I’ve noticed CloudFront tends to degrade the file from RAM to NVMe or SSD.
Speed Test #1 conclusion and analysis
Cloudinary beat CloudFront in hot cache, cold cache, and warm cache, due mainly to the image automatically compressed to 36 KB and CloudFront still trying to transfer a 720 KB PNG.
Impressively, the hot cache was a staggering 3x faster: 0.133 vs 0.397 seconds.
What if we even the odds and make CloudFront and Cloudinary transfer the same file? Surely CloudFront should beat Cloudinary now? It’s not that simple. See the next section for Speed Test #2.
I re-ran the speed test to make sure CloudFront and Cloudinary were downloading the exact same file. To do this, I just uploaded the Cloudinary 36 KB file to CloudFront CDN.
Notes:
- Cold cache: I guess Cloudinary keeps the file in hot cache for an absurdly long time (this is a good thing for the end-consumer. It does not cost extra money for this feature). it takes a long time for Cloudinary’s cache to turn cold…so I’m skipping this test.
- Hot cache: Cloudinary still slightly faster (12%). Absolute difference is only 0.016 seconds. Not a big deal I guess.
- Warm cache: after waiting 30 minutes, Cloudinary didn’t degrade the cache at all, while CloudFront faced a 2x slow down.
Cold cache measurement and analysis (speed test #2)
So….I found out Cloudinary doesn’t really have a cold cache? Even if I try running the test a few days after my last fetch…it is still pretty much the exact same as the hot cache result….
For example, I ran the Cloudinary test again, and I noticed the cold cache was only 0.02 seconds slower than the hot cache…nearly all of which was in the DNS lookup.
The coldest I’ve ever seen Cloudinary was when I waited a couple weeks to test Cloudinary. If you test only a few days after, it is pretty much the same as hot cache.
This is actually a good thing for the consumer. It means Cloudinary doesn’t degrade the file down to cold cache until a couple weeks. It is kind of incredible.
I’m not going to bother writing much of an analysis for this.
Cloudinary’s cold cache is pretty much as fast as the hot cache, with the only difference of the DNS lookup (was not cached by intermediate routers).
CloudFront’s cold cache is just the same as last time except for significantly faster Data Transfer phase since we are making CloudFront download 36 KB.
Hot cache measurement and analysis (speed test #2)
Much closer where we can say that perhaps CloudFront is as fast as Cloudinary. Cloudinary holds a slight 12% faster time.
Cloudinary still 12% faster than CloudFront. Although the absolute difference is only 0.016 seconds so perhaps not a big deal.
DNS lookup, TCP handshake, SSL handshake, Data Transfer were all pretty close (these are the same operations for any file size or type).
The main difference of 0.010 seconds comes from Request Latency, where CloudFront takes 50% long (0.030 vs 0.20 seconds). Honestly not sure but perhaps Cloudinary has a more efficient file cache read than CloudFront. Cloudinary is built on Fastly, which has significant market share among enterprise companies.
So Cloudinary is faster, but it’s not really worth stressing about it at this point. The difference is small enough.
Warm cache measurement and analysis (speed test #2)
Same thing that happened in Jetpack CDN vs AWS CloudFront. CloudFront warm cache degraded significantly (relative to hot cache) — 2x slower (0.25 seconds vs 0.126 seconds). Cloudinary warm cache pretty much the same as hot cache.
Regarding Request Latency: the big rock was Request Latency: 0.076 vs 0.0168 seconds. Cloudinary’s Request Latency was the same in both hot and warm cache, so Cloudinary must be keeping the file in RAM. CloudFront definitely downgraded the file from RAM down to disk. This is very consistent behavior that I’ve seen in the Jetpack CDN vs Cloudfront comparison as well.
(reminder: warm cache is download taken exactly 30 minutes after hot cache)
CloudFront’s speed performance is once again significantly slower as found when I was doing Jetpack CDN vs CloudFront. Cloudfront’s slowdown came in 3 areas: DNS lookup, SSL handshake, and Request Latency.
Regarding DNS lookup: Cloudfront is slower here because the DNS name d20zaq59cm4c4j.cloudfront.net is entirely unique to my account so it is even more unlikely to be cached in routers than the Cloudinary DNS name res.cloudinary.com (which appears to be shared with multiple users).
Interestingly, Cloudinary’s DNS lookup was significantly slower when going from the hot cache to the warm cache (0.035 seconds vs 0.0016 seconds).
Another interesting point is Jetpack CDN’s DNS lookup was only 0.0092 seconds. This happened because the Jetpack CDN DNS name i0.wp.com is heavily used across millions of wordpress installations and Cloudinary has significantly smaller install base, so the Jetpack’s DNS name is way more likely than Cloudinary to be cached in routers.
Regarding SSL handshake: it is really unusual that Cloudfront’s SSL handshake takes nearly 2x as long as Cloudinary (0.084 vs 0.045 seconds), because the TCP handshake occurred with equal speed on both CDNs, which indicates the problem is not the internet speed. There’s 2 possibilities: 1) for some reason the certificate authority is taking a long time to respond, and 2) the CloudFront is serving the traffic from a cheap CPU server so the SSL calculations are taking much longer?
I guess possibility #2 is more likely. Still weird.
My thoughts on this are extremely mixed. I would say it highly depends on your circumstances whether you should use Cloudinary.
Reasons to use Cloudinary
Cloudinary is a highly performant CDN built on top of Fastly CDN, very widely used among tech companies such as Reddit and even Amazon itself (which is weird because Amazon has its own CDN called CloudFront…).
Cloudinary has some very impressive image compression capabilities. The 700 KB test image was compressed down to 36 KB. It’s possible to do the same thing in CloudFront…but it’s more involved. You’d need a WordPress plugin that was as good at compressing images. Actually Jetpack CDN was already compressing, but only down to 216 KB. Again, really impressive stuff from Cloudinary. I guess nobody is compressing the image that well…(…yet. I will probably build this WordPress plugin).
Cloudinary keeps files in a hot cache state for weeks (much longer than CloudFront which degrades the file in less than 30 minutes).
Cloudinary also boasts video CDN delivery. I have not tested using videos so I can’t make an evaluation here. Although…it’s likely the video CDN performs as well as the image CDN, although we still need to test it.
Even in the scenario that CloudFront is delivery the same compressed file as Cloudinary, I’m seeing Cloudinary (i.e. Fastly) is performing better on cold, hot, and warm cache.
- 12% faster on hot cache
- 2x faster on warm cache
- Cloudinary’s cold cache is a couple weeks, not 30 minutes.
So…Cloudinary has been extremely impressive indeed.
Reasons not to use Cloudinary
However, the free tier is extremely limited to only 25 GB/month of data transfer, while the premium tier is very expensive (starting at $99/month). Cloudinary is by far the most expensive CDN I’ve tested so far. To put the $99/month in perspective: CloudFront only charges $0.08/GB, and Cloudinary has a free tier of 25 GB and premium tier of 225 GB. Run the numbers: 1) for the free tier, CloudFront would cost only 25 GB * $0.08/GB = $2.00. 2) for the premium tier, it’s 225 GB * 0.08/GB = $18. And…Cloudinary’s charging $99/month? Wow.
On the plus side, if you’ve got a low or medium traffic website and you’re sure you’re transferring less than 25 GB a month…by all means go for Cloudinary. Since you’re staying in the free tier.
Another limitation is you can’t host a static HTML page on Cloudinary like you can with CloudFront and CNAME Aliases. You can host static assets like CSS, JS, images, and videos…but not HTML. For example, this entire blog is hosted off CloudFront, even the initial HTML request. I can’t do that with Cloudinary…just offload the images/videos/CSS/JS, that’s it.
As an aside: there’s a good reason why Cloudinary is so expensive…it is built on top of Fastly CDN. Cloudinary’s primary value-add are image and video transformations. And Fastly is already fairly expensive (minimum $50/month).
Final Analysis: to use or not use Cloudinary?
In the final analysis, what you’re paying Cloudinary for are the “on-the-fly” image and video transformations. Cloudinary contains some bells and whistles that makes it a powerful image & video CDN. Is this worth $99/month to you? i.e. $1188/year.
Also remember Cloudinary is basically a reseller of the Fastly CDN. Cloudinary’s primary value-add is in the image and video transformations. Are you the kind of person or business that needs Cloudinary’s advanced features? It costs $99/month…
What kind of person or business are you?
- Hobby blog: since you’re probably a low-traffic site, you could easily stay within Cloudinary’s free tier of 25 GB.
- High traffic blog: in my opinion, you can easily compress the images yourself to get it as good as Cloudinary. Then for fast delivery, either use Fastly directly or use CloudFront. Since using Cloudinary would simply be using Fastly indirectly.
- Your website earns $1000/month: well $99/month is 10% of your profits. Not really worth it.
Software engineer building complex app: Cloudinary has some complex image transformations like cartoonify, brightness, rotate. That’s neat. Using Cloudinary as a REST API could save you some time (and developer time is always the most expensive).
Python CDN Speed Test Script: I use this to automate the collection of metrics about CDNs.
APPENDIX: Wanna reproduce our test results? Here’s how to setup Cloudinary
- Login or Sign up in Cloudinary.
- Install and Activate Cloudinary Plugin in WordPress
Top comments (0)