Don't use CDN!

If you are an experienced web developer/architect, you may find the title of this article questionable. The conventional wisdom is that you SHOULD use CDNs for better performance, easy backups and many other benefits.

What is a CDN?

For people who may not be familiar with the term, let's make it clear: CDN means Content Delivery Network. The idea behind CDNs is simple. The primary customers of CDNs are content providers on the web. As a content provider you have many users around the world. Typically, the content is dynamically generated, and it will have references to static assets. There are some completely static sites where it's just hosted HTML, CSS, JS, and other static assets like libraries, fonts, videos and images but most websites will have some part of their content generated on the server.
This is true even for single page applications. That's what we mean by dynamic content. It is the data coming from the server, usually with a database.

CDNs provide a means for you to upload your static assets and have it distributed to their own servers around the globe. CDNs operate with a domain name that resolves to the closest server of the user trying to reach it. That way, users can download your static content much more efficiently from a server that is closer to them physically than your own servers, which can mean longer distance and slower download.

This idea is simple, but it has an assumption that is easy to overlook: connection costs.

How I got to this point?

I have built two website editing platforms in my career. In the first one, we setup a load balanced cluster. It was 2010, so we actually built the entire thing with bare metal servers and F5 load balancers. We stored the static assets in Lustre shared parallel file system and served them from the same server as the dynamic content. All that time, I could not shake the feeling that I was supposed to be using CDNs because that's the conventional wisdom, but I could not find the opportunity to implement it while I was there.

In my second company (WebDigital), we used AWS. Determined to do things right this time, I used S3 to store digital assets. I setup Cloudfront for domain resolution, and I was expecting top-notch performance. And it was pretty good. See the page loading graph for desktop:

Then I looked at the page speed insights. Desktop looked good, but mobile was showing a warning. See below.

First contentful paint is over 3 seconds. To be clear, that's basically Google emulating a low-powered mobile phone on a 3G network. Before you say "nobody uses 3G anymore", see Surma & Jake Archibald's video explaining why it's still important to be able to handle these low-powered phones on slow networks.

The problem

I went to webpagetest.org and ran a mobile test. Here's what it looked like:

Here are some average stats for this:

• WebDigital: 444ms DNS + 302ms connect + 341ms SSL negotiation = 1087ms
• AWS: 313ms DNS + 303ms connect + 379ms SSL negotiation = 995ms
• Google fonts: 302ms DNS + 302ms connect + 343ms SSL negotiation = 947ms

As you can see, the problem is that these low-powered mobile phones take more than a second to establish a connection to ANY server!!! It doesn't matter if you have many servers around the globe closer to the users. The problem is CPU power! DNS query, TCP connection and SSL handshake take more than a second, regardless of the server. Yes, AWS and Google seem to be able to respond a bit faster to DNS queries, but connection and SSL take just as long, and this is BEFORE we have a single byte.

If you look at TTFB (time to first byte) on these servers, it's typically around 1.2 seconds. Think about this. Your own page comes at 1.2 seconds. At best, you will then connect to the CDN, which takes another 1.2 seconds. You end up with only 0.6 seconds left to do anything on the page, and it's not enough in most cases.

And for people that will recommend techniques like lazy loading, async loading, defer, push, etc. I tried them all. It really does not change much. Most of the time, you need the static assets to be loaded for the page to be considered loaded. Here's an example of what it looks like with HTTTP2 push:

Use Single Server

Here's the bottom line and performance principle learned: use single server if possible. I spent the next eight months moving S3 to EFS to eliminate the connection and added ability to download the Google fonts to it and serve it from our servers. After all the hard work, here's the stats with single server page load:

Looks nice, huh? No more long connection wait times. Here's what it looks like on page speed insights:

Never Use CDN?

Note that the performance principle does NOT say: "don't use CDN" - it says use single server. For example, if you have a static website where you upload your page to the CDN along with other static assets, CDN is a great fit! That is to say there are still many cases where CDNs are useful. However, if you have a dynamic content generating application, and you want to host the static assets on a CDN, you have to consider that low-powered mobile phones will not be able to load your pages under 3 seconds.