Like many people, I’m a Dolly Parton fan. I’ve been wearing a mask with her face on it all year (thanks, Chloe!); I celebrated her birthday in January with a Chrome DevTools script.
I’m also an ice cream fan.
There are lots of us who really like both of these things! Jeni’s Splendid Ice Creams combined two crowd pleasers when they collaborated with Dolly on a limited edition Strawberry Pretzel Pie pint to benefit her Imagination Library.
When the pints went on sale online, Jeni's website crashed, fast:
Jeni's Splendid Ice Creams@jenisicecreams
Did y’all just break our website?16:02 PM - 08 Apr 2021
This Jeni’s Splendid Schadenfreude inspired me to ask: What actually happens when a website crashes? My Daily colleague Brian was kind enough to entertain my curiosity. Read on for what I learned from our conversation, including a review of what happens when you visit a URL in your browser, an intro to CDNs, and a few possible failure modes. To be clear, that last part will just be some Splendid Speculation.
We all scream for ice cream
It’s a question almost as popular during tech interviews as Dolly is all the time: What happens when you visit a URL in your browser?
I wrote about what happens when you visit a video chat URL in your browser, and at the highest level it’s largely the same for any link. The browser starts the Domain Name System (DNS) lookup process, which matches the URL to the IP address of the server hosting the site. Assuming the lookup is successful, the browser initiates a TCP connection with that IP, and sends a request for the site’s resources. Back and forth negotiation ensues, and if all goes well, the requested resource is returned.
That’s a lot of back and forth for a single request. Multiply that by the number of people who scream for both Jeni’s ice cream and Dolly Parton, and very quickly there are a lot of requests and negotiations for a single server to handle.
Unlike novelty ice cream flavors, managing web traffic is not a new and limited edition problem. Content Delivery Networks (CDNs) have been around for years to help businesses manage it.
CDN: a coat of many servers
(Okay, a CDN is really more like a network of many servers, but I had to try to make the song work).
When a website uses a CDN, the CDN makes its resources, like images and JavaScript, available across the CDN’s server network. There are lots of advantages to this. If resources were instead concentrated at a single server location, then it matters where site visitors physically are. The farther away a visitor is from the server, the longer it would take for the server to return the resources they request from it. Websites get around this problem by using a CDN. The CDN is configured to know where to find all of the website’s resources, and the website then points its DNS records at the CDN. Then, when a visitor looks up the website and their browser initiates the DNS process, it finds the addresses at the CDN, and the nearest server in the network returns the resource.
Unfortunately, even with a CDN, lots of things can still go wrong. I was one of many Jeni's customers who saw this message from CDN provider Cloudflare when trying to get my hands on a pint of Dolly's ice cream.
While it’s impossible to know what exactly happened, we can think of a few things.
Failure modes: the salt in my tears
A “thundering herd” took down the host
Like many hyped limited product releases, Jeni’s launched the Dolly pints for sale online at a specific time: 12pm ET on April 8, 2021.
Many fans across the country flocked to the website on the dot, flooding the CDN with requests for resources. If the CDN servers all across the network don’t have the resources stored locally (stay tuned for more on caching configuration!), they’ll need to request them from the host site. All of those geographically distributed servers making the same request at the same time from the same host may have quickly overwhelmed it.
Depending on different servers’ configurations across the network, some servers at some locations could be requesting different resources, or be closer to the host. That explains why some customers, though very few, were able to successfully buy ice cream, and why so many different people saw so many different screens.
CDN configuration conflicted with site traffic needs
When a website sets up a CDN, it configures how long the CDN holds onto cached resources. The CDN can update resources every time a visitor requests them (generally leading to slower responses and not ideal!), or to only refresh a resource every X or so minutes, depending on how okay the host is with a page or image being stale when somebody visits the site.
If lots of updates are happening, let’s say the number of available pints in a database is decreasing, but a CDN only checks for that update every X minutes, somebody making a request could be on the receiving end of stale data, and then unable to complete a checkout.
The way CDN settings interact with host settings, especially under a large amount of traffic, can quickly get complicated and break things by sending too many requests.
Host-specific processes got overwhelmed
While a CDN can cache static resources like images on a marketing page, it can’t actually help with some of the more interactive parts of a website that involve sharing unique customer information, like entering a credit card or billing address. It’s possible that the origin server could’ve been overwhelmed with people trying to actually complete purchases. CDNs improve overall capacity with lots and lots of servers in lots and lots of places, but CDN resource caching doesn’t necessarily help the website process individual purchases. The error screen I saw on jenis.com did indeed look different from the error screen I saw on shop.jenis.com.
It’s all wrong, but it’s all right
Jeni’s specifically called out website traffic testing in their statement about the Dolly debacle:
Of course, it’s always possible to make a mistake in preparedness, like just not having enough servers, etc. But, it’s also really, really hard to simulate a Dolly Parton level of website traffic! Predicting the exact requests to be made in the exact configuration is impossible to simulate to a 100% degree of accuracy. And, while there are lots of advanced website testing services today, nothing exactly replicates real requests from real customers in real time. It’s kind of like how Dolly herself lost a Dolly Parton lookalike contest. Imitation and reality are not the same.
Dolly’s ice cream didn’t break the internet. It just overloaded Jeni’s host server in some way, the details of which are impossible to pinpoint from the outside. But, like a country song, it gives us some things to think about and stories to tell.
Top comments (0)