loading...
Cover image for Check your settings, save mad loot

Check your settings, save mad loot

jdorfman profile image Justin Dorfman Updated on ・2 min read

Photo by Christine Roy

TL;DR

I co-founded and have been leading the BootstrapCDN project since 2012. I have made many mistakes, here is one I hope others can learn from. “Set and forget” web services can be a blessing and a curse. No matter what service you are using so, log in and audit the bill.


Since 2013 we have been using S3 as our object store origin. You see, the beauty of S3 is how reliable/secure it is. That’s the blessing. The curse to that is you rarely have to login.

Late last November, I had to login to check a setting and had some time to kill so decided to check the cost explorer. I always knew our S3 costs where high but never dove into it because it was never an issue. So I decided I see what was happening and digging through the S3 logs, noticed strange query strings (e.g. ?v=1&v=2&v=3...&v=5409) which was treated as a separate object that couldn't be cached (on the CDN) since it was constantly changing, "Cache Busting" so to speak. I logged into the MaxCDN Control Panel, and disabled “Treat as separate cacheable item”.

This is what 1 simple change did:

Things I learned

  • At the time I started the investigation, at our scale, a 0.43% non-cache hit ratio equaled 297mm+ requests going back to the origin i.e. S3.
  • Making changes to a service that serves ~72 billion requests a month is scary but as long as you have a team to help monitor abnormalities, don’t be afraid.
  • No matter how well things are working, login and audit!

Before change (October 2017):

After change (January 2018):

In closing

Writing this post wasn’t easy. Admitting to something so stupid opens yourself up to ridicule, but I am 99.57% positive there are others making similar mistakes. No matter what service you are using, login and audit. In the meantime, I'll be finding out why 211mm+ requests are cache misses.🤔

Posted on by:

jdorfman profile

Justin Dorfman

@jdorfman

👋 Hi! I started to get heavily involved with the FOSS community in 2012. I contributed to Bootstrap, Font Awesome, jQuery, GNU Bash, and many others. I also host a weekly podcast called "Sustain".

Discussion

markdown guide
 

Damn the size of the numbers mentioned in this post amazes me.

 
 

Hah, awesome.
AWS is such a money sink until you start realizing where you're losing the most. It takes a while to a get your bearings, but until then you're already down a few K.

Thanks for sharing real numbers, always interesting to read something like this!

 

I got charged ~$100 on AWS for a Docker project that I forgot about. All it did was serve cat gifs.