DEV Community

Cover image for Check your settings, save mad loot
Justin Dorfman
Justin Dorfman

Posted on • Updated on

Check your settings, save mad loot

Photo by Christine Roy

TL;DR

I co-founded and have been leading the BootstrapCDN project since 2012. I have made many mistakes, here is one I hope others can learn from. “Set and forget” web services can be a blessing and a curse. No matter what service you are using so, log in and audit the bill.


Since 2013 we have been using S3 as our object store origin. You see, the beauty of S3 is how reliable/secure it is. That’s the blessing. The curse to that is you rarely have to login.

Late last November, I had to login to check a setting and had some time to kill so decided to check the cost explorer. I always knew our S3 costs where high but never dove into it because it was never an issue. So I decided I see what was happening and digging through the S3 logs, noticed strange query strings (e.g. ?v=1&v=2&v=3...&v=5409) which was treated as a separate object that couldn't be cached (on the CDN) since it was constantly changing, "Cache Busting" so to speak. I logged into the MaxCDN Control Panel, and disabled “Treat as separate cacheable item”.

This is what 1 simple change did:

Things I learned

  • At the time I started the investigation, at our scale, a 0.43% non-cache hit ratio equaled 297mm+ requests going back to the origin i.e. S3.
  • Making changes to a service that serves ~72 billion requests a month is scary but as long as you have a team to help monitor abnormalities, don’t be afraid.
  • No matter how well things are working, login and audit!

Before change (October 2017):

After change (January 2018):

In closing

Writing this post wasn’t easy. Admitting to something so stupid opens yourself up to ridicule, but I am 99.57% positive there are others making similar mistakes. No matter what service you are using, login and audit. In the meantime, I'll be finding out why 211mm+ requests are cache misses.🤔

Top comments (4)

Collapse
 
ben profile image
Ben Halpern

Damn the size of the numbers mentioned in this post amazes me.

Collapse
 
jdorfman profile image
Justin Dorfman

It wasn't overnight:

Collapse
 
nebojsac profile image
Nick Cinger

Hah, awesome.
AWS is such a money sink until you start realizing where you're losing the most. It takes a while to a get your bearings, but until then you're already down a few K.

Thanks for sharing real numbers, always interesting to read something like this!

Collapse
 
r3lik profile image
Mike Czarny

I got charged ~$100 on AWS for a Docker project that I forgot about. All it did was serve cat gifs.