As a kid, I had the annoying habit of turning lights on in the house and forgetting to turn them off. My dad had to lecture me a few times about how electricity costs money and how I was wasting it.
Today, I don't leave the lights on anymore. However, I found myself involved in a different kind of waste.
As an SRE at Indeed, the first project I tackled was to reduce our Datadog spend. I found myself looking at a multi-million dollar budget on various Datadog expenses - metrics, logging, tracing - the whole shebang. The task was overwhelming, but my manager did point me to a quick win - reducing our metrics spend.
How Metrics Billing Works
Datadog bills metrics on two factors - the ingestion volume of each metric, and the number of unique combinations of tags each metric has, known as cardinality.
Depending on the number of different tag values a metric had, the cardinality blew up exponentially. We were spending hundreds of thousands of dollars per year tagging these metrics.
How I saved $100k+ per year
I catalogued the progress of cost savings per metric via a spreadsheet, tracking metric names, tags, cardinality, and potential cost savings, starting with the biggest offenders. I calculated potential savings with a tool to identify which tags were being queried within our Datadog namespace (which is now located under the Manage Tags menu in the metric view) and which ones were not. The view also calculates the hypothetical cardinality if you change the tags on a metric without saving the changes. By removing the unqueried tags, I calculated the theoretical cost savings and started reaching out to metric owners who had the highest theoretical savings.
Many of these folks added extra tags for debugging purposes and didn't realize how much it would cost. The most expensive metrics had cardinalities upwards of 200,000! Most were willing to part ways with tags they didn't use for dashboards, but some held on for debugging purposes. Even so, after a month of chasing down metric owners, I managed to trim down about $100k in annual metric costs.
Takeaways
- Having many tag keys wasn't as big a problem as having many tag values. More tag values = exponential cardinality growth.
- Look before you leap. Make sure metric users are on board before cutting down tags.
- Doing this once wasn't enough. Metric bloat inevitably happened, so regular check-ins helped.
Happy savings!
Top comments (0)