DEV Community

Vlad Z
Vlad Z

Posted on

An $11,400 AWS bill, and CloudWatch Logs was $2,100 of it

I got access to the account on a Tuesday morning

The founder had sent over credentials the night before with a message that said roughly: I think we're overpaying but I don't know where

Total monthly spend: $11,400

Their product had about 800 active users. For 800 users that number felt wrong immediately

First 15 minutes: just looking. Not touching anything. Building a mental map of what's here and roughly why

CloudWatch Logs was $2,100 a month. For an account this size that's the first red flag

I pulled the log groups. Forty-three of them. Most of them with no retention policy set, which means logs accumulate forever. Several of them receiving data from services that no longer existed. Three of them receiving data at a rate that made no sense for their stated purpose

One log group was ingesting 180GB of data a month from a service that was logging at debug level in production

Not info level. Debug level. Every function call, every variable state, every internal operation. All of it being shipped to CloudWatch and stored indefinitely at full price

That one log group was $900 a month

The service had been deployed eight months ago. Debug logging had been turned on during initial development and never turned off. Nobody had looked at the CloudWatch bill specifically. It was just part of the total that nobody questioned because the total was growing gradually

We set retention policies on all 43 log groups. Turned off debug logging on the offending service. Deleted the log groups receiving data from services that no longer existed

CloudWatch bill the following month: $340

$1,760 saved. One afternoon of work

And we hadn't touched anything else yet

Top comments (1)

Collapse
 
florian131313 profile image
G •

The $900 group bills on the way in. A retention policy caps how long those bytes stay, it doesn't shrink what you already shipped this month. Debug off is what moves the next invoice. The 43 groups with no retention are a different leak: storage that never expires, including services that are gone.