Someone flipped the log level to debug to chase a payment bug on a Tuesday evening. The bug was found and fixed by Thursday. The log level stayed at debug for five weeks, and we noticed because the logging bill for that month was larger than the compute bill for the service producing it.
The volume was the visible problem. About ninety times our normal ingest, most of it from a request-tracing statement inside a hot loop that had been perfectly reasonable when written for local development. Retention on the hot tier is expensive per gigabyte, so it took barely a fortnight to overshoot the whole quarter's budget.
The worse problem was what was in it. Debug level in that service logged full request bodies before validation, which meant card tokens, full addresses, and in the checkout path a field the mobile client was sending that we didn't even know existed. We had shipped personal data into a third-party log platform, past our retention policy, for five weeks. That turned a cost incident into a privacy one, with a report to write and a purge to coordinate against an index that was never designed for selective deletion.
Nothing about the fix required cleverness. Log level is now configuration with a TTL: raising it above info in production writes an expiry timestamp, and a scheduled job reverts anything past it and posts in the channel. Debug statements that log a whole object were rewritten to log named fields, and a redaction list runs at the emitter rather than in the platform, because redacting after shipping is redacting after the mistake. We put an alert on ingest volume per service per hour with a hard cap, so the next runaway trips in twenty minutes rather than five weeks.
The habit I'd argue for is treating an increase in log verbosity as a temporary change to production, the same category as scaling something up during an incident or disabling a rate limit. Both of those we already reverted reliably, because both had an obvious cost. Logging felt free at the moment it was turned on.
A debugging change with no expiry is a permanent change you made without meaning to.
– Sergey Shinder
Top comments (0)