Most log platforms treat each line of stdout as a separate log entry. So one plain-text Java stack trace arrives as 40 or more entries: each one billed, most with no level or timestamp, and impossible to search as a single error. Python tracebacks, Node errors and .NET exceptions have the same problem.
It rarely shows up as a line item. It just makes the bill bigger and outages harder to debug.
How to tell if it's happening to you
Search your logs for entries that start with:
- whitespace or a tab
-
at(Java and .NET frames) -
Caused by:or... N more Traceback (most recent call last)
Another sign: your platform reports far more "errors" than your application actually threw. If a single failed request produces dozens of error entries, you're paying for line splitting.
Fix 1 (best): structured logging
Have the logger serialize the exception into one field of one JSON line:
{"level":"ERROR","message":"payment failed",
"error.type":"java.net.SocketTimeoutException",
"error.stack":"java.net.SocketTimeoutException: Read timed out\n\tat ...",
"order_id":"8812"}
One error, one entry, fully searchable. It fixes levels, timestamps and request context at the same time. Every major language has a JSON encoder for its standard logger: Logback or Log4j2 for Java, pino or winston for Node, python-json-logger or structlog for Python, Serilog for .NET.
Fix 2 (stopgap): join the lines in the collector
When you can't change the application yet, have the log shipper stitch multi-line events back together before they're sent. With Fluent Bit:
[FILTER]
Name multiline
Match kube.*
multiline.key_content log
multiline.parser java, python, go
The CloudWatch agent and Google's Ops Agent have equivalent multi-line settings. It's a stopgap because the joined entry is still unstructured text, but it stops the per-line billing today.
Bonus: AWS log groups that never expire
While you're looking at logging costs: CloudWatch log groups never expire by default. Any group nobody configured keeps growing forever. This lists every group with no retention set:
aws logs describe-log-groups \
--query 'logGroups[?!retentionInDays].logGroupName' --output text
Then set one, for example 30 days:
aws logs put-retention-policy --log-group-name <name> --retention-in-days 30
For comparison, Google Cloud's _Default bucket keeps logs for 30 days unless you change it, and Azure Log Analytics defaults to 30 days for most tables.
The short version
- Search for stack-trace fragments to see how much of your volume is split lines.
- Switch to JSON logging where you can.
- Join lines in the collector where you can't yet.
- Put a retention policy on every log group.
Originally published on the Lean Team blog. Free download: the Codebase Antipattern Checklist, 48 signs a codebase is getting harder to change.
Top comments (1)
The split line problem is easy to miss because the cost never appears as a separate line item, and the diagnostic detail also points the right way. Searching for entries that start with whitespace or Caused by gives teams a quick way to measure the waste before changing loggers. The CloudWatch retention note is a useful second check, since log groups with no expiry keep growing long after the original incident is forgotten.