Watch out your Cloudwatch logs, or it could be expensive!

A couple of years ago when we started working with AWS, just a couple of weeks after we deployed our very first lambda + API Gateway, we received a message on Slack from one of our DevOps asking what the heck we where doing with our application, since it was quite contributing to the bill, despite the low usages.

The app was mostly internal and wasn't doing much, still, Cloudwatch logs were growing huge and huge.
What was wrong? Well, first of all, the app had been deployed to production with all logging enabled, and more importantly, we were logging the entire result of the lambda before returning it - and that result was a JSON file. A pretty big one!

And with Cloudwatch, you end up paying for all the bytes ingested and stored. That's why we were paying so much - without much advantage!

Disable all unnecessary logs

• Do NOT use console.log
• Use one of the many solutions out there to log based on level and environment.

set up a retention period on your logs.

Normally you don't need your logs to remain stored for too long. Unless you have to deal with some auditing, you can safely delete them and free some space ( and save some money).
Again, it is advisable to do that base on the stage/environment, you definitely don't need your development logs for more than the time you are debugging.

With the CDK is just a matter of passing that value to your lambda configuration.

 logRetention: isProduction ? RetentionDays.ONE_WEEK : RetentionDays.ONE_DAY

in Serverless it is a bit more cumbersome because you can't rely on such conditionals

custom:
  logRetentionDaysByStage:
    production: 7
    other: 1

provider:
logRetentionInDays:  ${self:custom.logRetentionDaysByStage.${self:provider.stage}, self:custom.logRetentionDaysByStage.other}

Something we did not do until recently though, was actively monitoring the logs being ingested and stored, which is important because you might need to log information and keep that for some time, but due to some specific circumstance, a server returning some weird messages, a sudden unexpected peak in the usage of your app, you could end up logging ( and paying) way too much).

Check CloudWathc Metrics

• Go to the Cloudwatch Metrics / Logs / Log Group Metrics and select the resources you want to monitor (for example your Lambda - remember to add the namespace (/aws/lambda) to filter among all of them ).
• Adjust the dimension and period and you will have all the info and select Cloudcreate a Dashboard with

Here, for example, I selected SUM for the last 7 Days:

This is handy, but not much if you have to check it often and you have to click around every time in the UIConsole.

Much better if you set up a Dashboard with those Metrics and Graphs so that you can go back to it ever time. (watch out though, because AWS will charge you also for the number of dashboards)

Another option could be writing a script. Using the AWS CLI you can run
aws logs describe-log-groups to get all your log groups or specify --log-group-name-prefix /aws/lambda to filter the resources and get only your lambdas ( as you did in the console)

This will return a description of the Logs stored for each resource:

     {
            "storedBytes": 18670230,
            "metricFilterCount": 4,
            "creationTime": 1582203444572,
            "logGroupName": "/aws/lambda/YOUR-LAMBDA",
            "retentionInDays": 30,
            "arn": "arn:aws:logs:eu-west-1:YOUR_ACCOUNT:log-group:/aws/lambda/YOUR-LAMBDA:*"
        }

Running such scripts allows you to quickly monitor if everything was set up correctly ( in terms of retention days and metrics) and how big your storage is growing.
Is'already very helpful when you want to run big spring cleaning every now and then.

If on the other hand, you want to monitor how much data you are ingesting/writing to your logs then grab the logs name from the above list and for each resource request the statistic.

aws cloudwatch get-metric-statistics --metric-name IncomingBytes --start-time 2020-04-10T00:00:00Z --end-time 2020-04-16T23:59:59Z --period 604800 --namespace AWS/Logs --statistics Sum --region eu-west-1 --dimensions Name=LogGroupName,Value=/aws/lambda/YOUR_LAMBDA --unit Bytes

This will return the same values you can see in the console.

{
    "Datapoints": [
        {
            "Timestamp": "2020-04-10T00:00:00Z",
            "Sum": 56184051.0,
            "Unit": "Bytes"
        }
    ],
    "Label": "IncomingBytes"
}

Of course, depending on how many lambdas you have and on the size of your business, and bill, you might want to choose the best option between running manually a script, paying for multiple dashboards, or eventually having alarms notifying you when something goes over the expected threshold.

Hope it helps