Automated Report Generation with Amazon CloudWatch and Lambda - (Let's Build 🏗️ Series)

Monitoring your stack is one of the most important skills, especially when your architecture is becoming larger and you have many different services being used.

In this article, we are going to see how we can generate automated reports using Amazon CloudWatch and AWS Lambda.

The main parts of this article:
1- About AWS Services
2- Technical Part (code)
3- Result
4- Conclusion

About AWS Services

1- AWS Lambda: Which holds the code and the business logic
2- AWS IAM: For all the permissions inside the AWS cloud
3- Amazon CloudWatch: Monitoring service, where we can get logs

Technical Part

Now let's see our Lambda function that queries CloudWatch metrics for a specified EC2 instance and generates a simple report.

import boto3
from datetime import datetime, timedelta

def generate_report():
    cloudwatch = boto3.client('cloudwatch')

    namespace = "AWS/Lambda"
    metric_name = "Invocations"
    lambda_name = "test-python"
    period = 300

    start_time = datetime(2024, 1, 21)
    end_time = datetime(2024, 1, 22)

    response = cloudwatch.get_metric_statistics(
        Namespace=namespace,
        MetricName=metric_name,
        Dimensions=[
            {
                'Name': 'FunctionName',
                'Value': lambda_name
            },
        ],
        StartTime=start_time,
        EndTime=end_time,
        Period=period,
        Statistics=[
            'Sum'
        ],
    )

    return response

def lambda_handler(event, context):
    report_data = generate_report()
    print(report_data)

    return {
        'statusCode': 200,
        'body': 'Report generated successfully.'
    }

Make sure that your Lambda function has the following role enabled too cloudwatch:GetMetricStatistics

Result

First, let's see my Lambda invocations metrics:

Now, once we trigger the Lambda function that collects these metrics, we get the following result:

{
   "Label":"Invocations",
   "Datapoints":[
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         16,
         45,
         "tzinfo=tzlocal())",
         "Sum":3.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         16,
         50,
         "tzinfo=tzlocal())",
         "Sum":1.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         16,
         40,
         "tzinfo=tzlocal())",
         "Sum":4.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         20,
         15,
         "tzinfo=tzlocal())",
         "Sum":3.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         16,
         30,
         "tzinfo=tzlocal())",
         "Sum":3.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         20,
         20,
         "tzinfo=tzlocal())",
         "Sum":1.0,
         "Unit":"Count"
      },
      {
         "Timestamp":datetime.datetime(2024,
         1,
         21,
         16,
         20,
         "tzinfo=tzlocal())",
         "Sum":1.0,
         "Unit":"Count"
      }
   ],
   ...
}

Conclusion

Monitoring and always being aware of your application logs is one of the most important tasks of a DevOps Engineer. Using AWS Lambda and Amazon CloudWatch you can create very useful features that help you to increase your observability on your application. You can even create scheduled jobs to automate the metrics generation based on your needs.

If you did like my content, and want to see more, feel free to connect with me on ➡️ Awedis LinkedIn, happy to guide or help with anything that needs clarification 😊💁