Lee O'Connell

Posted on Aug 14, 2020

Custom Cloudwatch Metrics using the CDK

#aws #python #devops

Over the past few months, I have been working to create an internal dashboard using AWS CloudWatch for my summer internship.

After trying to find a way to easily deploy the dashboard I came across the AWS Cloud Development Kit. This allowed me to create a Stack via a language I am familiar with (Python), rather than using CloudFormation.

This is by no means a tutorial with best practices on creating a stack with the CDK, but hopefully someone can find it helpful for getting started.

Thanks to Simon-Pierre Gingras for his post that helped me through custom metrics and the CDK.

Installing the CDK

I will link to the AWS Docs for installing the CDK: Installing CDK Docs

Let's get started!

Once we have the CDK installed we need to create a directory for our project to live in, once created we need to change directory into it and create the cdk project.

mkdir my-project
cd my-project
cdk init app --language python

This creates all the code/files necessary for the CDK. It also creates a python venv which we will need to start and install the requirements.txt onto.

source .env/bin/activate
python -m pip install -r requirements.txt

Creating the Dashboard

Now we have the project created we can get into creating a basic dashboard and include some AWS widgets for now.

Firstly we will need to import a few things from aws_cdk
I will explain what each is used for as we go along.

from aws_cdk import(
  aws_lambda as _lambda,
  aws_cloudwatch as cw,
  core
)
from aws_cdk.aws_cloudwatch
import TextWidget, SingleValueWidget, GraphWidget

Now we can create the dashboard and add some widgets. To do Under the __init__ function in the my_project_name_stack.py file we will add the following code:

cw.Dashboard(
  self,
  "dashboard_name",
  widgets = [
    TextWidget(
      markdown = 'Title widget'
      width = 24,
      height = 2
    )
  ]
)

This code will create a dashboard with the name of dashboard_name and will display a single text widget with the markdown specified. We have also added width and height attributes which will tell the dashboard how to display the widget.

We will add a more useful AWS default metric under the previous widget using the SingleValueWidget function:

SingleValueWidget(
  metrics = [cw.Metric(
    metric_name = 'ResourceCount',
    namespace = 'AWS/Usage',
    dimensions = dict(
      Type = 'Resource',
      Resource = 'vCPU',
      Service = 'EC2',
      Class = 'Standard/OnDemand'
    )
  )],
  width = 6,
  height = 3,
)

Add the code above to metrics=[]. Following up each separate widget with a , as the metrics=[] is expecting an list of widgets and uses the list to format the dashboard which we will look at shortly.

Deploy the stack

Now we have some basic widgets in the stack, we can try to deploy the dashboard to AWS using the CLI.

Firstly make sure you have valid AWS credentials in the .aws/credentials file within the project directory.
To test we don't have any errors we can use cdk synth to convert our CDK code into a cloudformation change set (This is what AWS cloudformation will receive when building the stack)
Finally we can deploy the stack to AWS using the cdk deploy command this will create the change set and send it to AWS CloudFormation which will create the dashboard with our widgets for us. (This can take a few minutes)
Once deployed, have a look for our new dashboard within the CloudWatch page on the AWS Console. If everything is working we can take down the stack again using cdk destroy this will delete all the resources from the stack so we wont get charged for them.

Custom Metrics time!

Now we have a basic dashboard working from the CDK, we can start to add some custom metrics. To do this we will need to create a lambda function using boto3 to access cloudwatch. We will also need to add the lamdba into our XXXX_stack.py file as it will need to be deployed at the same time as our dashboard.

Firstly we need to add a lambda folder into the project, and create a file within called custom_metric.py within the new file add the following code:

import boto3

metric_data = <The value you want the metric to display: int>
metric = {
    'MetricName': 'Instance Metrics',
    'Dimensions': [
        {
            'Name': 'MetricName',
            'Value': 'CustomNumberMetric',
        },
    ],
    'Unit': 'None',
    'Value': metric_data
}

MetricName the name of the metric within the namespace.
Name The name of the metric column.
Value the value of the metric itself.

Now we need to add the code to submit the metric to cloudwatch.

cloudwatch = boto3.client('cloudwatch', <RegionName>)
cloudwatch.put_metric_data(
    Namespace='CustomMetrics',
    MetricData=metric
)

Let's add the lambda function to the CDK!

Now we have our basic custom metric created we can add it into our stack. To do this we need to add a few other items. The CDK will try its best to create IAM roles for the lambda functions to give the access they need to function. In this tutorial we will add our own roles so our lambda can access CloudWatch.

Creating the IAM role:

Firstly we need to add another import statement aws_iam as iam inside the from aws_cdk import() import statement
Next we create the role with the following code

custom_metric_role = iam.Role(
            self, 'custom_metrics_role',
            assumed_by=iam.ServicePrincipal("lambda.amazonaws.com"),
         managed_policies=[
         iam.ManagedPolicy.from_managed_policy_arn(
             scope=self,
             id='ExeRole',                 
         managed_policy_arn='arn:aws:iam::aws:policy/service-role/AWSLambdaKinesisExecutionRole'
        )
    ]
)

custom_metric_role.add_to_policy(
    statement=iam.PolicyStatement(
        actions=[
            "cloudwatch:PutMetricData"
        ],
        resources=[
            "*"
        ]
     )
)

Note: This probably isnt the best way to create a role (I am still learning ahah!)

Adding the Lambda Function to the stack

Now we have our role we can add the lambda function into the stack.

Once again we need to add another import statement. Add aws_lambda as _lambda to the from aws_cdk import()
Now we add the code below for the lambda function

_lambda.Function(
  self, "custom_metric_lambda",
  runtime = _lambda.Runtime.PYTHON_3_7,
  code = _lambda.Code.from_asset('lambda'),
  handler = 'custom_metric.lambda_handler',
  role = custom_metric_role,
  timeout = core.Duration.seconds(10)
)

Add the metric to the dashboard

Add the following code inside the metric=[] argument of the cloudwatch dashboard function.

SingleValueWidget(
  metrics = [cw.Metric(
    metric_name = 'Instance Metrics',
    namespace = 'CustomMetrics',
    dimensions = dict(
      MetricName = 'CustomMetric',
    )
  )],
  width = 6,
  height = 3
)

That's all the coding finished!

Final Deployment!

Now all we have to do is deploy the stack.

In the terminal use cdk diff this command will run our code and will show any errors. It will also show you the differences in the code base compared to the last deployment.
If that all passes we can deploy using cdk deploy ***

Thank You!

I am by no means an expert in the topic but found the documentation really awkward to get your head around so hopefully this 'tutorial' helps.

Notes

Any comments would be greatly appreciated, I have a lot more to learn about the CDK and would love to create a sever-less app using this stack and the AWS SAM model.

This is also my very first Dev.to post, would love suggestions of good topics to look at as a student dev.