Setup business metrics with Amazon CloudWatch and AWS Lambda at lower cost

Any good business runs with goals and key performance indicators (KPIs) that are monitored and managed. These can be user signups, or Subscriptions sale trend, or customer churn. Any Cloud-born or cloud-native business is no exception to this.

This post provides how to guide for creating business metrics using Amazon CloudWatch and AWS Lambda services.

Readers, familiar with EMF format for CloudWatch, can jump to the Implementation section.

Introduction

Before we dive deeper, let's have a tweet size introduction to AWS CloudWatch and AWS Lambda service.

Amazon CloudWatch helps to observe and monitor AWS resources and applications in the cloud and on premises.

AWS Lambda is a serverless, event-driven compute service that lets you run code for any type of application or backend service without provisioning or managing servers. AWS Lambda helps to focus on delivering customer value.

Any AWS service publish metrics in Amazon CloudWatch. Amazon CloudWatch can also publish your own application specific metrics. These metrics are called as custom metrics. The number of third-party API calls, or the count of status codes returned by an API, etc are a few examples. Based on these metrics, you can build your own dashboards or alert notifications etc.

Ways to add Amazon CloudWatch custom metrics

PutMetricData API

Amazon CloudWatch provides an API through which you can put the custom metrics. But, sending the API request is a synchronous process. For high-volume applications, you have to batch up the requests for CloudWatch API – "PutMetricData" API. Otherwise, your requests will be throttled by CloudWatch API. CloudWatch API usage for adding custom metrics also incurs cost.

Metric Log Filters

Metric log filters are another way to create CloudWatch custom metrics. The Metric log filters can search and filter data points needed to create metrics from CloudWatch log groups. It's a better option as compared to using CloudWatch API since it's asynchronous. The drawback of this method is, every time you need to create a new metric, you have to create the metric log filters.

EMF For CloudWatch

A third option is using Amazon CloudWatch specification for Embedded Metric Format (EMF). It's again an asynchronous way to create custom metrics. But with EMF format, there are no incurring charges for customer. There are no prerequisites as previous method of log filters.

Implementation

This walkthrough assumes familiarity with AWS Lambda and AWS CloudWatch services. It also assumes that you can deploy AWS lambda in your own AWS account.

The implementation uses AWS Lambda and Amazon CloudWatch. But, you can use Amazon CloudWatch EMF format with any other AWS compute services such as Amazon ECS, Amazon EKS and AWS Fargate.

Let's create AWS Lambda function to log custom metrics. In our example, below, the lambda function orderservice represents a fictitious order service. In this service, we would be creating metrics for average order value amount trend.

• For order service lambda function, first create a directory and initialize npm.

 mkdir orderservice && cd orderservice && npm init -y

• Now, install aws-embedded-metrics library in the orderservice project directory. aws-embedded-metrics library handles logging custom metrics in Amazon CloudWatch EMF format.

  npm install aws-embedded-metrics

This will create a node_modules directory with the required dependencies.

• Once you have installed the dependencies, create a new file in the project directory and name it index.js. In index.js, export the function as shown below.

const { metricScope } = require("aws-embedded-metrics");

const orderMetrics = metricScope(metrics =>
  async () => {
    // ...
  });

exports.handler = orderMetrics;

• Now, within the function, you have to create your own metric namespace and select the dimensions for the metrics. For high cardinality values, you can use property rather than dimensions. So, complete orderMetrics function is as follows

const { metricScope } = require('aws-embedded-metrics');

const orderMetrics = metricScope(metrics => async event => {
  //define the namespace for custom metrics 
  metrics.setNamespace('OrderManagement');

  //set the dimentions 
  metrics.putDimensions({ Service: 'OrderService', CustomerId:'26102022' });

  // write the metric value to cloudwatch
  metrics.putMetric('Avg. Order Value',Math.floor(Math.random() * 100) , 'None');

  // add context as properties to build cloudwatch log insights queries
  metrics.setProperty('AccountId', '123456789012');
  metrics.setProperty('RequestId','45b7d3b0-ca99-44a2-93ad-05e29c9d40e6');
  metrics.setProperty('OrderId', '7b125931-7898-4eab-bf16-78b8dc87707d');
  metrics.setProperty('Payload', {
    sampleTime: new Date(),
    customerId: 26102022,
  });

});

exports.handler = orderMetrics;

Note that, putMetric parameter value is randomly generated for demo purposes. You have to replace it with your own metric value.

• Now, you need to compress all the files in the project directory and upload the ZIP file to Lambda console.

zip -r ../orderservice.zip .

• After uploading the orderservice.zip to lambda function, you can test the lambda with test event. Since, we have randomized the putMetric input value, we are expected to see the logs having the custom metric. There can be a couple of seconds of lag for reflecting values in Amazon CloudWatch metrics page and dashboard.

• To simulate real world traffic, you have to invoke orderservice lambda several times. You can use lambda function's Test feature with sample event to simulate invocations.

• Now, you will see the ordermanagement metric as shown below under CloudWatch Metrics.

• Clicking on the OrderManagement metric, you can see CustomerId,Service,ServiceType and ServiceName metric. After small touch ups on the graph, the final output of custom metric looks as below.

• You can consider this business metric of Avg. Order Value as any other Amazon CloudWatch metric. Furthermore, you can create CloudWatch alerts based on CloudWatch custom metrics, for example – set up an alert when we receive an order of more than USD 50 million etc.

The same business metrics can be utilized to create CloudWatch dashboards to better serve your customers.

In summary, EMF format for Amazon CloudWatch is a great feature which can be utilized for creating the custom business metrics. Many customers have enjoyed up to 65% of cost reduction with using EMF format for Amazon CloudWatch, among other benefits. What you are waiting for? Now Go Build!