Deploying A Metrics Aggregation Solution On AWS

Log analytics is the process of ingesting streaming log data, aggregating that data, and persisting the aggregated data so that it can be analyzed and visualized.
It is a common big data use case that allows you to analyze log data from websites, mobile devices, servers, sensors, and more for a wide variety of applications such as digital
marketing, application monitoring, fraud detection, ad tech, games, and IoT.

Metrics on the other hand, are measures of quantitative assessment commonly used for comparing, and tracking performance or production.

In this project, we will create a complete end-to-end metrics collection system that integrates several AWS services.

Our log analytics architecture is similar to the one above from AWS docs.

We have a 3-tier application deployed in our VPC. The web servers are Amazon Elastic Compute Cloud (Amazon EC2) instances deployed in an auto-scaling group.

Here is how it works:

1. The CloudWatch agent is automatically installed on the application servers.
2. The CloudWatch agent will collect logs and metrics and deliver them to CloudWatch.
3. CloudWatch Metrics continuously forwards Metrics to an Amazon Kinesis Data Firehose delivery stream.
4. Amazon Kinesis Data Firehose writes each metric record to Amazon Simple Storage Service (Amazon S3) for durable storage of the raw log data.

Here are the AWS services involved in this solution:

• CloudWatch
• Kinesis Data Firehose
• Simple Storage Service (S3)

It is important to note that while our application is hosted on EC2 instances, our log analytics solution is completely serverless.

Step 1: Setup The 3-tier Application

I have already designed a 3-tier appplication ready to be deployed to AWS using terraform. To learn more about 3-tier applications and how you can design yours, read this article.

The terraform code used to automatically deploy this architecture is on my GitHub page, so to set this up, all we have to do is clone this GitHub repository.
Once you have cloned it, navigate to the terraform directory and run terrform init, then run terraform plan. After that, run terraform apply -auto-approve.
Note: You must configure your own backend in the backend.tf file.

Terraform Init

Terraform Plan

Terraform Apply

Our instance launch template already has cloudwatch agent instaled and running. The instance role has also been modified to include the "CloudWatchAgentServerPolicy" permissions.

Note that it might take around 10 to 15 minutes for terraform to be fully provision the resources and an additional 5 minutes after provisioning is complete before the site becomes active.

The load alancer DNS name will be appear on the terminal after the terraform apply successfully runs.
Visit the load balancer DNS name on your web browser to view the web application.

Once you register/login, you should see something like this:

Step 2: Create A Kinesis Firehose Delivery Stream

Here, we will be creating a kinesis Firehose Delivery stream and configure Cloudwatch to continually send our analytics data into this stream. CloudWatch is the data producer that will send our data into Firehose delivery streams. To learn more about kinesis, checkout my blog post on getting kinetic with Amazon Kinesis.

• On the kinesis Data Firehose console, click on "Create delivery stream".
• Under "Source", select "Direct PUT".
• Under "Destination", select "S3".

• Under "Destination settings", click on "Create bucket".
• Make sure your bucket name is unique.
• Leave every other setting as default and click on "Create bucket".
• Back on the Firehose console, select the bucket you just created.
• You can optionally define an S3 bucket error output prefix.

• Click on "create delivery stream".

Step 3: Create A CloudWatch Metric Stream

Here, we have to configure CloudWatch to stream our aplication metrics to the Firehose delivery stream we created in step 2.

• On the CloudWatch Console, click on "Metrics".
• Click on "Create Metric Stream".

• Choose "Custom setup with firehose", also select the firehose stream we created earlier on.

• Scroll down to 'metrics to be streamed".
• Choose "Select metrics" > "Include".
• Choose the "CWAgent".

Note that "CWAgent" will only be available if you have successfully deployed the 3-tier application in your AWS environment.

• Under "select or type metrics", choose "CWAgent: All metrics names".
• Give your stream a name and click on "Create metric stream".

Now we have created a complete end-to-end solution for aggregation, collection and storage of our application metrics.

Step 4: Monitor Metrics Collection

You will have to periodically monitor metrics collection and the behaviour of both the CloudWatch Stream and Firehose delivery stream.

1. To get insights into the Firehose delivery stream:
2. Visit the Firehose console, go to monitoring to view Firehose metrics. These metrics will give you insight into how your solution is performing.

1. To get insights into the CloudWatch Stream:
2. Visit the CloudWatch Console
3. Go to "Metrics" > "Streams" and select your stream. The dashboard shows you performance insights.

Finally, you can check your S3 bucket to see the aggregated logs.

Here is mine: