DEV Community

Cover image for Amazon CloudWatch multi data source querying
Dimple Vaghela
Dimple Vaghela

Posted on

Amazon CloudWatch multi data source querying

Amazon CloudWatch multi data source querying
Embracing a commitment to innovation and user empowerment, AWS CloudWatch continues to evolve with cutting-edge features that redefine the landscape of cloud monitoring and management. In this blog post, we delve into the latest advancements within AWS CloudWatch, showcasing how these enhancements empower users to gain deeper insights, optimize performance, and seamlessly navigate the dynamic world of cloud computing.

Introduction

Gain visibility across your hybrid and multicloud metrics in a single view with Amazon CloudWatch multi data source querying. With this feature, you can consolidate and visualize metrics from sources such as Amazon OpenSearch Service, Amazon Managed Service for Prometheus, Azure Monitor, your own custom data sources, and query those metrics in real time, increasing visibility into your application health and helping you resolve critical events faster.
As we know now CloudWatch can support multi data source querying. The source can be any of the options listed below.

Image description

Image description

Image description

Image description

You have the flexibility to query, visualise, and set alarms for all metrics, irrespective of their origin.

Image description
In this blog, we will explore the process of obtaining the source data from an S3 bucket and creating a dashboard—a practical use case that highlights the integration and visualization capabilities within the context of the discussed topic.

Image description

Step 1 Create bucket

Create bucket name cw-bucket1

Image description

Upload the csv formatted file with the first column timestamp example below.

Image description

Select the S3 bucket that you have created

Image description

In this example the file name is final.csv

Upload the final.csv file to this bucket

Finally it should look likes as below

Image description

So all good to go for CloudWatch now

Step 2 Cloudwatch multi source query initial setup

Access the CloudWatch service console by opening it. Navigate to the Metrics section and expand it.

Then, proceed to click on "All Metrics."
Image description

Image description

You will observe a display resembling the following.
Select Multi source query

Image description

Click “Cloudwatch Metrics Insights” pull dropdown menu

Image description

Select “Create and manage data sources”

Image description

It will look like the screenshot below. You can see the supported data sources.

Image description

Click on getting started.

Image description

Select S3 as the data source for the example and click next.

Image description

Enter data source name project-csv

Acknowledged to the IAM resources which automatically created with CloudFormation.

Image description

Click “Create data source” right side lower cornet.

Image description

Once you click it will run a cloudformation template to create resources.

Image description

Wait until the cloudformation template create the necessary resources.

Image description

Image description

After completing the CloudFormation template the below screen will be displayed.

Image description

Then the last step to point out the bucket and the key file.

Step 3 Point out the source as bucket and key file.

Click on “Query from Cloudwatch Metrics” link

Image description
We need to select the bucket and key

Image description

After adding the bucket name and key, it will look like below.

Image description

The final steps to query

Step 4 Cloudwatch multi source Query

Click on Graph Query

Image description

In case required adjust the time range from top

Image description

Based on your data the graph will be displayed

Image description

You can play with different kinds of graphs which suit the data.

Image description

So finally we finished the Cloudwatch group query with source as S3 Bucket.

Image description

Step 5 Final steps is clean up

Open CloudFormation console.

Note : Do not forget to check the Lambda function dashboard.

Image description

Just delete the CloudFormation template. Wait till the Cloudformation delete is completed.

Image description

Open the S3 console.
Empty the bucket cw-bucket1 and delete the bucket.

Image description

Lambda function screenshots Additional.

Image description

Image description

Conclusion

In conclusion, CloudWatch enables seamless querying of S3 functions without the need for additional code. This blog has provided insights into one of the sources for CloudWatch's multi-source query capabilities, focusing specifically on S3 integration. Stay tuned for future blog posts, where we will explore and delve into other data sources, expanding our understanding of CloudWatch's versatile querying capabilities. The overall takeaway emphasises the simplicity and efficiency with which CloudWatch handles multi-source queries, opening avenues for comprehensive monitoring and analysis across various data streams.

Reference
Read the AWS News Blog and watch one more demo for multi source in the What’s new with AWS observability and operations breakout.

Top comments (0)