Amazon Web Services(AWS) reports some good metrics on the console by default, like CPU, but some key metrics like memory usage or disk space are missing; these are important to monitor to ensure instance up-time and health.
In this post we'll look at how we can use CloudWatch to monitor these extended metrics, allowing you to build reports, dashboards, and alerts.
📘 NOTE
Before we begin, note that standard Amazon CloudWatch usage charges will be applicable for these scripts. For more information, see the Amazon CloudWatch pricing page.
✋ For simplicity, let's assume that we are using EC2 with Amazon Linux operating system. For other operating systems, you can use respective commands to achieve the results.
Creating an IAM role to access the metrics
In order to pass metrics data from EC2 to AWS Cloudwatch, we will have to create a user with the following IAM role access:
cloudwatch:PutMetricDatacloudwatch:GetMetricStatisticscloudwatch:ListMetricsec2:DescribeTags
Got confused? Just create a policy (say - cloudwatch-ec2-access) with the above permissions. After that, create a user (cloudwatch-stats-user) and attach the created policy(cloudwatch-ec2-access) to the user. Also, store the generated AWSAccessKeyId and AWSSecretKey of this user which will be required at a later stage.
SSH to EC2
SSH to your EC2 instance and perform the following steps:
1. Create a script folder
I generally prefer to keep everything in a separate folder for clarity, though not mandatory. As we are using Amazon Linux 2 AMI, steps will look similar to this:
# current folder /home/ec2-user
$ mkdir cloudwatch_logs
$ cd cloudwatch_logs
2. Install the required packages
In order to be able to run the AWS scripts, we will have to install some packages. Also, note that the command may change based on your operating system. Check this link for more information.
sudo yum install -y perl-Switch perl-DateTime perl-Sys-Syslog perl-LWP-Protocol-https perl-Digest-SHA.x86_64
3. Download the scripts from AWS
Now that we have installed all the packages, we need to download the Perl scripts provided by AWS.
curl https://aws-cloudwatch.s3.amazonaws.com/downloads/CloudWatchMonitoringScripts-1.2.2.zip -O
4. Unzip the scripts
unzip CloudWatchMonitoringScripts-1.2.2.zip
# remove the zip
rm CloudWatchMonitoringScripts-1.2.2.zip
# move to the unzipped folder
cd aws-scripts-mon
The package for the monitoring scripts contains the following files:
-
CloudWatchClient.pm– Shared Perl module that simplifies calling Amazon CloudWatch from other scripts. -
mon-put-instance-data.pl– Collects system metrics on an Amazon EC2 instance (memory, swap, disk space utilization) and sends them to Amazon CloudWatch. -
mon-get-instance-stats.pl– Queries Amazon CloudWatch and displays the most recent utilization statistics for the EC2 instance on which this script is executed. -
awscreds.template– File template for AWS credentials that stores your access key ID and secret access key. -
LICENSE.txt– Text file containing the Apache 2.0 license. -
NOTICE.txt– Copyright notice.
5. Add Access ID and Secret Key of cloudwatch user (cloudwatch-stats-user)
As mentioned in the earlier step, amazon provides a template file awscreds.template which can be used to create a conf file to store AWS credentials.
# creates a conf file from the template. Make sure the filename is as is
cp awscreds.template awscreds.conf
Next, add your AWS Access ID and secret key in this file and save it.
6. Verify if statistics are captured correctly
We will now verify if everything is in place and works smoothly with following command:
# change the paths according to your folder structure
/home/ec2-user/cloudwatch_logs/aws-scripts-mon/mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-util --mem-used --mem-avail --disk-space-util --disk-space-avail --disk-path=/ --verify --verbose
7. Final Step
Congratulations! 👏 You have successfully configured Cloudwatch metrics on your EC2 instance. Now one last step is to add it to cron so it can send metrics after every 5 mins.
# Open the crontab file
crontab -e
# Add the following line and save it
# Cloudwatch Monitoring Metrics (AWS)
*/5 * * * * /home/ec2-user/cloudwatch_logs/aws-scripts-mon/mon-put-instance-data.pl --mem-used-incl-cache-buff --mem-util --mem-used --mem-avail --disk-space-util --disk-space-avail --disk-path=/ --from-cron
Where to find these metrics on AWS?
In your AWS console, go to Cloudwatch service.
All your metrics are available in Metrics menu. A new Custom Namespaces will be added in your metrics view once Cloudwatch starts receiving metrics from the scripts.
Click on it, and you will get 2 options as depicted in the image below:
-
Filesystem, InstanceId, MountPath- All your EC2 disk metrics will be available inside this option. -
InstanceId- All your EC2 memory metrics will be available inside this option.
Points to Remember
- Whenever you reboot or allocate/deallocate disk space, your EC2 disk filesystem path may change and hence you might require to reconfigure any alarms or dashboards you had created for monitoring EC2.
- If you have launched an EC2 instance from an AMI which had these metrics configured, then the above setup is already present. Just clear the cache via command:
rm /var/tmp/aws-mon/instance-id - AWS has recently launched
CloudWatch Agentto collect both system metrics and log files from Amazon EC2 instances. Hence, it is recommended to useCloudWatch Agentto collect metrics and logs in place of these monitoring scripts. However, there might be certain cases where you would require to use these monitoring scripts and this post might help you to configure it.
I hope you enjoyed the post. See ya! until my next post 😋
Top comments (2)
I think this is the old way, the recommended way by AWS is to use the unified CloudWatch Agent which can be installed via Systems Managers Run Commands.
Its recommended but it can be a bit of a pain to hunt down the step by step instructions. I've been meaning to tutorialize the steps at some point.
You are right Andrew. That's the reason of mentioning the same in the last point in
Points to remembersection. And yeah, AWS documentation is sometimes a bit of a pain to understand and hence wrote this post to easily understand the steps to get started.P.S. Your Udemy tutorial really helped me boost my knowledge of AWS when I had just started. Keep sharing! 😅😊