How to Send Log Files to AWS CloudWatch Logs

AWS CloudWatch allows you to collect logs from your AWS EC2 instances. Files such as the audit, access and error logs that are commonly found on web servers. Or any /var/log file. This is specially useful if you have a instances behind a load balancer. Rather than connecting to each instance and manually searching the logs with grep, CloudWatch centralizes the logs into one log stream, allowing you to search all your log files from one place.

For example, we have a few EC2 instances behind a Load Balancer. we send the contents of the log to CloudWatch. Even with just a few servers it’s much easier than logging into each one individually and searching each file with grep commands. Our User Journey infrastructure has over 100 servers and would be practically impossible to manage through the command line alone. If an issue occurs we’re able to see all the logs in the AWS Console without wasting time logging onto everything.

CloudWatch set-up on EC2-instance , you need to complete the following:

Create a new IAM role (one time only)
Attach the IAM role to an EC2 instance
Install and configure the awslogs service in EC2-Machine.
View logs in CloudWatch Logs groups.

1. Create a New IAM Role

To allow an EC2 instance to communicate with CloudWatch, you first need to create an IAM Role. You only need to do this once.

Open AWS Console and open the IAM console. .
From the AWS menu, select Roles and then click the Create role button. Choose the service that will use this role, select EC2 and click the Next: Permissions button:
Search for the CloudWatchFullAccess, check the checkbox and click Next:
Enter a Role name (e.g. Cloudwatchlogs). Then click Create role.

2. Attach the IAM Role

To attach the IAM Role to the EC2 instance, you can either do it through the AWS console.

Using the AWS Console Go to the EC2 Dashboard, select Instances from the menu and check the checkbox next to the EC2 instance you want to stream the logs from. To attach the IAM Role, click the Actions dropdown and select Security > Attach/Replace IAM Role:

Search for and select the IAM role created above (e.g. CloudWatchlogs), then click Apply to attach the IAM role:

Verify IAM Role in Security

3. Install AWS logs service in EC2-Instance

ssh ec2-instace and install awslog service
Edit the /etc/awslogs/awslogs.conf file to configure the logs to track

[ec2-user@ip-172-31-95-1 ~]$ sudo vim /etc/awslogs/awscli.conf
[plugins]
cwlogs = cwlogs
[default]
region = us-east-1

Start the awslogs service.

[ec2-user@ip-172-31-95-1 ~]$ sudo  systemctl enable --now awslogsd.service

4. View logs in CloudWatch Logs groups.

Once the log file you are watching has data written to it, you’ll be able to find it in CloudWatch. Go to the CloudWatch Overview and select Logs from the menu. You should see the /var/log/message.
Click on the log group name to see the log streams. Each log stream uses the EC2 instance ID, so you know which EC2 instance logged the data:
To search the logs, click the Search Log Group button. In the filter text box, enter a search term to search all your log files in one go: