Publish PM2 logs to AWS CloudWatch

Pushing your PM2 logs from an EC2 machine to AWS CloudWatch requires a few crucial steps. In this article, we'll go through each step in detail, but before we begin, it's essential to understand that EC2 logs will not automatically be pushed to CloudWatch. To facilitate this process, a CloudWatch agent needs to be installed on your EC2 instance.

To begin with, ensure that the correct IAM permissions have been set up. Here's a template for the permissions required:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents",
        "logs:DescribeLogStreams"
      ],
      "Resource": ["*"]
    }
  ]
}

Alternatively, you can use logs:*" for simplicity and add it to the existing group

To install the CloudWatch log agent, use the following command:

sudo yum install y awslogs.

Note that this command is specific to Amazon Linux, and you may need to adjust it based on your Linux distribution.

Next, update your region in /etc/awslogs/awscli.conf. By default, it points to

us-east-1:

[plugins]
cwlogs = cwlogs
[default]
region = ap-southeast-1

To specify the logs to be tracked, edit /etc/awslogs/awslogs.conf. By default, this file tracks logs from /var/log/messages. To get logs from your specific files, change the configuration. For example:

[/var/log/Your-Chosen-Name/error.log]
datetime_format = %b %d %H:%M:%S
file = /var/log/Your-Chosen-Name/error.log
buffer_duration = 5000
log_stream_name = {instance_id}
initial_position = start_of_file
log_group_name = Your-LogGroup-Name

Here, log_stream_name = {instance_id} signifies that the log stream will be named after the instance id of the EC2 instance sending the logs. The initial_position = start_of_file tells the agent to start reading from the beginning of the file. Lastly, log_group_name = Your-LogGroup-Name refers to the name of the log group on CloudWatch. If it doesn't already exist, CloudWatch will create it for you.

To send your PM2 logs to this new location, you'll need to modify the ecosystem.config.js file like so:

module.exports = {
  apps: [{
    name: "Your-App-Name",
    script: "Start-Up-File-Name",
    error_file: "/var/log/Your-Chosen-Name/error.log",
    out_file: "/var/log/Your-Chosen-Name/out.log",
    watch: true,
    env: {
      NODE_ENV: 'Your-ENV'
    }
  }]
};

Save this file and restart PM2. With these steps, your logs are now being saved in the new directory.

Finally, start the CloudWatch agent using sudo service awslogs start (or sudo systemctl start awslogsd if you're using Amazon Linux). To ensure the agent starts upon system reboot, run sudo systemctl enable awslogsd.service.

Now, login to the AWS console, navigate to CloudWatch, and check the 'logs' tab. Here, you should be able to find your log group and see your logs streaming from your EC2 instance. For more detailed instructions, check out the AWS documentation here.

Comments

[Pratap Vhatkar]

Impressive guide! Your blog post outlining the steps to send PM2 logs from an EC2 instance to AWS CloudWatch is incredibly helpful. Your clear instructions and concise explanations make a potentially complex process seem straightforward.

I particularly appreciate the template you provided for the required IAM permissions; it simplifies things greatly. Your inclusion of Linux command variations and code snippets adds a practical touch that sets your guide apart.

The way you explained the file editing process, from /etc/awslogs/awscli.conf to /etc/awslogs/awslogs.conf, shows your attention to detail. Your directions for modifying the ecosystem.config.js file are on point, making it easy to understand how to channel logs to the desired location.

You've managed to cover all the essentials while keeping the guide concise. Thanks for sharing your expertise in such a reader-friendly manner!