DEV Community

Cover image for If you are deploying the CloudWatch agent in an on-premises environment, you should also install the Systems Manager Agent.
AWS Community Builders profile image NaoyukiFujita
NaoyukiFujita for AWS Community Builders

Posted on

If you are deploying the CloudWatch agent in an on-premises environment, you should also install the Systems Manager Agent.

#aws #cloudwatch #systemsmanager

Introduction

Last year, after support for Windows 10 ended, I converted a PC that couldn't be upgraded to Windows 11 into a Linux machine (running Ubuntu 24.04). I wasn't sure how to put it to use, but I decided to try setting up a Linux PC management environment as a way to learn about AWS Systems Manager (SSM) and Amazon CloudWatch.

One issue I encountered was that the standard procedure for configuring authentication for the CloudWatch agent on an on-premises machine requires specifying an IAM user's access key and secret key.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-premise.html

Even though this was just for personal testing, I hesitated because I was concerned about the risk of an AWS environment compromise resulting from an exposed access key. However, I discovered that if the device is registered as a managed node using the Systems Manager "Hybrid Activations" feature, it is possible to configure authentication for the CloudWatch agent without using access keys.

In this article, I will cover the installation of the SSM Agent and CloudWatch agent, as well as the authentication configuration for the CloudWatch agent.

This article reflects the author's personal views. As the results are based on personal use, please treat them only as a reference. I would also appreciate it if you could point out any errors or omissions in the content.

Configuration

The setup is as shown in the diagram below. You need to install two components on the Linux PC: the CloudWatch agent and the Systems Manager Agent.

Agent Overview

AWS Systems Manager Agent

This agent enables the management, updating, and configuration of devices (EC2 instances, on-premises servers, and virtual machines) via AWS Systems Manager (SSM).

It comes pre-installed on some AMIs. However, for on-premises devices like the ones in this scenario, manual installation is required.

https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html

To bring an on-premises device under SSM management, you must register it as a managed node using hybrid activation.

https://docs.aws.amazon.com/systems-manager/latest/userguide/activations.html

CloudWatch Agent

This agent collects metrics, logs, and traces from Amazon EC2 instances, on-premises servers, and containerized applications. It enables you to view metrics in CloudWatch and monitor logs using CloudWatch Logs.

The URL below details the metrics that can be collected, such as memory utilization, CPU utilization, and network interface traffic volume.

https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/metrics-collected-by-CloudWatch-agent.html

Reasons for Using SSM Agent Temporary Credentials for the CloudWatch Agent

  • To prevent AWS environment compromise resulting from access key leakage
  • To eliminate the burden of manual key rotation

These are the two primary reasons.

Even though the goal is to use the CloudWatch agent, you generally do not want to store long-term credentials—such as access keys—on the device itself. If they were to leak, it could lead to a compromise of your AWS environment.

The SSM Agent can assume the IAM role associated via hybrid activation as temporary credentials. This allows it to utilize the permissions defined in the IAM policies attached to that role. In other words, if an IAM policy granting permission to upload (Put) logs via the CloudWatch agent is attached, you can upload logs from an on-premises environment.

Installing the SSM Agent

I installed the SSM Agent by following the instructions below.

Since permissions are required to connect to SSM from an on-premises environment, I created a hybrid activation and then registered the instance as an SSM managed node.

https://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-hybrid-multicloud.html

Configuring Hybrid Activation

I executed the commands described in this section using AWS CloudShell.

① Create the trust policy JSON file shown below. You can choose any filename; I used TrustPolicy.json.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Service": "ssm.amazonaws.com"
      },
      "Action": "sts:AssumeRole"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

② Create an IAM role by specifying the trust policy JSON file.

aws iam create-role \
--role-name IAMRoleName \
--assume-role-policy-document file://SSMService-Trust.json
Enter fullscreen mode Exit fullscreen mode

③ Attach the AmazonSSMManagedInstanceCore IAM policy (required for basic SSM functionality) and the CloudWatchAgentServerPolicy IAM policy (required for the CloudWatch agent to send metrics and logs).

# Basic SSM functionality
aws iam attach-role-policy \
--role-name IAMRoleName \
--policy-arn arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore

# For the CloudWatch agent to send metrics and logs
aws iam attach-role-policy \
--role-name IAMRoleName \
--policy-arn arn:aws:iam::aws:policy/CloudWatchAgentServerPolicy
Enter fullscreen mode Exit fullscreen mode

If you are not using the CloudWatch agent, the AmazonSSMManagedInstanceCore IAM policy alone is sufficient; however, to achieve the goal of sending CloudWatch agent metrics and logs without using access keys, we configure the CloudWatchAgentServerPolicy here.

④ Create a hybrid activation. I used the following settings:

  • Management instance name: "instance-name"
  • IAM role name to specify: IAMRoleName
  • Registration limit for instances: 1
  • Region for hybrid activation: ap-northeast-1 (Tokyo Region)
  • Activation expiration date: YYYY-MM-DD 0:00:00 (YYYY = year, MM = month, DD = day) 
aws ssm create-activation \
--default-instance-name "instance-name" \
--iam-role IAMRoleName \
--registration-limit 1 \
--region ap-northeast-1 \
--expiration-date "YYYY-MM-DDT00:00:00"
Enter fullscreen mode Exit fullscreen mode

⑤ Make a note of the Activation ID and Activation Code output after executing the command above.

{
    "ActivationId": "<Activation Id>"
    "Activation Code": "<Activation Code>"
}
Enter fullscreen mode Exit fullscreen mode

SSM Agent Installation and Managed Node Registration

The commands in this section are to be executed on the device where SSM Agent is being installed.

① Download ssm-setup-cli.
② Run ssm-setup-cli, specifying the Activation ID, Activation Code, and the region for registration.

mkdir /tmp/ssm
curl https://amazon-ssm-ap-northeast-1.s3.ap-northeast-1.amazonaws.com/latest/debian_amd64/ssm-setup-cli -o /tmp/ssm/ssm-setup-cli
sudo chmod +x /tmp/ssm/ssm-setup-cli
sudo /tmp/ssm/ssm-setup-cli -register -activation-code "<Activation Code>" -activation-id "<Activation Id>" -region "ap-northeast-1"
Enter fullscreen mode Exit fullscreen mode

③ Upon success, a message like the following will be displayed.

(Omitted)
2026-10-08 02:12:32 INFO Starting agent using SnapSystemctl service manager
2026-10-08 02:12:32 INFO Agent is running
2026-10-08 02:12:32 INFO Successfully started agent, reloading registration info
2026-10-08 02:12:32 INFO Successfully registered the instance with AWS SSM using Managed instance-id: mi-xxxxxxxxxxxxxxxxx
2026-10-08 02:12:37 INFO Process Path: /snap/amazon-ssm-agent/xxxxx/amazon-ssm-agent
2026-10-08 02:12:37 INFO Agent process count: 1
2026-10-08 02:12:37 INFO Agent registration completed
Enter fullscreen mode Exit fullscreen mode

④ Run the following command and verify that Active: active (running) is displayed (confirming that the SSM Agent is running).

sudo systemctl status snap.amazon-ssm-agent.amazon-ssm-agent.service
Enter fullscreen mode Exit fullscreen mode

Private Key Automatic Rotation Settings

The maximum validity period (in days) for the on-premises private key is currently set to 0 days (no rotation); change this maximum validity period to 1 day.

① Create a backup of /etc/amazon/ssm/amazon-ssm-agent.json.

sudo cp /etc/amazon/ssm/amazon-ssm-agent.json /etc/amazon/ssm/amazon-ssm-agent.json.bak
Enter fullscreen mode Exit fullscreen mode

② Copy the entire contents of the following JSON:

https://github.com/aws/amazon-ssm-agent/blob/mainline/amazon-ssm-agent.json.template

③ Clear the contents of /etc/amazon/ssm/amazon-ssm-agent.json completely, then paste the JSON copied in step ②.

④ Change the value of KeyAutoRotateDays to "1" (rotation every day).

⑤ Save and close the file.

This completes the installation of the SSM Agent and the registration of the managed node.

CloudWatch Agent Installation and Configuration

I installed the agent and configured the credentials by referring to the following:

*Reference for agent installation
https://docs.aws.amazon.com/ja_jp/AmazonCloudWatch/latest/monitoring/install-CloudWatch-Agent-on-premise.html

*Reference for authentication information settings (AWS re:Post article)

https://repost.aws/ja/knowledge-center/cloudwatch-on-premises-temp-credentials

The commands described in this section were executed by the author using AWS CloudShell.

Agent Installation

① Execute the following using SSM Run Command.

aws ssm send-command \
  --document-name "AWS-ConfigureAWSPackage" \
  --targets "Key=instanceids,Values=mi-xxxxxxxxxxxxxxxxx" \
  --parameters '{"action":["Install"],"name":["AmazonCloudWatchAgent"]}' \
  --region ap-northeast-1
Enter fullscreen mode Exit fullscreen mode

② It will be displayed as follows.

{
    "Command": {
        "CommandId": "XXXXXXXX-XXXX-XXXX-xxxx-xxxxxxxxxxxx",
        "DocumentName": "AWS-ConfigureAWSPackage",
        "DocumentVersion": "$DEFAULT",
        "Comment": "",
        "ExpiresAfter": "2026-10-08T15:47:53.058000+00:00",
        "Parameters": {
            "action": [
                "Install"
            ],
            "additionalArguments": [
                "{}"
            ],
            "name": [
                "AmazonCloudWatchAgent"
            ]
        },
        "InstanceIds": [],
        "Targets": [
            {
                "Key": "instanceids",
                "Values": [
                    "mi-XXXXXXXXXXXXXXXXX"
                ]
            }
        ],
        "RequestedDateTime": "2026-10-08T12:47:53.058000+00:00",
        "Status": "Pending",
        "StatusDetails": "Pending",
        "OutputS3Region": "ap-northeast-1",
(Omitted)
Enter fullscreen mode Exit fullscreen mode

CloudWatch Agent Credential Configuration

The commands described in this section were executed after connecting to the Linux PC using Session Manager.

① Edit common-config.toml. On a Linux machine, this file is typically located under /opt/aws/amazon-cloudwatch-agent/etc/.

sudo nano /opt/aws/amazon-cloudwatch-agent/etc/common-config.toml
Enter fullscreen mode Exit fullscreen mode

② If [credentials] is commented out, uncomment it and specify the credential profile and credential file.

For non-EC2 instances, SSM Agent credentials are stored in /root/.aws/credentials.
*For details regarding credentials, please refer to "SSM Agent Credential Precedence" below.

https://docs.aws.amazon.com/ja_jp/systems-manager/latest/userguide/ssm-agent-technical-details.html#credentials-precedence

# [credentials]
#    shared_credential_profile = "{profile_name}"
#    shared_credential_file= "{file_name}"
Enter fullscreen mode Exit fullscreen mode 
[credentials]
  shared_credential_profile = "default"
  shared_credential_file = "/root/.aws/credentials"
Enter fullscreen mode Exit fullscreen mode

③ Set the region in /root/.aws/config.

sudo mkdir -p /root/.aws
sudo nano /root/.aws/config
Enter fullscreen mode Exit fullscreen mode

④ Create the CloudWatch agent configuration file. Save it in JSON format at /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json.

The following is an example configuration:

- Metrics
  - CPU usage
  - Memory usage
  - Disk usage
- CloudWatch Logs
  - Target for transmission: "/var/log/syslog"
  - CloudWatch Logs log group: "ubuntu-pc-syslog"
  - Sent to CloudWatch Logs stream: {hostname}
Enter fullscreen mode Exit fullscreen mode 
{
  "agent": {
    "metrics_collection_interval": 60,
    "run_as_user": "root"
  },
  "metrics": {
    "metrics_collected": {
      "cpu": {
        "measurement": ["cpu_usage_idle", "cpu_usage_user", "cpu_usage_system"],
        "totalcpu": true
      },
      "mem": {
        "measurement": ["mem_used_percent"]
      },
      "disk": {
        "measurement": ["used_percent"],
        "resources": ["/"]
      }
    }
  },
  "logs": {
    "logs_collected": {
      "files": {
        "collect_list": [
          {
            "file_path": "/var/log/syslog",
            "log_group_name": "ubuntu-pc-syslog-all",
            "log_stream_name": "{hostname}"
          }
        ]
      }
    }
  }
}
Enter fullscreen mode Exit fullscreen mode

⑤ Start the CloudWatch agent by specifying the CloudWatch agent configuration JSON file you created. Success is indicated when "Configuration validation succeeded" is displayed.

Meaning of each option:
-a fetch-config: Load configuration
-m onPremise: On-premises mode
-s: Start the agent after loading
-c file:...: Path to the configuration file

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl \
  -a fetch-config \
  -m onPremise \
  -s \
  -c file:/opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.json
Enter fullscreen mode Exit fullscreen mode

⑥ Execute the following command and confirm that "status": "running" is displayed.

sudo /opt/aws/amazon-cloudwatch-agent/bin/amazon-cloudwatch-agent-ctl -a status
Enter fullscreen mode Exit fullscreen mode

This completes the installation of the CloudWatch agent and the configuration of credentials.

Verification

Confirm registration as a managed node

Output the node information for the corresponding managed node.

aws ssm describe-instance-information \
  --filters "Key=ResourceType,Values=ManagedInstance" \
                 "Key=PingStatus,Values=Online" \
  --query "InstanceInformationList[*].[InstanceId,PingStatus,AgentVersion,PlatformType,ComputerName,ResourceType]" \
  --output table \
  --region ap-northeast-1
Enter fullscreen mode Exit fullscreen mode 
--------------------------------------------------------------------------------------------
|                                DescribeInstanceInformation                               |
+-----------------------+---------+-------------+--------+-------------+-------------------+
|  mi-xxxxxxxxxxxxxxxx |  Online |  3.3.4793.0 |  Linux |  XXXXXXXXXX |  ManagedInstance  |
+-----------------------+---------+-------------+--------+-------------+-------------------+
Enter fullscreen mode Exit fullscreen mode

Retrieving Metrics

Verify that the metrics configured for the CloudWatch agent are being retrieved in CloudWatch Metrics.

aws cloudwatch get-metric-statistics \
  --namespace "CWAgent" \
  --metric-name "mem_used_percent" \
  --dimensions Name=host,Value=XXXXXXXXXX \
  --start-time $(date -u -d '1 hour ago' +%Y-%m-%dT%H:%M:%SZ) \
  --end-time $(date -u +%Y-%m-%dT%H:%M:%SZ) \
  --period 300 \
  --statistics Average \
  --region ap-northeast-1
  --output table
Enter fullscreen mode Exit fullscreen mode 
------------------------------------------------------------------
|                       GetMetricStatistics                      |
+------------------+---------------------------------------------+
|  Label           |  mem_used_percent                           |
+------------------+---------------------------------------------+
||                          Datapoints                          ||
|+---------------------+-----------------------------+----------+|
||       Average       |          Timestamp          |  Unit    ||
|+---------------------+-----------------------------+----------+|
||  25.064903122713268 |  2026-10-10T06:08:00+00:00  |  Percent ||
||  24.864319857743233 |  2026-10-10T06:18:00+00:00  |  Percent ||
||  25.052294740787254 |  2026-10-10T06:13:00+00:00  |  Percent ||
|+---------------------+-----------------------------+----------+|
Enter fullscreen mode Exit fullscreen mode

Retrieving Logs with CloudWatch Logs

Verify that the logs configured for the CloudWatch agent are being collected in CloudWatch Logs.

aws logs get-log-events \
  --log-group-name "LogGroupName" \
  --log-stream-name "StreamName" \
  --limit 50 \
  --query "events[*].[timestamp,message]" \
  --output table \
  --region ap-northeast-1
Enter fullscreen mode Exit fullscreen mode

Points to Consider

By using SSM Agent's temporary credentials for the CloudWatch agent, you eliminate the need to generate and store access keys on your PC or manually rotate keys and update local credentials. However, certain risks remain:

  • Compromise of the credentials on the device where the agent is installed
  • Theft of the device (in the case of a PC)

Additionally, since SSM Agent's temporary credentials are used to send metrics and logs to CloudWatch, you must pay close attention to the IAM policies attached to the hybrid activation IAM role. For example, if an IAM policy allowing access to S3 objects is attached, a stolen device could potentially be used to access objects in your S3 buckets without authorization.

  • Ensure IAM policies attached to the IAM role adhere to the principle of least privilege
  • Prevent device loss
  • Manage device accounts

You should consider these countermeasures as well.

Conclusion

In this article, I covered how to install the SSM Agent and CloudWatch agent, as well as how to configure authentication for the CloudWatch agent.

Using access keys and secret access keys—even just for the CloudWatch agent—entails operational overhead and the risk of credential leakage. If you are installing the CloudWatch agent on an on-premises device, installing the SSM Agent alongside it to convert the device into a managed node is likely the better, more secure, and less labor-intensive approach.

There may be specific reasons why the official documentation prioritizes procedures using access keys; however, given that the use of access keys is no longer recommended, I believe it is better to focus on procedures that include the SSM Agent whenever possible.

I was able to successfully manage my personal PC via SSM and monitor resources and logs using CloudWatch, so I look forward to utilizing these tools for future learning. Since CloudWatch Logs added support for journald logs on August 28, 2026, I plan to try that out and write another article about it.

https://aws.amazon.com/jp/about-aws/whats-new/2026/08/amazon-cloudwatch-agent-journald/

I hope this serves as a helpful reference for those about to start managing on-premises environments.

Thank you for reading to the end!

Top comments (0)