Understand Amazon SSM Agent In 2 Minutes

🚀 Install SSM Agent on Ubuntu Server instances

To install SSM Agent on Ubuntu Server 20.10 STR & 20.04, 18.04, and 16.04 LTS 64-bit instances (with Snap package)

~ $:/home/ubuntu# sudo snap install amazon-ssm-agent --classic

🚀 Check SSM Agent log

~ $:/home/ubuntu# systemctl restart snap.amazon-ssm-agent.amazon-ssm-agent.service                                                                                                                                                                                                 
~ $:/home/ubuntu# tail -f /var/log/amazon/ssm/amazon-ssm-agent.log                                                                                                                                                                                                                 
        status code: 400, request id: ea74ed4f-70d4-4610-8221-ce7868c3c9fb                                                                                                                                                                                                                
2021-01-08 08:40:20 INFO [amazon-ssm-agent] [SelfUpdate] Initializing self update ...
2021-01-08 08:40:20 INFO [amazon-ssm-agent] Starting Core Agent: amazon-ssm-agent - v3.0.161.0
2021-01-08 08:40:20 INFO [amazon-ssm-agent] OS: linux, Arch: amd64
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker is not running, starting worker process
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] [WorkerProvider] Worker ssm-agent-worker (pid:11067) started
2021-01-08 08:40:22 ERROR Error adding the directory to watcher: no such file or directory
2021-01-08 08:40:22 INFO [amazon-ssm-agent] [LongRunningWorkerContainer] Monitor long running worker health every 60 seconds
2021-01-08 08:40:22 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Dial to Core Agent broadcast channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Create new startup processor
2021-01-08 08:40:22 INFO [ssm-agent-worker] Start to listen to Core Agent health channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] Start to listen to Core Agent termination channel
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Executing startup processor tasks
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: Amazon SSM Agent v3.0.161.0 is running
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsProductName: Ubuntu
2021-01-08 08:40:22 INFO [ssm-agent-worker] [StartupProcessor] Write to serial port: OsVersion: 18.04
2021-01-08 08:40:23 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/SSMAutomation/i-06f3424a03d04c66d is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-06f3424a03d04c66d
        status code: 400, request id: f0580f42-a0c8-4038-8242-46e4818a391b

• It shows that the SSM agent does not have permission on SSM action such as UpdateInstanceInformation for the instance that it relies on

2021-01-08 08:40:23 INFO [ssm-agent-worker] Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/SSMAutomation/i-06f3424a03d04c66d is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-06f3424a03d04c66d

🚀 Attach instance profile for ssm agent permission

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "AmazonSSMtoEC2",
            "Effect": "Allow",
            "Action": [
                "ssm:*",
                "ssmmessages:CreateControlChannel",
                "ssmmessages:CreateDataChannel",
                "ssmmessages:OpenControlChannel",
                "ssmmessages:OpenDataChannel",
                "ec2messages:AcknowledgeMessage",
                "ec2messages:DeleteMessage",
                "ec2messages:FailMessage",
                "ec2messages:GetEndpoint",
                "ec2messages:GetMessages",
                "ec2messages:SendReply"
            ],
            "Resource": "*"
        }
    ]
}

• Restart the ssm agent for the role affects

2021-01-08 08:44:27 INFO [ssm-agent-worker] Starting SSM Agent Worker: amazon-ssm-agent - v3.0.161.0
2021-01-08 08:44:27 INFO [ssm-agent-worker] OS: linux, Arch: amd64
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [HealthCheck] HealthCheck reporting agent health.
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting message polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [OfflineService] Starting send replies to MDS
2021-01-08 08:44:27 INFO [ssm-agent-worker] [LongRunningPluginsManager] starting long running plugin manager
2021-01-08 08:44:27 INFO [ssm-agent-worker] [LongRunningPluginsManager] there aren't any long running plugin to execute
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting message polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] Starting send replies to MDS
2021-01-08 08:44:27 INFO [ssm-agent-worker] [instanceID=i-06f3424a03d04c66d] Starting association polling
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Launching response handler
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] [EngineProcessor] Initial processing
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Initializing association scheduling service
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessagingDeliveryService] [Association] Association scheduling service initialized
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] Starting session document processing engine...
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] [EngineProcessor] Starting
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] SSM Agent is trying to setup control channel for Session Manager module.
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] agent telemetry cloudwatch metrics disabled
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] Setting up websocket for controlchannel for instance: i-06f3424a03d04c66d, requestId: 8ad8e181-fff5-4aee-81ca-4bbaf6542c3e
2021-01-08 08:44:27 INFO [ssm-agent-worker] [MessageGatewayService] listening reply.

🚀 Conclusion

• For using AWS Systems Manager Run Command, not only set IAM policy for the instance sending SSM command but also the target need IAM policy so that its SSM agent can assume EC2 resource

Mirror

• https://github.com/vumdao/ssm-agent

Read More

• Pelican-resume with docker-compose and AWS + CDK
• Using Helm Install Botkube Integrate With Slack On EKS
• Ansible AWS EC2 Dynamic Inventory Plugin
• How To List All Enabled Regions Within An AWS account
• Using AWS KMS In AWS Lambda
• Create AWS Backup Plan
• Techniques For Writing Least Privilege IAM Policies
• EKS Persistent Storage With EFS Amazon Service
• Create k8s Cronjob To Schedule Delete Expired Files
• Amazon ECR - Lifecycle Policy Rules
• Connect Postgres Database Using Lambda Function
• Using SourceIp in ALB Listener Rule
• Amazon Simple Systems Manager (SSM)
• Invalidation AWS CDN Using Boto3
• Create AWS Lambda Function Triggered By S3 Notification Event
• CI/CD Of Invalidation AWS CDN Using Gitlab Pipeline
• Create CodeDeploy
• Gitlab Pipeline With AWS Codedeploy
• Create AWS-CDK image container
• Deploy Python Lambda Functions With Container Image
• Custom CloudWatch Events
• How To Get Lastest Image Version in AWS ECR
• Multi Threading In Lambda Function

🌠 Blog · Web · Linkedin · Group · Page · Twitter 🌠

Comments

[Oleksiy Holubyev]

Hmm, I have exactly this problem, but I don't understand where I should attach the instance profile. My EC2 instance already has AmazonSSMRoleForInstanceQuickSetup role attached, which has the AmazonSSMManagedInstanceCore policy, which should have all of the listed permissions. But I still get the error mentioned above.

[🚀 Vu Dao 🚀]

Attache instance profile which provide ssm:* permisson to the EC2 instance which you got the ssm-agent error

[Oleksiy Holubyev]

I have that. But I still get the error.

[🚀 Vu Dao 🚀]

Can you show you policy and the log

[Oleksiy Holubyev]

Sure. The role in the instance profile is AmazonSSMRoleForInstancesQuickSetup, the associated policy is arn:aws:iam::aws:policy/AmazonSSMManagedInstanceCore:

{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
],
"Resource": ""
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": ""
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
}
]
}

and the error is:

Entering SSM Agent hibernate - AccessDeniedException: User: arn:aws:sts::111111111111:assumed-role/AmazonSSMRoleForInstancesQuickSetup/i-instance-id
is not authorized to perform: ssm:UpdateInstanceInformation on resource: arn:aws:ec2:eu-central-1:111111111111:instance/i-instance-id
status code: 400, request id:

[🚀 Vu Dao 🚀]

Why the resource is "" for ssm action?

[Oleksiy Holubyev]

It's just the text box interpreted the asterisk as italic formatting. The asterisk is there in the policy itself.

[Duy Nguy3n]

It solved my problem with SSM agent :)

[🚀 Vu Dao 🚀]

Glad to hear that

[Duy Nguy3n]

Thanks for sharing