Access EC2 Instances Privately Using AWS Systems Manager

Hi Everyone,

In this article, we will learn how to access AWS EC2 instances using AWS Systems Manager. This is quite useful,

1. when you don't want to expose your instance to public internet via a public ip.
2. when you want to manage a large number of instances without sharing ssh keys.
3. run commands in many instances without logging in to them.

but this does come with a few prerequisites.

1. SSM agent should be up and running to connect to this instance (SSM agent comes with Amazon Linux and Ubuntu amis by default).
2. The instance should have access to the public internet (or access to SSM endpoints via AWS private endpoints).
3. The instance should have a role with the required permissions.

to continue with this post I will assume that you have access to an AWS account and have a basic idea of AWS.

keep in mind that if you are not in the AWS free tier this will incure you a cost.

With the above in mind let's get started.

First, we will create an aws instance role which is required to grant permissions to the aws instance so that the ssm agent in the ec2 instance can connect with the aws systems manager.

navigate into aws iam roles and click create a role as below.

in next page add trusted entity EC2 and select EC2 Role for AWS Systems Manager

next in permissions since we chose EC2 Role for AWS Systems Manager it will automatically set AmazonSSMManagedInstanceCore policy which is the required permission for this use case.

after hitting next add a name for your role and a description.

and review whether all settings are mentioned below and click Create role.

and we are done with iam role for our instance. now let's create an ec2 instance.

navigate to the ec2 menu and click launch instance

I will add an instance name as shown below but this is optional.

next select ami for this instance I will choose the default Amazon Linux 2023 image for this purpose.

next, specify the instance type and key pair for login. for this example, I will add a keypair since it will be useful if there is an issue with agent connectivity although we are not using it in this tutorial.

next under the network setting, I will remove all inbound access to this instance to demonstrate we can connect without public access. but keep in mind that outbound access is required for ssm agent connectivity, which is added by default when creating a security group.

I will keep storage as default as below.

in the advanced setting we need to add iam role(IAM instance profile) we created in the previous steps.

next hit the launch instance button.

and you will get a similar output as below.

click on the instance id to navigate to the instance page. Here we can recognize the newly created instance. click on the instance id to view the instance details page.

click on Connect to navigate to the Connect page.

in the connect page navigate to the session manager tab and press connect

if you receive an error as below, check the troubleshooting section below.

if everything goes well new tab will be created with a terminal displayed as below. now you are connected to the instance with a terminal.

• note user is ssm user.

Troubleshooting.

check if the instance has outbound internet access in security group rules.

make sure the instance has iam instance role attached.

check iam instance role permissions.
only require permissons is AmazonSSMManagedInstanceCore policy.

check if the instance ssm agent is reporting to the aws ssm fleet manager.
it should be listed as online.

If everything above is in place and you still cannot get ssm connectivity you will need to further troubleshoot issues in ssm agent.
a good place to start troubleshooting is by checking logs.

check ssm agent logs under /var/log/amazon/ssm

here we can see there is an access denied error in errors.log

here we can see there is an access denied error in amazon-ssm-agent.log and it is sleeping for 30 minutes.
this could happen if we attach the role after the instance starts,
so ssm agent checked to authenticate and failed then the next retry will be in 30 minutes.

we can restart ssm agent so it will try to reauthenticate and succeed this time.

This is just one scenario of debugging and your specific scenario might change but going through logs will give an insight into what the problem.

Thank you for reading. Share your thoughts in comments section.