There are many reasons to have solid logging practices in place, after all it can help reduce troubleshooting time, and depending on the environment, customer or data, there is often audits which require it.
Setting up, configuring, and securely storing SSH logs can be a lot of work, as the number of server instances increases the problem only gets more challenging.
I am going to walk you through a demo of using AWS services (see below for a list) to securely record and store logs.
I am assuming you have some prior AWS administration and security experience before jumping into this, I wont go into great deal on everything , but I will try and provide detailed example configs.
AWS Native Services we will be using for this setup.
- AWS EC2 instances
- AWS IAM
- AWS KMS
- AWS CloudWatch
- AWS S3
- AWS Systems Manager/SSM
For testing and demo purposes, I will be using an Amazon Linux, this instance also has the AWS SSM agent installed already out of the box, but you are free to use another linux OS if you want.
Before we get started with launching anything, we will need setup a new IAM role which will be assigned to our new test EC2 server. This new role will allow for logs to be sent to and stored in both AWS CloudWatch and S3.
Creating a new IAM ROLE
I simply called my role ssm-session-logs-demo
Then I created and attached two permissions policies
s3-ssm-session-logs-07
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel",
"ssm:UpdateInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:PutObject"
],
"Resource": "arn:aws:s3:::ssm-session-logs-07/*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetEncryptionConfiguration"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"kms:Decrypt"
],
"Resource": "arn:aws:kms:us-east-1:12345678910:key/1234abcd"
},
{
"Effect": "Allow",
"Action": "kms:GenerateDataKey",
"Resource": "*"
}
]
}
AmazonEC2RoleforSSM
{
"Version": "2012-10-17",
"Statement": [
{
"Effect": "Allow",
"Action": [
"ssm:DescribeAssociation",
"ssm:GetDeployablePatchSnapshotForInstance",
"ssm:GetDocument",
"ssm:DescribeDocument",
"ssm:GetManifest",
"ssm:GetParameters",
"ssm:ListAssociations",
"ssm:ListInstanceAssociations",
"ssm:PutInventory",
"ssm:PutComplianceItems",
"ssm:PutConfigurePackageResult",
"ssm:UpdateAssociationStatus",
"ssm:UpdateInstanceAssociationStatus",
"ssm:UpdateInstanceInformation"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ssmmessages:CreateControlChannel",
"ssmmessages:CreateDataChannel",
"ssmmessages:OpenControlChannel",
"ssmmessages:OpenDataChannel"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2messages:AcknowledgeMessage",
"ec2messages:DeleteMessage",
"ec2messages:FailMessage",
"ec2messages:GetEndpoint",
"ec2messages:GetMessages",
"ec2messages:SendReply"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"cloudwatch:PutMetricData"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ec2:DescribeInstanceStatus"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"ds:CreateComputer",
"ds:DescribeDirectories"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"logs:CreateLogGroup",
"logs:CreateLogStream",
"logs:DescribeLogGroups",
"logs:DescribeLogStreams",
"logs:PutLogEvents"
],
"Resource": "*"
},
{
"Effect": "Allow",
"Action": [
"s3:GetBucketLocation",
"s3:PutObject",
"s3:GetObject",
"s3:GetEncryptionConfiguration",
"s3:AbortMultipartUpload",
"s3:ListMultipartUploadParts",
"s3:ListBucket",
"s3:ListBucketMultipartUploads"
],
"Resource": "*"
}
]
}
Once the polices are created, you can head over to AWS EC2 and launch a new instance, using the IAM ROLE we created above. Dont worry too much about the VPC or the security group settings, as we wont really need to configure anything and the default can be used. In fact, for the security group, we wont need to allow any inbound rules, not even SSH.
Setting UP KMS
From the AWS Key Management Service console, select customer managed keys.
Select Create Key. I named my Key sessionlogstocloudwatch , but you can call it whatever you want.
The key type needs to be symmetric, key material origin will be KMS or AWS_KMS, key usage will be encrypt and decrypt. Key rotation is set as auto every year.
Sample Key Policy
{
"Version": "2012-10-17",
"Id": "key-consolepolicy-3",
"Statement": [
{
"Sid": "Enable IAM User Permissions",
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::12345678910:root"
},
"Action": "kms:*",
"Resource": "*"
},
{
"Sid": "Allow access for Key Administrators",
"Effect": "Allow",
"Principal": {
"AWS": [
"arn:aws:iam::428298842207:user/CMKAdmin"
]
},
"Action": [
"kms:Create*",
"kms:Describe*",
"kms:Enable*",
"kms:List*",
"kms:Put*",
"kms:Update*",
"kms:Revoke*",
"kms:Disable*",
"kms:Get*",
"kms:Delete*",
"kms:TagResource",
"kms:UntagResource",
"kms:ScheduleKeyDeletion",
"kms:CancelKeyDeletion"
],
"Resource": "*"
},
{
"Sid": "Allow access to CloudWatch Log",
"Effect": "Allow",
"Principal": {
"Service": "logs.us-east-1.amazonaws.com"
},
"Action": [
"kms:Encrypt",
"kms:Decrypt",
"kms:ReEncrypt*",
"kms:GenerateDataKey*",
"kms:DescribeKey"
],
"Resource": "*",
"Condition": {
"ArnLike": {
"kms:EncryptionContext:aws:logs:arn": "arn:aws:logs:us-east-1:12345678910:*"
}
}
}
]
}
AWS CloudWatch Log Group
Next we will head over to the AWS CloudWatch Logs, and Log Groups. This log group is where we will send SSH session logs.
We will select create log group
I named my log group ssm-session-logs
I set the retention period to 1 month
I set the KMS Key ARN to
"arn:aws:kms:us-east-1:12345678910:key/1234abcd"
this exact value will be different for you, it can be found when we created the KMS key above.
AWS S3 - Storing SSH Logs in an encrypted S3 bucket
This step is optional, but in addition to CloudWatch, I also wanted to store logs in S3.
See the IAM Role and Permissions above, your S3 bucket name will be different, but basically we need to include this in the policy to allow EC2 to have the proper permissions.
One other note, when I created my new S3 bucket, access should be set as "Bucket and objects not public", Block all public access - On, default encryption is enabled, server-side encryption using AWS Key Management Service key (SSE-KMS), AWS KMS Key ARN is the same ARN value as what I created above with the KMS service.
AWS Systems Manager Session Manager
We are going to head over to the AWS System Manger, on the left side if we scroll down to node management, then select session manager. On the session manager screen, we want to select preferences, your screen may look similar to this when you are done.
Testing Sessions
From the AWS Systems Manager, Session Manager section, we will select Start Session. I will select the instance name "ssmlogs", your name will be different, remember this was the AWS Linux EC2 instance we launched above with the new IAM ROLE and permission policies.
Notice the session is encrypted using AWS KMS!!
Now if we go back to the AWS CloudWatch Log groups, and open the log group we created, we will see new log events! These contain the commands and other useful info.
Now if we go back to the S3 bucket we created, and open the newly generated folders and log files, we will see new logs event similar to what we saw in CloudWatch.
Top comments (0)