Bastion host in AWS

What is a Bastion host

I like to see this as a security pattern for temporary access to services on a private network in the cloud. One popular use case is to access the Amazon RDS from a developer machine.

The bastion host is a machine that acts like a proxy/bridge/jump box between the public internet and the private network.

This machine has only one task and that is to create a secure communication channel to a specific service on the private network. Therefore it's important to make it as secure as possible, such as having strong authentication, software up to date, not running unnecessary software or open ports and monitoring.

How to setup a Bastion host for AWS

There is a short cut available at the great aws-quickstart page, which has an already baked cloudformation template file for us to use. Which sets up the following.

• A highly available architecture that spans two Availability Zones.
• A virtual private cloud (VPC) configured with public and private subnets, according to AWS best practices, to provide you with your own virtual network on AWS.
• In the public subnets:
  • Managed network address translation (NAT) gateways to allow outbound internet access for resources in the private subnets. *1–4 Linux bastion hosts in an Amazon Elastic Compute Cloud (Amazon EC2) Auto Scaling group for connecting to Amazon EC2 instances and other resources deployed in public and private subnets.
• An Amazon CloudWatch log group to hold the Linux bastion host shell history logs.
• AWS Systems Manager for access to the bastion host.

There are two templates available, depending on your needs, one for an existing VPC and one for a new VPC.

Both templates deploys a Linux bastion host in an auto-scaling group which make it easy to start/stop the service. We can later enable the feature scheduled actions which will stop the bastion host at a specific time, which is perfect, since people often forget to turn it off and this solution should only be used temporarily.

Step by step guide

This guide will enable access from a developer machine with SSH. The default is to only allow access from AWS Systems Manager to the bastion host.

1. Download the template, linux-bastion-entrypoint-existing-vpc.template.yaml, to your machine
2. Open up a new tab and go the AWS Management console
3. Navigate to the EC2 service, and in the left pane click on the Key Pairs menu item.
4. In Key pairs view, click the Create key pair button and fill out the form.
5. When the key pairs is created and you have saved the pem file in a secure location, this certificate should be treated as a secret! Be restrictive of how it's shared, and to whom, navigate to the CloudFormation service
6. In the Stacks view, click the button Create stack -> With new resources (standard)
7. In the Create stack dialog, choose Template is ready and upload the downloaded template from step one.
8. Fill out the form, Specify stack details
   • Network configuration is all about selecting correct VPC and subnets.
   • Under Amazon EC2 configuration, remember to select your key pair
   • The other parameters can use the defaults
9. Click Next, and then click Next for Configure stack options, and finally click Submit
10. Wait for the Cloudformation to deploy all the resources
11. Done with the first part

To add a scheduled action that turns off the bastion host automatically we will need to do the following.

1. Navigate to the EC2 service and click on the Auto Scaling Groups link in the bottom of the left pane.
2. In the list of auto scaling groups, click on the list item representing the bastion host.
3. In the next view click on the tab, Automatic scaling
4. At the bottom of the page, click on the button Create scheduled action, and fill out the form.
5. Done

One other option is to update the Cloudformation template file and add the action there by adding some yaml code to the template;

ScheduledActionDown:
  Type: "AWS::AutoScaling::ScheduledAction"
  Properties:
    AutoScalingGroupName:
      Ref: BastionAutoScalingGroup
    MinSize: 0
    MaxSize: 0
    DesiredCapacity: 0
    Recurrence: "0 19 * * *"
    TimeZone: "Europe/Stockholm"

The bastion host is now configured to be started on-demand and is taken down automatically at 19:00 Swedish time. To start the bastion host we will need to follow these instructions.

1. Use EC2 service view and navigate to the Instances view in the left pane 2 If the instance BastionHost is listed and has state Running - it's already started. Otherwise, follow these steps
   1. Go to the Auto Scaling Groups view in the left pane
   2. Click on the list item named bastion-host-...
   3. Click the Edit button for the Group details
   4. Set all values to 1 (Desired capacity, Minimum capacity, Maximum capacity) and click the Update button
   5. The bastion host will now start one instance in 10-30 seconds.

Connect to the bastion host with SSH

The bastion host should almost be ready to be connected to from your local machine via a SSH-tunnel. To be able to connect to the bastion host you will need to use an IP address that is allowed by the ingress rule on the security group for the bastion host.

1. Once more jump to the EC2 service and navigate to the Security Groups view via the left pane
2. Locate the security group name bastion-host-... in the list of groups and click on the security group id for that item
3. Click on the Edit inbound rules button
4. Add a new rule for your IP address (CIDR block)
5. Done, now should SSH traffic be allowed from your IP address

To open up a SSH tunnel to the bastion host from a developer machine that is using Ubuntu we can use this procedure with the command line interface.

1. Now we need to grab the pem file from when we created the key pair in the beginning of this blog
2. Put a copy of the pem file in the folder ~/.ssh/

3. Make sure that the file has correct permissions
   

   chmod 600 ~/.ssh/bastion-host.pem
   

4. Open the SSH tunnel
   

   ssh -i ~/.ssh/bastion-host.pem  ec2-user@[ip-of-bastion-host]
   # The [ip-of-bastion-host] is found at
   # AWS Console -> EC2 -> Instances -> BastionHost -> Public IPv4 address
   

5. Now the the tunnel is up between your machine and the bastion host.

Connect to a service on the private network

More configuration is needed to enable us to communicate with a service on the private network via the bastion host. Lets see how we could enable the possibility to communicate with a database hosted by RDS in AWS. What we need to is to enable the correct network traffic in the security group for the bastion host. This is similar to how we configured the inbound rule for allowing our machine to talk to the bastion host, but this time we need to allow outbound traffic.

1. Go to the EC2 service and navigate to the Security Groups view via the left pane
2. Locate the security group name bastion-host-... in the list of groups and click on the security group id for that item
3. Click on the Edit outbound rules button in the Outbound rules tab
4. In the edit outbound rules view, click on the Add rule button and configure the rule for the outgoing database traffic
   • Select a type that match your protocol, the PostgreSQL is selected above for communicating with a PostgreSQL database in RDS
   • In the destination you should select the RDS security group
5. Open up a SSH tunnel to the database

ssh -i ~/.ssh/bastion-host.pem ec2-user@[ip-of-bastion-host] -L 5432:[internal-dns-name-of-rds-instance]:5432
# Where the [internal-dns-name-of-rds-instance] can be found at
# AWS Console -> RDS -> Databases -> [your database instance] -> Endpoints

You should now be able to open your Database management tool and connect to the database on AWS

These rules can of course be added to the Cloudformation template if that feels like a better approach.

BastionSecurityGroup:
  Type: AWS::EC2::SecurityGroup
  Properties:
    GroupDescription: Enables access to bastion hosts
    VpcId: !Ref VPCID
    SecurityGroupEgress:
      - IpProtocol: tcp
        FromPort: 5432
        ToPort: 5432
        DestinationSecurityGroupId: !Ref <reference  to your RDS security group>
      - IpProtocol: tcp
        FromPort: 443
        ToPort: 443
        CidrIp: "0.0.0.0/0"
RDSInstanceSecurityGroupIngressRule:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    GroupId: !Ref <reference  to your RDS security group>
    IpProtocol: tcp
    FromPort: 5432
    ToPort: 5432
    SourceSecurityGroupId: !Ref BastionSecurityGroup
RDSIngressRule:
  Type: AWS::EC2::SecurityGroupIngress
  Properties:
    IpProtocol: tcp
    FromPort: 5432
    ToPort: 5432
    SourceSecurityGroupId: !Ref BastionSecurityGroup
    GroupId: !Ref <reference  to your RDS security group>

The reference to the RDS security group could be given as a parameter to the Cloudformation template like this.

RDSInstanceSecurityGroup:
  Type: AWS::EC2::SecurityGroup::Id
  Description: The RDS instance security group 

What else

If you need to create a tunnel often, then this shell alias might be of use.

# Add these rows in ~/.bashrc 
# NOTE: That I use local port 5434 here instead,
# that is because usually I already have a local database
# on that port and this enable to have both running

rdstunnelup () {
  ssh -i ~/.ssh/bastion-host.pem ec2-user@$1 -L 5434:<internal-dns-name-of-rds-instance>:5432
}

Reread the .bashrc file

source ~/.bashrc

With that in place, I can use the alias like this.

rdstunnelup <the ip address to the bastion host>

If your are using Windows as your local machine then you can use PuTTY tools to connect to the bastion host. Since PuTTY needs the certificate in the .ppk format we will need to convert the .pem file.

1. Start PuTTYGen (one of the tools in the PuTTY-package)
2. Select Conversion -> Import Key from the menu
3. Select the pem file that is used for the bastion host
4. Enter a passphrase (and do not forget this passphrase)
5. Click on the Save private key button

When you have a .ppk file you can proceed with next step to setup the SSH authentication agent. Pageant is an SSH authentication agent. It holds your private keys in memory, already decoded, so that you can use them often without needing to type a passphrase.

1. Open Pageant (one of the tools in the PuTTY-package)
2. Click on the Add key button
3. Use the ppk file
4. Enter the passphrase for the ppk file when prompted
5. Close the window.

Now everything is set for configuring the PuTTY client for port forwarding.

1. Open the PuTTY client (one of the tools in the PuTTY-package)
2. In the left pane (Category), select Connection -> SSH -> Auth and make sure that "Allow agent forwarding" is selected.
3. In the left pane (Category), select Connection -> SSH -> Tunnels and do following;
   • In the Source port field, add 5434
   • In the Destination field, add the [internal-dns-name-of-rds-instance] and port (5432)
   • Click on the Add button
4. In the left pane (Category), select Session and do following:
   • In the Host Name (or IP address) field, add the [ip-of-bastion-host] which is found under AWS Console -> EC2 -> Instances -> BastionHost -> Public IPv4
   • In the Saved Sessions field, add bastion-host
   • Click on the Save button
5. Click on the Open button at the bottom of the window
6. Enter the user name: ec2-user, when prompted the login as
7. Now you should be able to connect to localhost:5434 to talk to the database in AwS.

Summary

There is a lot of configuration to setup a bastion host, however that is only done once. When everything is at place it will be easy to connect to your service in the private network in AWS.

It is really educational exercise to setup this solution on AWS, if you have the possibility to this then you should.

Happy configuration!