Secure Proxy Server in AWS

Securely access third-party content with whitelisted IP from wherever you are.

If you want to jump directly to the solution using CDK, go here.

Proxy servers are used for several purposes, and in general they provide a gateway between users and the destination they want to access.

In this example, we will tackle the scenario where you want to access third-party content protected by a firewall that can only be accessed from specific white-listed IPs wherever you are.

Well, I’m sure there are a couple of ways to solve it, but here I’ll describe a solution that doesn’t depend on increasing too much costs and complexity in your infrastructure, like dealing with VPN clients or Direct Connect links, while not compromising security.

Although in general NAT Gateways should be avoided, as they might incur additional unnecessary costs when outbound connectivity from private subnets could be solved differently (e.g., by using VPC endpoints or IPv6 egress-only internet gateways), in this case, I opted to configure my VPC with a private subnet with NAT Gateway so I would have enough flexibility at my side to control how I want to proxy the requests to the third party service while still maintaining a fixed small list of public IPs which would be whitelisted at the third part service (e.g.: 3 elastic IPs from NAT Gateway, one for each AZ).

For the proxy server, I chose to use EC2 with Squid Cache as, in this case, I wanted to keep things simple while I also didn’t need 4 9s availability for this server, and if I ever need it, I can restart it or quickly spin up a new one. Of course, if you want to go cheaper, you might also consider spot instances or set functions for starting and stopping the instance when you need it, or even do it manually if it’s a once-in-a-while usage.

A couple of important things about the EC2 setup:

• Do not assign a key pair for login (it is not needed; generally, having long-living keys lying around is a security risk).

• Assign the EC2 instance to a private subnet to reduce exposure to the public internet.

• Use a security group with no INGRESS rules (we will use SSM VPC endpoints and instance connect to access it instead).

• Ensure that EGRESS TCP is enabled for all (or restrict it to ephemeral ports used to communicate with AWS services and the services you want to allow your proxy access).

• Make sure you assign the ‘AmazonSSMRoleForInstancesQuickSetup’ IAM instance profile (or a custom one with the same permissions) to it.

• Use Amazon Linux 2 or newer AMI (check which AMIs support instance connect)

• Add a ‘Name’ tag with a value of ‘proxy-server’, for example, so you can easily automate the tunnel creation later with a script.

Use the following user data for installing and starting squid-cache, instance connect and SSM agent:

#!/bin/bash

yum update -y -q

sudo yum install ec2-instance-connect
sudo systemctl enable amazon-ssm-agent
sudo systemctl start amazon-ssm-agent

sudo yum -y install squid

sudo service squid restart

To access the EC2 instance, we will use instance connect. As the EC2 is in a private subnet, we need to create the following VPC Endpoints to be able to access it:

• ssm.region.amazonaws.com

• ssmmessages.region.amazonaws.com

• ec2messages.region.amazonaws.com

Finally, you need a role you will assume for creating a tunnel and port forwarding to your proxy server, through a temporary ssh session started by SSM. This role must have the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession",
        "ec2-instance-connect:SendSSHPublicKey"
      ],
      "Resource": [
        "arn:aws:ec2:*:*:instance/*"
      ],
      "Condition": {
        "StringEquals": { "aws:ResourceTag/Name": "proxy-server" }
      }
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ssm:StartSession"
      ],
      "Resource": [
        "arn:aws:ssm:*:*:document/AWS-StartSSHSession"
      ]
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ssm:TerminateSession",
        "ssm:ResumeSession"
      ],
      "Resource": ["arn:aws:ssm:*:*:session/$${aws:username}-*"]
    },
    {
      "Sid": "",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Resource": "*"
    }
  ]
}

The statements above give permissions for describing all EC2 instances in your account, sending SSH public keys, and starting and terminating sessions on EC2 instances with the ‘proxy-server’ name.

On your computer, to start the session and create the tunnel, you need to install and configure the aws-cli and the session manager plugin for aws-cli.

Finally, you can create the tunnel with the following script:

FORWARDED_PORT=3128
AWS_REGION=<<your AWS region>>

ec2_instance_id=$(aws ec2 describe-instances \
  --filters Name=tag:Name,Values=proxy-server Name=instance-state-name,Values=running \
  --output text --query 'Reservations[*].Instances[*].InstanceId')

ec2_az=$(aws ec2 describe-instances \
                --filters Name=tag:Name,Values=proxy-server Name=instance-state-name,Values=running \
                --output text --query 'Reservations[*].Instances[*].Placement.AvailabilityZone')


echo "Generating temporary keys"

TMP=$(mktemp -u key.XXXXXX)".pem"

ssh-keygen -t rsa -f "$TMP" -N "" -q -m PEM

aws ec2-instance-connect send-ssh-public-key \
  --region ${AWS_REGION} \
  --instance-id ${ec2_instance_id} \
  --availability-zone ${ec2_az} \
  --instance-os-user ec2-user \
  --ssh-public-key "file://$TMP.pub"

ssh -i $TMP \
      -Nf -M \
      -L ${FORWARDED_PORT}:localhost:${FORWARDED_PORT} \
      -o "UserKnownHostsFile=/dev/null" \
      -o "StrictHostKeyChecking=no" \
      -o IdentitiesOnly=yes \
      -o ProxyCommand="aws ssm start-session --target %h --document AWS-StartSSHSession --parameters portNumber=%p --region=eu-central-1" \
      ec2-user@${ec2_instance_id}

rm $TMP "$TMP.pub"

Before running the script above, ensure your aws-cli sso is properly configured to access the account you deployed your proxy server to.

The script above will create a temporary key and upload it to the ec2 instance, enabling you temporary access to this instance with this key. The key is automatically removed after 60 seconds, and if you haven’t accessed your ec2 instance with that key during that period, you’d need to create and send a new one. After the key is sent, a port forwarding proxy is created on port 3128. Finally, you can access the third-party content through your proxy by using localhost:3128, as in the curl example below:

curl -x "127.0.0.1:3128" "http://httpbin.org/ip"

To destroy the tunnel, you may execute the following command:

lsof -P | grep ':'${FORWARDED_PORT} | awk '{print $2}' | xargs kill -9

That’s it. Consider fine-tuning /etc/squid/squid.conf to secure it even further against misuse.

Example with CDK is available on github.