AWS Networking with Terraform: VPC Gateway Endpoint to connect to S3 over AWS PrivateLink

In our previous post, we explained how to use VPC Interface Endpoints to establish secure connections with various AWS services via the AWS Private Network.

Now, let's advance our AWS networking knowledge by exploring the implementation of VPC Gateway Endpoints, particularly focusing on connecting to Amazon S3. We'll walk through the process of setting up Terraform modules to streamline the creation of distinct components within this architecture.

A VPC endpoint enables customers to privately connect to supported AWS services and VPC endpoint services powered by AWS PrivateLink. Amazon VPC instances do not require public IP addresses to communicate with resources of the service. Traffic between an Amazon VPC and a service does not leave the Amazon network.

There are two types of VPC endpoints:

1. interface endpoints
2. gateway endpoints

Gateway endpoints

A gateway endpoint targets specific IP routes in an Amazon VPC route table, in the form of a prefix-list, used for traffic destined to Amazon DynamoDB or Amazon Simple Storage Service (Amazon S3). Gateway endpoints do not enable AWS PrivateLink. Instances in an Amazon VPC do not require public IP addresses to communicate with VPC endpoints, as interface endpoints use local IP addresses within the consumer Amazon VPC. Gateway endpoints are destinations that are reachable from within an Amazon VPC through prefix-lists within the Amazon VPC’s route table. There is no additional charge for using gateway endpoints.

Architecture Overview:

Before we embark on our journey, let's grasp the architectural layout we'll be working within:

Step 1: Create a VPC

Create a VPC with a public and private subnet. Please refer to my github repo in resources section below.

Step 2: Establish IAM Role with Policy and Instance Profile for S3 Access

Facilitate secure access to Amazon S3 by configuring an IAM role with a corresponding policy and instance profile tailored for S3 interactions.

####################################################
# Create the IAM role for EC2 assumption
####################################################
resource "aws_iam_role" "ec2_s3_role" {
  assume_role_policy = jsonencode({
    "Version" : "2012-10-17",
    "Statement" : [
      {
        "Effect" : "Allow",
        "Action" : [
          "sts:AssumeRole"
        ],
        "Principal" : {
          "Service" : [
            "ec2.amazonaws.com"
          ]
        }
      },
    ]
  })
  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-ec2-iam-role"
  })
}

####################################################
# Create the IAM policy to allow all s3.* actions
####################################################
resource "aws_iam_policy" "ec2_s3_policy" {
  name = "ec2-iam-s3-policy"
  path = "/"
  policy = jsonencode(
    {
      "Version" : "2012-10-17",
      "Statement" : [
        {
          "Action" : [
            "s3:*"
          ],
          "Effect" : "Allow",
          "Resource" : "*"
        },
      ]
    }
  )
  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-ec2-s3-policy"
  })

}

####################################################
# Attach IAM policy to the role
####################################################
resource "aws_iam_policy_attachment" "ec2_s3_role_policy" {
  policy_arn = aws_iam_policy.ec2_s3_policy.arn
  roles      = [aws_iam_role.ec2_s3_role.name]
  name       = "${var.naming_prefix}-ec2-s3-policy-att"
}

####################################################
# Create an EC2 instance profile with the role
####################################################
resource "aws_iam_instance_profile" "ec2_s3_instance_profile" {
  role = aws_iam_role.ec2_s3_role.name
  tags = merge(var.common_tags, {
    Name = "${var.naming_prefix}-ec2-s3-instance-profile"
  })
}

Step 3: Deploy Bastion and Private Host with Instance Profile

Instantiate a bastion host and a private host, private host equipped with the requisite instance profile. Also amend security group to connect private host from bastion host.

####################################################
# Create EC2 Server Instances
####################################################

module "vpc_a_bastion_host" {
  source           = "./modules/web"
  instance_type    = var.instance_type
  instance_key     = var.instance_key
  subnet_id        = module.vpc_a.public_subnets[0]
  vpc_id           = module.vpc_a.vpc_id
  ec2_name         = "Bastion Host A"
  sg_ingress_ports = var.sg_ingress_public
  common_tags      = local.common_tags
  naming_prefix    = local.naming_prefix
}

module "vpc_a_private_host" {
  source           = "./modules/web"
  instance_type    = var.instance_type
  instance_key     = var.instance_key
  subnet_id        = module.vpc_a.private_subnets[0]
  vpc_id           = module.vpc_a.vpc_id
  ec2_name         = "Private Host A"
  sg_ingress_ports = var.sg_ingress_private
  common_tags      = local.common_tags
  naming_prefix    = local.naming_prefix
  instance_profile = module.iam_ec2_s3.instance_profile_name
}

####################################################
# Amend Private Host SG to allow traffic from Bastion Host SG
####################################################
resource "aws_security_group_rule" "public_in_ssh" {
  type                     = "ingress"
  from_port                = 22
  to_port                  = 22
  protocol                 = "tcp"
  security_group_id        = module.vpc_a_private_host.security_group_id
  source_security_group_id = module.vpc_a_bastion_host.security_group_id
}

Step 4: Provision an S3 Bucket

Set up an Amazon S3 bucket to serve as the repository for your data, enabling scalable and durable object storage.

####################################################
# Create an S3 Bucket
####################################################
resource "aws_s3_bucket" "s3_bucket" {
  bucket = "s3-test-bucket-ct"

  tags = merge(local.common_tags, {
    Name = "${local.naming_prefix}-s3-bucket"
  })
}

Step 5: Create a VPC Gateway Endpoint Connected to the Private Route Table

Establish a VPC Gateway Endpoint intricately linked to the private route table, thus enabling secure and efficient communication between your VPC and Amazon S3.

####################################################
# Create Security Group and Gateway Endpoint
####################################################
resource "aws_vpc_endpoint" "s3_vpc_ep_gateway" {
  vpc_id            = module.vpc_a.vpc_id
  vpc_endpoint_type = "Gateway"
  service_name      = "com.amazonaws.${var.aws_region}.s3"
  route_table_ids   = "${[module.vpc_a.private_route_table_id]}"

  tags = merge(local.common_tags, {
    Name = "${local.naming_prefix}-gateway-endpoint"
  })
}

Steps to Run Terraform

Follow these steps to execute the Terraform configuration:

terraform init
terraform plan 
terraform apply -auto-approve

Upon successful completion, Terraform will provide relevant outputs.

Apply complete! Resources: 18 added, 0 changed, 0 destroyed.

Outputs:

s3_bucket_name = "s3-test-bucket-ct"
vpc_a_bastion_host_IP = "54.173.10.180"

Testing the outcome

VPC Interface Endpoint:

Prefix list association added to private route table:

S3 Bucket:

Accessing S3 bucket from private host via gateway endpoint:

Cleanup

Remember to stop AWS components to avoid large bills.

terraform destroy -auto-approve

With this we've successfully established a VPC Gateway Endpoint to establish secure connections with Amazon S3 over the AWS private network.

In our upcoming module, we'll understand S3 Simple Storage Service to host a static website.

Resources:

Github Link: https://github.com/chinmayto/terraform-aws-networking-vpc-gateway-endpoint
VPC Endpoints: https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html