Deploying the VPC Endpoint - Gateway for Amazon S3

Hello everyone!

In this post, we will discuss the VPC Endpoint - Gateway for Amazon S3, along with various scenarios that highlight its importance. Additionally, I will guide you on deploying this resource using the AWS Console and provide the Terraform code from my GitLab repository for deploying the proposed topology.

This EC2 instance is running inside of a Public subnet, and there are some data stored in S3 bucket. Let's say that the EC2 instance needs to upload and download some data stored in S3 bucket, so basically the EC2 instance would make an API call, and then it will access the data over the public internet through the Internet Gateway (IGW). It's obvious architecture, and it works perfectly there is no problem at all.

Now let's move on to an other scenario, where there is a similar architecture, but now the EC2 instance redise inside of the Private subnet. Now what happens? The EC2 instance needs to upload and download data in S3 bucket. We have learned on the old scenario that the EC2 instance needs to access the internet to upload/download the datas, but now as this EC2 instance reside into a private subnet, how will it be able to access the S3 bucket? So, that's why exist a resource called NAT Gateway. We would add a NAT Gateway in a public subnet, modify the private route table, and then the traffic will go to the S3 bucket over the public internet. Now the EC2 instance will be able to access the S3. So this works perfectly too. There is no problem with that too.

Yeah, probally you are asking yourself, why do I need to use VPC endpoint services? If you noticed, there is one thing in commun on this both scanarios. In both scanarios, the EC2 instance need to reach out the S3 bucket over the public internet, but what AWS says, if all the services are into the same region, as your VPC, including S3 service that we saw, there is a better way to access these services and that better way is called VPC endpoint services.

So what are VPC endpoints, endpoints allow you to connect to AWS services, using a private network instead of going for a public network. Then, we won't need to use the IGW or NAT Gateway to reach the S3 bucket, which it's good because we can save money, for example, NAT Gateway charge per hour running as well as data flow, our traffic wouldn't go out over public internet that it's not safe like a private network, and other benefits.

There are two types of the VPC endpoint, Gateway and Interface endpoint, but now, let`s focus on the Gateway endpoint.

Now we are going to see how we can deploy a Gateway endpoint to permit the private EC2 instance (ec2_app_web) inside of the Private subnet without Internet access can reach the S3 bucket through Gateway Endpoint. As the private EC2 instance is not reachable through the internet we are going to create a Linux Bastion Host (linux_bastion_host) to access the private EC2 instance, and then test the connectivity with S3 bucket, as we can see on the digram below.

Note: I assume you know how to create the VPC, subnets, SG, route tables, so this post will only focus on deploying the resources to create the Gateway endpoint.

Deployment steps:

1 - Creating the Policy to permit List, Get and Put data into the S3 bucket.

In IAM dashboard, click on Policies, and then click policy button

Click on JSON button, paste the policy and click in Next

Policy:

{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "Statement1",
"Effect": "Allow",
"Action": [
"s3:Get*",
"s3:List*",
"s3:Put*",
"s3-object-lambda:Get*",
"s3-object-lambda:List*",
"s3-object-lambda:Put*"
],
"Resource": "*"
}
]
}


Define a name, check the access level, and click on the create policy button

Policy created

Now, let's create the IAM Role to attach the private EC2 instance (ec2_app_web). On the IAM dashboard, click in Roles, and then create role button.

Select these option, and click in next

Find the the policy name created on the step before, select it, and next.

Define a name, description, check the policy and click in create role

Role created

Now, let's assgin the IAM Role created to the private EC2 instance (ec2_app_web). Select the instance, Action, Security and Modify IAM Role

Find the IAM Role created, select it and update IAM Role

2 - Create the Gateway endpoint for accessing the S3.

In VPC dashboard, select Endpoint, and click in create endpoint button

Define a name, select AWS service, select com.amazonaws.us-east-1.s3 (type Gateway), select the VPC, and route table that is assign to private subnet because the route table will update with a prefix to reach S3 service through the VPC endpoint. At the bottom of the page click on the create endpoint button

VPC endpoint created

If you check the route table that belong to the private subnet, you will see the route to reach the VPC endpoint.

3 - Create S3 bucket.

On the S3 management console, click in create bucket

Define a name, and region where EC2 instance was deployed. At the bottom of the page click on the create bucket button

Bucket created

4 - Check the connectivity

Now, we are going to access the Linux bastion host (linux_bastion_host), and from it access the private EC2 instance (ec2_app_web) to list the S3 bucket.

Accessing the Linux bastion host

Accessing the Private EC2 instance, listing all buckets, creating a file, and uploading it to the bucket.

Based on this result, we completed the deployment and validated connectivity to the S3 bucket from an EC2 instance without internet access.

If you would like to deploy this env via Terraform code as the purposed topology, so feel free to access my Gitlab repository to clone and deploy it. Before deploy via Terraform, replace the profile in provider.tf file and replace your Key_name at the EC2 instace code as per your own settings.

Conclusion:

In conclusion, we have explored different scenarios that demonstrate the significance of utilizing the VPC Endpoint - Gateway for Amazon S3. By understanding its role and benefits, we can effectively deploy this solution to enhance our infrastructure. I trust that you found this discussion enjoyable.

Reference Link:
https://docs.aws.amazon.com/vpc/latest/privatelink/gateway-endpoints.html
https://docs.aws.amazon.com/vpc/latest/privatelink/vpc-endpoints-s3.html