I Gave an EC2 Instance Zero Internet and It Still Uploaded to S3

The challenge

The client's requirement fit in one sentence:

The private backend must talk to Amazon S3 without ever touching the public internet.

The environment had two VPCs:

• A custom VPC with a private EC2 instance. It had no Internet Gateway and no NAT Gateway.
• A default VPC with a public-facing EC2 instance acting as a bastion (jump host).

The plan was to SSH into the bastion, hop to the private machine, and run aws s3 cp from there while keeping every packet on the AWS network. That took five pieces working together: VPC Peering, route tables, security groups, an IAM role and an S3 Gateway Endpoint.

![Full architecture: default VPC with bastion, custom VPC with private EC2, peering link, IAM role and S3 Gateway Endpoint]

The diagram above is the whole design. The rest of this post builds it one layer at a time and shows where each layer can fail.

The roadblocks

1. There was no road out.
With no IGW and no NAT, the private subnet has no default route. Any call to the S3 public endpoint just hangs until it times out.

2. Peering is easy to create and easy to get wrong.
A VPC Peering connection (pcx-...) is only a handshake. Traffic flows only when both route tables point at it. My first ping test looked like this:

$ ping 172.31.22.46
8 packets transmitted, 0 received, 100% packet loss, time 7274ms

A common cause of this is a one-sided route. The request reaches the target, but the reply has no route back, so ping reports nothing received.

3. Locking down security groups without locking yourself out.
Both instances started with 0.0.0.0/0 on SSH and ICMP, and the console warns about this for good reason. Replacing those rules with /32 entries means one typo can cut off your own access, so the order of changes matters.

4. Credentials.
Running aws configure and pasting access keys onto an EC2 instance is exactly what a secure design should avoid.

The implementation

Step 1: Create and accept the VPC Peering connection

Request the connection from one VPC and accept it from the other. It is not active until accepted, and that step is easy to forget.

Step 2: Build symmetric route tables

This was the fix for the 100% packet loss. Each side needs a route to the other:

Route table	Destination	Target
Custom VPC (private route table)	Default VPC CIDR	pcx-xxxxxxxx
Default VPC (main route table)	Custom VPC CIDR	pcx-xxxxxxxx

Edit the route table that is actually associated with the subnet. Across two VPCs there are several route tables, including main tables with no explicit associations, so it is easy to edit the wrong one.

Step 3: Harden security groups to a /32 micro-perimeter

Each instance accepts SSH (22) and ICMP only from the other instance's exact private IP:

• Private instance SG: allow from <bastion-private-ip>/32
• Bastion SG: allow from <private-instance-ip>/32

Do this after peering and routing are confirmed working, and keep one SSH session open while you change the rules. Use private IPs here. Traffic across the peering connection arrives from the peer's private address, not its public one.

Step 4: Attach an IAM role instead of access keys

Create an IAM role for EC2 with S3 permissions and attach it to the private instance as an instance profile. The AWS CLI picks up temporary credentials from the instance metadata service automatically, so there is nothing to leak.

I used AmazonS3FullAccess to validate the architecture. For production, scope it down:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": ["s3:ListBucket"],
      "Resource": "arn:aws:s3:::YOUR-BUCKET"
    },
    {
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::YOUR-BUCKET/*"
    }
  ]
}

Step 5: Deploy an S3 Gateway Endpoint

In VPC console > Endpoints, search for the S3 service and choose the Gateway type: com.amazonaws.us-east-1.s3. Associate it with the private subnet's route table.

AWS adds a route that sends S3 traffic (via a managed prefix list) to the endpoint, so S3 requests stay on the AWS network. The bucket must be in the same Region as the endpoint.

Where a request can fail, hop by hop

Each hop has its own checks and its own symptom. This map helped me debug faster than guessing:

![Hop-by-hop map from laptop to S3 showing the checks that must pass at each hop and the symptom when one fails]

A short rule of thumb:

• Timeout or 100% packet loss means a network problem: peering, routes or security groups.
• AccessDenied (403) means an identity problem: the role policy or bucket policy.
• Unable to locate credentials means no IAM role is attached to the instance.

The outcome

Once the routes were symmetric, the same ping that had failed worked immediately:

$ ping 172.31.22.46
64 bytes from 172.31.22.46: icmp_seq=1 ttl=127 time=0.877 ms
64 bytes from 172.31.22.46: icmp_seq=2 ttl=127 time=0.508 ms
...
6 packets transmitted, 6 received, 0% packet loss
rtt min/avg/max/mdev = 0.490/0.564/0.877/0.139 ms

![Before and after ping: 100% packet loss with one-sided routes versus 0% packet loss with symmetric routes]

Sub-millisecond latency and 0% packet loss confirmed the peering was healthy. Then the real test, from the private instance:

$ aws s3 ls
2026-10-17 08:02:05 your-bucket-name

$ echo 'this is my first private machine upload' > test.txt
$ aws s3 cp test.txt s3://your-bucket-name/
upload: ./test.txt to s3://your-bucket-name/test.txt

$ aws s3 ls s3://your-bucket-name/
2026-10-17 08:04:59         75 test.txt

The CLI listed buckets and uploaded the file. Authentication came from the IAM role, and the request was routed through the Gateway Endpoint.

What the design delivers:

• No static credentials anywhere. Temporary role credentials only.
• A tiny attack surface. SSH and ICMP are allowed only between two exact private IPs.
• A private S3 path. Traffic to S3 stays on the AWS network instead of crossing the public internet.
• Lower cost. Gateway Endpoints for S3 carry no additional endpoint charge, unlike Interface Endpoints (PrivateLink), which bill hourly and per GB.

Debugging checklist

Work through these in order. A failure early in the chain hides everything after it:

![Six-step checklist table with what to verify, the symptom if wrong, and the fix]

1. Is the peering connection Active (accepted)?
2. Do both route tables have the peer CIDR pointing at pcx-?
3. Is each route table associated with the right subnet?
4. Do the security groups allow the peer's private IP as /32?
5. Is the Gateway Endpoint attached to the private route table, in the bucket's Region?
6. Does the instance have an IAM role whose policy allows the S3 actions you call?

Final thoughts

Most "it doesn't work" moments in AWS networking come down to routes and security groups. With peering, creating the connection is the easy part, and the route tables on both sides are what make it work. When something fails silently, check both directions before blaming the service.

Have you run into cross-VPC routing problems like this?

Tell me in the comments: do you use peering, Transit Gateway, or PrivateLink for this kind of setup?