DEV Community

Tejas Shinkar
Tejas Shinkar

Posted on

AWS VPC Hands-On Lab

#aws #cloud #devops #sideprojects

Focus: A practical record of the VPC networking work completed in AWS.
This document records what was built, tested, observed, troubleshot, and learned during the hands-on sessions.

šŸ“š VPC Theory & Concepts

The concepts behind these hands-on practicals are covered in the following VPC articles:

These articles cover the underlying concepts in detail, while this document focuses specifically on the hands-on implementation, testing, troubleshooting, and observations from the VPC practicals.

1.Building a VPC from Scratch with Public and Private Subnets

Objective

Build the core VPC networking architecture manually instead of using the AWS VPC and more wizard. The goal was to understand how VPC, subnets, route tables, an Internet Gateway, a NAT Gateway, and EC2 instances fit together to provide public and private connectivity.

Architecture 

VPC: 10.200.0.0/16

                         Internet
                            |
                           IGW
                            |
              +-------------+-------------+
              |                           |
       Public Subnet                Private Subnet
       10.200.0.0/24                10.200.1.0/24
              |                           |
         Public EC2                   Private EC2
              |                           |
              +---- NAT Gateway <---------+
                         |
                  Public Subnet
Enter fullscreen mode Exit fullscreen mode

What We Built

  • VPC: lab-vpc
  • CIDR: 10.200.0.0/16
  • Public subnet: 10.200.0.0/24
  • Private subnet: 10.200.1.0/24
  • Internet Gateway: lab-igw
  • Public route table
  • Private route table
  • NAT Gateway: lab-nat-gw
  • Elastic IP for NAT Gateway
  • Public EC2
  • Private EC2

Implementation

VPC

Created VPC lab-vpc with CIDR 10.200.0.0/16, using VPC only so the networking components could be built manually.

Subnets

Created:

  • public-subnet-1 ā€” 10.200.0.0/24 ā€” ap-south-1a
  • private-subnet-1 ā€” 10.200.1.0/24 ā€” ap-south-1b

The subnets were placed in different Availability Zones to introduce a basic multi-AZ layout.

Internet Gateway

Created and attached lab-igw. The IGW alone was not enough to make the subnet public; the public subnet also needed a route to the IGW.

Public Route Table

Configured 0.0.0.0/0 ā†’ Internet Gateway and associated it with public-subnet-1.

NAT Gateway

Created lab-nat-gw. The NAT Gateway was placed in the public subnet and given an Elastic IP.

The private route table was then configured with 0.0.0.0/0 ā†’ NAT Gateway. This produced the outbound path:

Private EC2 ā†’ Private Route Table ā†’ NAT Gateway ā†’ Internet Gateway ā†’ Internet

EC2 Testing

Created public-ec2 and private-ec2.

The public EC2 had a public IP. The private EC2 had no public IP.

From the public EC2, we connected to the private EC2 using its private IP.

From the private EC2, we tested outbound Internet access:

curl https://checkip.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

The returned public IP corresponded to the NAT Gateway's Elastic IP rather than the private EC2's private address.

Evidence / Result

The lab successfully demonstrated:

Public EC2 ā†’ Internet

Private EC2 ā†’ NAT Gateway ā†’ Internet

The private EC2 could reach the Internet without having a public IP.

Problems / Troubleshooting

EC2 Instance Connect

Browser-based EC2 Instance Connect caused connectivity issues during the lab. The SSH Security Group rule was temporarily broadened to 0.0.0.0/0 to avoid making SSH access itself the blocker.

This was a temporary lab workaround, not a production configuration.

NAT Gateway placement

The placement initially raised the question of why the NAT Gateway belongs in a public subnet. The final working architecture made the reason clear:

NAT Gateway ā†’ Public Subnet ā†’ Public Route Table ā†’ IGW ā†’ Internet

Hands-On Takeaways

  • A subnet uses the route table associated with it.
  • The public route table pointed to the IGW.
  • The private route table pointed to the NAT Gateway.
  • The NAT Gateway provided outbound Internet access for the private EC2.
  • A private EC2 does not need a public IP to initiate outbound Internet traffic.
  • NAT Gateway placement and routing must be considered together.

2.Blocking HTTP While Allowing HTTPS with a Custom NACL

Objective

Create a custom Network ACL for a public subnet and use an Apache web server to prove that NACL rules can selectively allow or reject traffic.

The practical goal was simple: HTTP should fail, while HTTPS should work, allowing the NACL decision to be observed through an actual application.

Architecture 

Internet
   |
   v
Internet Gateway
   |
   v
Custom NACL
   |
   v
Public Subnet
   |
   v
EC2
 |
Apache
Enter fullscreen mode Exit fullscreen mode

What We Built

  • Custom VPC for the lab
  • Public subnet
  • Custom Network ACL
  • Security Group
  • Public EC2
  • Apache HTTP server
  • HTTPS configuration using a temporary self-signed certificate

NACL Configuration

The important rules were:

Rule Port Action
HTTP 80 DENY
HTTPS 443 ALLOW
SSH 22 ALLOW
Return traffic 1024ā€“65535 (TCP) ALLOW

The SSH rule was temporarily broadened during connectivity troubleshooting.

Apache Setup

Installed Apache:

sudo dnf install httpd -y
Enter fullscreen mode Exit fullscreen mode

Created a test page:

echo "<h1>NACL Lab - HTTP/HTTPS Test</h1>" | sudo tee /var/www/html/index.html
Enter fullscreen mode Exit fullscreen mode

Enabled HTTPS using mod_ssl and a temporary self-signed certificate. Apache was verified to be listening on port 443.

HTTP Test

Opened http://<EC2-public-IP>.

Result: ERR_CONNECTION_TIMED_OUT ā€” the request was blocked by the NACL.

HTTPS Test

Opened https://<EC2-public-IP>.

The browser displayed the expected self-signed certificate warning. After proceeding through the warning, the Apache page loaded successfully.

Evidence / Result

HTTP :80 ā†’ BLOCKED

HTTPS :443 ā†’ WORKED

This gave us a direct application-level demonstration of the NACL rules.

Problems / Troubleshooting

During Apache installation, the package manager initially timed out.

The problem turned out to be the NACL's stateless return traffic. The EC2 could send EC2:ephemeral-port ā†’ Internet:443, but the response returned to the EC2's ephemeral port and was not initially allowed.

Adding an inbound rule (1024-65535 ā†’ ALLOW) resolved the package installation.

This was an important real failure rather than a theoretical example.

Hands-On Takeaways

  • NACL rules apply at the subnet level.
  • Security Group and NACL rules both affect whether traffic reaches the instance.
  • A restrictive NACL must account for the return path.
  • Ephemeral ports become important when working with stateless NACLs.

3.Observing Network Traffic with VPC Flow Logs and CloudWatch

Objective

Enable VPC Flow Logs and use the resulting CloudWatch records to inspect real network traffic.

The goal was to move from simply configuring networking to observing what actually happened on the network.

What We Configured

Flow Logs were configured for all traffic, sent to CloudWatch Logs, with 1-minute aggregation, in log group /vpc/flow-logs/nacl-lab.

AWS created/used the required IAM service role for publishing Flow Logs to CloudWatch.

Traffic Generated

From the EC2:

curl http://checkip.amazonaws.com
curl https://checkip.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

SSH traffic had also been generated by connecting to the instance.

CloudWatch Investigation

CloudWatch ā†’ Logs ā†’ Log groups ā†’ /vpc/flow-logs/nacl-lab ā†’ Log stream

The records exposed fields such as Source IP, Destination IP, Source Port, Destination Port, Protocol, Action, Bytes, and Packets.

One observed HTTPS-related flow contained the important values:

Source IP:         13.213.253.85
Destination IP:    10.50.1.165
Source Port:       443
Destination Port:  32798
Protocol:          6 (TCP)
Action:            ACCEPT
Enter fullscreen mode Exit fullscreen mode

The reverse direction showed the corresponding flow back to the EC2's ephemeral port. We also observed REJECT records.

Evidence / Result

The Flow Logs successfully showed both ACCEPT and REJECT traffic, and allowed us to identify:

  • who was communicating
  • where traffic was going
  • which ports were involved
  • which protocol was used
  • whether the network flow was accepted or rejected

Hands-On Takeaways

  • Flow Logs are useful for troubleshooting network behavior.
  • They can expose rejected traffic that is otherwise difficult to understand.
  • They provide network-level evidence but do not replace application or authentication logs.
  • Flow Log records may take some time to appear because of aggregation.

4.Proving the Stateless NACL Gotcha with a Private EC2

Objective

Build a fresh private-network scenario specifically to demonstrate why a restrictive NACL can allow an outbound HTTPS request but still break the connection when the response returns.

The experiment was deliberately designed to fail first and then be fixed.

Architecture 

                         Internet
                            |
                           IGW
                            |
                    Public Route Table
                            |
                      NAT Gateway
                            |
                    Private Route Table
                            |
                     Private Subnet
                            |
                       Private EC2
Enter fullscreen mode Exit fullscreen mode

Lab CIDRs:

  • VPC: 10.60.0.0/16
  • Public subnet: 10.60.0.0/24
  • Private subnet: 10.60.1.0/24

Routing Configuration

  • Public route table: 0.0.0.0/0 ā†’ Internet Gateway
  • Private route table: 0.0.0.0/0 ā†’ NAT Gateway

The NAT Gateway was placed in the public subnet. The EC2 was placed in the private subnet with Auto-assign Public IP: Disabled.

Initial NACL

Outbound: 100 ā†’ HTTPS 443 ā†’ ALLOW, * ā†’ ALL ā†’ DENY
Inbound: * ā†’ ALL ā†’ DENY

The deliberate design was:

Allow the HTTPS request out, but do not allow the return traffic back in.

First Test ā€” Failure

From the private EC2:

curl https://checkip.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

The command hung. The request path was:

EC2 ā†’ NACL outbound ā†’ HTTPS 443 ALLOW ā†’ NAT Gateway ā†’ Internet

The response path returned to an ephemeral port:

Internet:443 ā†’ EC2:ephemeral-port ā†’ NACL inbound ā†’ DENY

Fix

Added the missing inbound rule: 100 ā†’ TCP 1024-65535 ā†’ ALLOW. The catch-all deny remained below it.

Second Test ā€” Success

Ran again:

curl https://checkip.amazonaws.com
Enter fullscreen mode Exit fullscreen mode

Result: 13.202.19.75 ā€” the HTTPS request now completed successfully.

Evidence / Result

Before: Outbound 443 ā†’ ALLOW. Return traffic ā†’ BLOCKED. curl ā†’ timeout.

After: Outbound 443 ā†’ ALLOW. Inbound ephemeral ports ā†’ ALLOW. curl ā†’ public IP returned.

Problems / Troubleshooting

The main issue was initially easy to misinterpret as a NAT or Internet problem.

Following the traffic path isolated the failure to the inbound NACL rule. The lab therefore became a useful troubleshooting exercise:

Request works? ā†’ check return path ā†’ NACL is stateless ā†’ check ephemeral ports

Hands-On Takeaways

  • A private EC2 can reach the Internet through a NAT Gateway without a public IP.
  • The private route table must point to the NAT Gateway.
  • The public route table used by the NAT Gateway must point to the IGW.
  • Restrictive NACLs must explicitly permit both sides of a connection.
  • Ephemeral ports are not an abstract networking detail; they directly affected the lab's outcome.
  • Following the packet path was more useful than randomly changing AWS resources.

Practical Reasoning Drills

P1.Longest Prefix Match ā€” Choosing the Most Specific Route

Objective

Use a route table with overlapping routes and determine which route AWS would select for different destination IPs.

Given:

Destination Target
10.0.0.0/16 Local
10.0.5.0/24 pcx-abc
0.0.0.0/0 igw-xyz

Results

  • 10.0.5.10 ā†’ 10.0.5.0/24 ā†’ pcx-abc (reason: /24 is more specific than /16 and /0)
  • 10.0.9.10 ā†’ 10.0.0.0/16 ā†’ Local (reason: it does not belong to 10.0.5.0/24, but it does belong to 10.0.0.0/16)
  • 52.1.1.1 ā†’ 0.0.0.0/0 ā†’ igw-xyz (reason: it does not match either 10.x.x.x route, so the default route is used)

Takeaway

More specific prefix wins: /24 > /16 > /0

p2.NAT Gateway Deletion ā€” Diagnosing a Blackhole Route

Objective

Work through a realistic failure where a NAT Gateway is deleted but private EC2 instances still depend on it for outbound Internet access.

Scenario

Private route table: 0.0.0.0/0 ā†’ NAT Gateway. NAT Gateway is deleted. The route can become 0.0.0.0/0 ā†’ NAT Gateway (BLACKHOLE).

Troubleshooting Path

Follow the traffic path:

EC2 ā†’ Security Group ā†’ NACL ā†’ Private Route Table ā†’ NAT Gateway ā†’ Public Route Table ā†’ IGW ā†’ Internet

In this exact scenario, the private route table is the high-value first check.

Takeaway

Don't immediately recreate the NAT Gateway. First identify where the traffic path breaks and check whether the route points to a valid target.

P3.Choosing Gateway vs Interface VPC Endpoints

Objective

Decide which endpoint type should be used when private workloads need AWS service access without relying on a public Internet path.

Service Endpoint Type
S3 Gateway Endpoint
DynamoDB Gateway Endpoint
Secrets Manager Interface Endpoint

Takeaway

Gateway endpoints integrate with route tables. Interface endpoints use private network interfaces/PrivateLink.

P4.Transit Gateway vs VPC Peering ā€” Scaling the Connectivity Model

Objective

Choose an architecture for an environment that has 3 VPCs today, expects 12 additional VPCs, and also needs VPN connectivity to the head office.

Peering Math

Full-mesh peering connections: N Ɨ (N - 1) / 2. For 3 VPCs ā†’ 3 connections. For 15 VPCs ā†’ 105 connections.

Decision

For this growth scenario, Transit Gateway is the more scalable direction. The model becomes:

VPC-A ā”€ā”
VPC-B ā”€ā”¤
VPC-C ā”€ā”¼ā”€ā”€ Transit Gateway ā”€ā”€ VPN ā”€ā”€ Head Office
VPC-D ā”€ā”¤
...    ā”˜
Enter fullscreen mode Exit fullscreen mode

Takeaway

The decision was based on connectivity scale and centralized routing, not simply on the number of VPCs that exist today.

VPC Flow Logs ā€” Investigating Suspicious SSH Traffic

Objective

Use Flow Logs as network-level evidence when investigating a suspected unauthorized SSH attempt against an EC2 instance.

Investigation

Enable Flow Logs for the relevant VPC/subnet and send records to CloudWatch. Look for TCP traffic on destination port 22, matching the target EC2's destination IP, from a suspicious source IP, and check whether the action was ACCEPT or REJECT.

The investigation can establish that a source IP attempted network-level SSH traffic.

Important distinction

An ACCEPT Flow Log does not prove that SSH authentication succeeded. If network traffic was accepted, OS-level SSH/authentication logs must be checked to determine whether a login actually occurred.

Problems and Lessons Across the Labs

EC2 Instance Connect vs Network Configuration

We encountered multiple browser-based EC2 Instance Connect issues.

The important lesson was to distinguish "can I manage/connect to the instance?" from "is the network experiment itself working?"

Temporary broad SSH rules were used only to remove management connectivity as a variable during the labs.

NAT Gateway and Private Subnet Design

The NAT Gateway placement initially required clarification. The working architecture established the relationship:

Private EC2 ā†’ Private Route Table ā†’ NAT Gateway ā†’ Public Route Table ā†’ IGW ā†’ Internet

The NAT Gateway therefore belongs in a subnet with a working path to the IGW.

Route Table Associations

A subnet association means:

The subnet uses this route table for its routing decisions.

This distinction became important when building separate public and private routing paths.

Duplicate Default Route

A private route table already containing 0.0.0.0/0 ā†’ IGW could not also contain another 0.0.0.0/0 ā†’ NAT Gateway. The existing default route had to be replaced with the correct target.

NACL Return Traffic

The most valuable troubleshooting lesson was that a restrictive NACL can allow the outbound request but still break the connection because the response arrives on an ephemeral port.

This was observed directly through failed commands and then fixed by adding TCP 1024-65535 ā†’ ALLOW.

Flow Logs as Evidence

Flow Logs provided actual network records instead of relying only on assumptions about what the network was doing. They were useful for identifying source, destination, ports, protocol, and ACCEPT/REJECT status.

Production-Relevant Takeaways from the Hands-On Work

These are intentionally limited to lessons that came directly from the practical work.

Separate public and private traffic paths

A production design should not expose application/database resources unnecessarily. A common model is:

Internet ā†’ ALB / Public tier ā†’ Private application tier ā†’ Private database tier

Plan NAT for both availability and cost

For multi-AZ workloads, a single NAT Gateway can create a cross-AZ dependency. NAT Gateway hourly and data-processing costs also make unnecessary outbound Internet traffic worth considering.

Use least-privilege Security Groups

The labs temporarily used broad SSH access to troubleshoot EC2 Instance Connect. Production should instead restrict management access to known sources or use an appropriate private management path.

Treat NACLs as an explicit network boundary

If restrictive NACLs are used, both directions of a connection need to be considered because NACLs are stateless.

Use VPC Flow Logs for visibility

Flow Logs can be useful for troubleshooting, rejected-traffic analysis, security investigations, and validating routing/security changes. But they provide network-level evidence, not application-level authentication evidence.

Design from the traffic requirement

A useful sequence is:

What traffic is required? ā†’ Where does it originate? ā†’ Where does it need to go? ā†’ What route provides that path? ā†’ What security controls permit it? ā†’ How will we observe/troubleshoot it?

Remaining VPC Practicals

The following two practicals remain to be completed:

Connecting VPCs with Peering, Then Scaling the Design with Transit Gateway

Build two VPCs, establish private connectivity using VPC Peering, then introduce a third VPC and move toward a Transit Gateway-based connectivity model.

The focus will be on the route changes and traffic paths required for VPC-to-VPC communication.

Private S3 Access with a Gateway Endpoint and Flow Log Verification

Add an S3 Gateway Endpoint to a VPC and verify the resulting traffic path using VPC Flow Logs.

The goal is to confirm how the endpoint changes the path used for S3 access and to distinguish that traffic from an Internet/NAT-based path.

Top comments (0)