Networking Series 7: VPC Monitoring with Flow Logs

#aws #awschallenge #monitoring #security

Introducing Today's Project!

What is Amazon VPC?
Amazon VPC is a secure virtual network in AWS, letting you control cloud resources and traffic flows.

How I used Amazon VPC in this project
Amazon VPC is a secure virtual network in AWS, letting you control cloud resources and traffic flows.

One thing I didn't expect in this project
I used Amazon VPC to create two virtual networks with EC2 instances, peer them for secure traffic flow, and monitor network activity via Flow Logs.

This project took 45 minutes to complete, including VPC setup, peering, EC2 launch, Flow Logs config, and traffic analysis.

In the first part of my project

Step 1 - Set up VPCs
Here, I'll be launching EC2 instances in each VPC to create real traffic between them, essential for testing peering and enabling network monitoring later.

Step 2 - Launch EC2 instances
I started this step by launching EC2 instances in each VPC. This lets us test the VPC peering setup and generate traffic, which is essential for monitoring with VPC Flow Logs later.

Step 3 - Set up Logs
I'm setting up VPC Flow Logs to track network traffic in and out of my VPCs. This helps me monitor activity, spot issues, and keep things secure. I'll store the logs in Amazon CloudWatch to analyze patterns and better understand my networkʼs health.

Step 4 - Set IAM permissions for Logs
I'm giving VPC Flow Logs permission to send traffic data to CloudWatch so I can monitor and analyze my subnetʼs activity. This helps improve security, performance, and visibility.

Multi-VPC Architecture

I started my project by launching two VPCs, each with one public subnet. These subnets enabled me to deploy EC2 instances and prepare for testing VPC peering and monitoring with Flow Logs.
The CIDR blocks for VPCs 1 and 2 are 10.1.0.0/16 and 10.2.0.0/16. They have to be unique because overlapping IPs would cause routing issues when the VPCs communicate through peering.

I also launched EC2 instances in each subnet
My EC2 instancesʼ security groups allow all ICMP from the other VPC's CIDR block and SSH traffic from any IP address. This is because Iʼll perform ping tests between VPCs and use EC2 Instance Connect, which requires SSH access.

Logs

Logs are records of VPC network traffic stored in CloudWatch, used to analyze data flow, track activity, and enhance security monitoring.

Log groups are folders in Amazon CloudWatch that organize related logs from the same source or application. They store VPC Flow Log data regionally, helping you manage and analyze network activity efficiently.

IAM Policy and Roles

I created an IAM policy because VPC Flow Logs needed permission to publish traffic data to CloudWatch for monitoring, troubleshooting, and analysis.

I also created an IAM role because VPC Flow Logs need permissions and a trusted entity to publish network traffic data to CloudWatch for monitoring.

A custom trust policy is a JSON config that defines which AWS service like vpc-flow
logs.amazonaws.com can assume a role to push logs to CloudWatch.

In the second part of my project

Step 5 - Ping testing and troubleshooting
I'm sending test messages from Instance 1 to Instance 2 to confirm our peering connection works and to generate traffic for analyzing network activity with Flow Logs later on.

Step 6 - Set up a peering connection
In Step 6, I'm setting up VPC peering so EC2s in both VPCs can talk using private IPs essential for monitoring network traffic with Flow Logs.

Step 7 - Analyze flow logs
I'm reviewing VPC 1ʼs public subnet flow logs to uncover insights into traffic sources, data volume, and blocked connections for better network security.

Connectivity troubleshooting

My first ping test between my EC2 instances had no replies, which means they arenʼt connected likely due to security group or routing issues.

I could receive ping replies if I ran the ping test using the other instance's public IP address, which means public routing works but private network setup needs fixes.

Connectivity troubleshooting

Looking at VPC 1's route table, I identified that the ping test with Instance 2's private address failed because no route to VPC 2's CIDR was set.

To solve this, I set up a peering connection between my VPCs
I also updated both VPCs' route tables so that each one could route traffic to the other's CIDR, making private IP communication possible.

I received ping replies from Instance 2's private IP address! This means my peering and route table updates worked, private traffic is flowing.

Analyzing flow logs

Flow logs tell us about VPC traffic details like source/destination IPs, ports, protocol, direction, and whether the traffic was accepted or rejected, all of which help monitor and secure your network.

For example, the flow log I've captured tells us which traffic was allowed or blocked, its origin and destination, and how it traveled, helping us assess the VPCʼs security and connectivity.

Logs Insights

Logs Insights is a CloudWatch feature for querying VPC Flow Logs to troubleshoot traffic, monitor performance, and uncover security issues.

I ran the query “Top 10 byte transfers by source and destination IP addresses.” This query analyzes VPC Flow Logs to find which IP pairs transferred the most data across the network.