DEV Community

Cover image for 🌐 AWS 149: Multi-VPC Log Aggregation - Peering, IAM Roles, and S3 Integration
Hritik Raj
Hritik Raj

Posted on

🌐 AWS 149: Multi-VPC Log Aggregation - Peering, IAM Roles, and S3 Integration

#aws #vpc #s3 #100daysofcloud

🚀 Secure Pipelines: Aggregating Logs across Private and Public VPCs

Hey Cloud Architects 👋

Cloud Networking

Welcome to Day 49 of the #100DaysOfCloud Challenge!
Today, we are building a sophisticated log management architecture. The Nautilus DevOps team needs to move logs from an isolated private environment to a public aggregation server, and finally to long-term storage in S3. This requires mastering VPC Peering, Route Tables, and IAM Instance Profiles to ensure data flows securely across different network boundaries.

This task is part of my hands-on practice on the KodeKloud Engineer platform, which provides excellent scenarios for complex networking.

🎯 Objective

  • Provision a Public VPC (devops-pub-vpc) with an Internet Gateway and proper routing.
  • Launch a public aggregator instance (devops-pub-ec2) with an IAM role for S3 access.
  • Establish a VPC Peering Connection between the private and public VPCs.
  • Update Route Tables in both VPCs to allow traffic via the peering link.
  • Automate log transfers using cron jobs to move boots.log from Private EC2 ➔ Public EC2 ➔ S3 Bucket.

💡 Why VPC Peering for Log Collection?

VPC Peering allows you to connect two VPCs using private IP addresses. This means traffic between your instances never traverses the public internet, reducing latency and increasing security. It's the standard way to connect a "Management" or "Logging" VPC to your "Production" VPCs.

🔹 Key Concepts

  • VPC Peering: A networking connection between two VPCs that enables you to route traffic between them privately.

  • IAM Instance Profile: Attaching a role directly to an EC2 instance so it can interact with S3 without needing hardcoded credentials.

  • CIDR Routing: Explicitly telling each VPC how to reach the other by adding destination CIDR blocks to the Route Tables.

🛠️ Step-by-Step: Infrastructure & Log Workflow

🔹 Phase A: Building the Public Infrastructure

We need a landing zone for our logs that has internet access to reach the S3 service.

  • VPC Setup: Create devops-pub-vpc and attach an Internet Gateway.
  • Public Route Table: Configure devops-pub-rt with a route 0.0.0.0/0 targeting the IGW.
  • Public Instance: Launch devops-pub-ec2 using the devops-key.pem.
  • IAM Role: Create devops-s3-role with PutObject permissions and attach it to the public instance.

🔹 Phase B: Establishing the Peering Connection

Now we "bridge" the two VPCs so they can communicate over private IPs.

  • Peering Request: Create devops-vpc-peering from the private VPC to the public VPC.
  • Acceptance: Accept the peering request in the console.
  • Route Table Updates: * In devops-priv-rt, add a route to the Public VPC CIDR via the Peering Connection.
    • In devops-pub-rt, add a route to the Private VPC CIDR via the Peering Connection.

🔹 Phase C: Log Transfer Configuration (Private ➔ Public)

On the devops-priv-ec2 instance, we set up the first leg of the journey.

  • SSH Setup: Ensure the public key is trusted between instances.
  • Cron Job: Set a cron task to periodically send the log file.
    • * * * * * scp -i ~/.ssh/devops-key.pem /var/log/boots.log ubuntu@<PUBLIC_INSTANCE_PRIVATE_IP>:/tmp/

🔹 Phase D: Final Upload (Public ➔ S3)

On the devops-pub-ec2 instance, we handle the final archival.

  • S3 Bucket: Create the private bucket devops-s3-logs-28057.
  • Cron Job: Set a cron task to push the local file to S3 using the AWS CLI.
    • * * * * * aws s3 cp /tmp/boots.log s3://devops-s3-logs-xxxx/devops-priv-vpc/boot/boots.log

✅ Verify Success

  • Peering Status: The connection is marked as "Active" in the VPC console.
  • Security Check: The devops-priv-ec2 can ping or SSH into devops-pub-ec2 using its Private IP.
  • Final Archival: Confirm the file exists in S3.

📝 Key Takeaways

  • 🚀 Zero Credentials: By using an IAM Role on the public EC2, we avoided storing secret keys on the server.
  • 🛡️ Network Isolation: The private instance remains private; it only communicates with our trusted public aggregator over the peering link.
  • 📦 Standardized Paths: Using bucket prefixes (folders) like devops-priv-vpc/boot/ makes it easier to manage logs as your infrastructure grows.

🚫 Common Mistakes

  • Overlapping CIDRs: VPC Peering will fail if both VPCs use the exact same IP range (e.g., both using 10.0.0.0/16).
  • Missing Route Back: Remember that routing is a two-way street; if you only update the private RT, the public instance won't know how to send a response back.
  • Security Group Blocks: Ensure the Security Group on the public instance allows inbound SSH/SCP traffic from the Private VPC's CIDR range.

🌟 Final Thoughts

Building a multi-tier logging architecture is a hallmark of professional cloud engineering. You've successfully implemented a system that is secure, automated, and scalable. This setup ensures that critical logs are always backed up while maintaining strict network isolation for your private assets.

🌟 Practice Like a Pro

Want to master VPC Peering and secure data pipelines? Get hands-on experience here:
👉 KodeKloud Engineer - Practice Labs

🔗 Let’s Connect

Top comments (0)