DEV Community: Ahmed Shendy

Aurora Serverless v2: When "I/O Optimized" Actually Costs You More

Ahmed Shendy — Thu, 25 Dec 2025 11:48:05 +0000

The Request

It started with a simple Slack message from my manager:

"We've been running on **Aurora I/O Optimized* for a few months. Can we save money by converting back to Standard?"*

We had enabled I/O Optimized previously because the promise of "Free IOPS" sounded perfect for our high-traffic workloads. But without recent validation, we were flying blind.

This is the story of how I validated the costs, the "CloudWatch Trap" I fell into, and why we ultimately decided to revert to Standard—saving the company up to 26% on our database bill.

The Difference: It’s a Billing Flag, Not Hardware

My first step was checking if reverting would hurt performance. I dug into the AWS Documentation and confirmed that switching between Standard and I/O Optimized does not change the underlying hardware. It is purely a billing configuration.

Here is the trade-off (based on eu-west-1 pricing):

Component	Standard Model	I/O Optimized Model (Our State)
Compute (ACUs)	~$0.14 / hr	~$0.19 / hr
Storage	~$0.11 / GB	~$0.25 / GB
I/O Requests	~$0.22 / Million	$0.00

The math seems simple: Were we actually consuming enough IOPS to justify the massive premium we were paying for Compute and Storage?

The "CloudWatch Trap"

To answer that question, I tried to write a Python script to pull metrics from CloudWatch. I calculated the ServerlessDatabaseCapacity (ACUs) using the "Average" statistic over 30 days.

The Result: The calculations was not accurate

Why? Granularity.

Aurora Serverless scales second-by-second. If your database spikes to 60 ACUs for 30 seconds and then drops, the AWS billing engine captures that cost perfectly. CloudWatch, however, averages that spike into the surrounding timeframe (often 1-5 minutes), smoothing out the data.

This makes the usage look much lower (and cheaper) than it actually is.

Lesson Learned: Never use CloudWatch averages to calculate your total monthly bill. It is a monitoring tool, not an accounting tool.

The Solution: The "10-Day Snapshot" Method

Since I couldn't trust CloudWatch for the total ACU cost, and AWS Cost Explorer is limited to 14 days of granular resource data, I developed a hybrid approach: The 10-Day Snapshot.

The logic is simple: If we are losing money over a representative 10-day period, we are losing money over the month.

Step 1: The Source of Truth (Cost Explorer)

I used Cost Explorer to get the exact billed amount for Compute (ACUs) for a specific window (e.g., Dec 11 - Dec 20).

Dimension: Resource
Filter:
Usage Type: EU-Aurora:ServerlessV2IOOptimizedUsage (ACU-Hr) (Since we were currently Optimized)
Resource: Select the relational databse then selct the DB instances
Result: Get the exact amount of the used I/O.

Step 2: The Variable Costs (CloudWatch)

Since Cost Explorer doesn't easily split out "Storage" vs. "I/O" quantity for a specific custom window, I used the CloudWatch Console to get the raw usage data for the exact same 10-day period.

Here is exactly how I got the numbers:

Navigate to Metrics:
- Go to CloudWatch > All metrics.
- Select AWS/RDS > DBClusterIdentifier.
- Search for your cluster ID (e.g., client-db-01).
Select the Metrics:
- Check VolumeBytesUsed (for Storage).
- Check VolumeReadIOPS (for I/O).
- Check VolumeWriteIOPS (for I/O).
Set the Timeframe:
- Click the date picker in the top-right.
- Select Custom > Absolute.
- Choose the exact same 10-day window you used in Cost Explorer.
Configure the Calculations (Graphed Metrics Tab):
- For Storage: Set the Statistic to Average.
  - Math: Add the expression m1/1024/1024/1024 to convert raw bytes into GB.
- For I/O: Set the Statistic to Sum.
  - Math: Add the expression m2+m3 to add Read and Write IOPS together for the Total IOPS.
- Important: Ensure the "Period" is set to 30 Days (or a custom value like 864000 seconds) so that CloudWatch gives you one single number for the entire duration, rather than a line graph.

Step 3: The Comparison

Now we have the exact numbers for Total I/O, the Storage, and consumed ACUs.
I put the numbers into a spreadsheet to compare our current costs vs. what we would pay if we switched to Standard.

Scenario A: I/O Optimized (Actual Cost)

ACU Cost: (Amount of used ACUs Cost Explorer) * 0.19
Storage Cost: Avg GB * $0.248 * (10/30)
IOPS Cost: $0

Scenario B: Standard (Projected Cost)

ACU Cost: (Amount of used ACUs Cost Explorer) * 0.14
Storage Cost: Avg GB * $0.11 * (10/30) (The cheaper storage rate)
IOPS Cost: (Total IOPS) / 1,000,000 * $0.22

The Verdict

The results were eye-opening. We were overpaying on every single cluster.

Conclusion

We immediately scheduled maintenance windows and converted all clusters back to Aurora Standard.

The validation proved that while "Optimized" sounds better, sometimes the "Standard" option is the true hero for your budget. If you are currently running on I/O Optimized, take a 10-day snapshot and check your storage costs—you might be surprised by what you find.

DNS Failures in EKS? The Real Bottleneck Was AWS Network Limits

Ahmed Shendy — Thu, 18 Dec 2025 06:51:04 +0000

During the DNS investigation, I initially focused on CoreDNS and NodeLocal DNS metrics.

The real breakthrough came when I started correlating DNS failures with AWS instance-level network limits.

The most useful signals came from network allowance metrics exposed by the EC2 ENA driver via ethtool.

AWS Network Allowance Metrics

The following metrics represent network limits enforced at the EC2 instance level.

ethtool_linklocal_allowance_exceeded

Packets dropped because traffic to link-local services exceeded the packets-per-second (PPS) limit.

This directly affects DNS, IMDS, and Amazon Time Sync. If you found this value above zero, you can 1) try increasing the number of CoreDNS replicas or 2) implement NodeLocal DNSCache or 3) check the ndots as mentioned in this post The Hidden DNS Misconfiguration That Was Killing Performance in Our EKS Cluster (and How We Fixed it)
ethtool_conntrack_allowance_available

Remaining number of connections that can be tracked before reaching the instance’s connection-tracking limit.

Supported on Nitro-based instances only.
ethtool_conntrack_allowance_exceeded

Packets dropped because the connection-tracking limit was exceeded and new connections could not be established.
ethtool_bw_in_allowance_exceeded

Packets queued or dropped because inbound aggregate bandwidth exceeded the instance limit.
ethtool_bw_out_allowance_exceeded

Packets queued or dropped because outbound aggregate bandwidth exceeded the instance limit.
ethtool_pps_allowance_exceeded

Packets queued or dropped because the bidirectional packets-per-second (PPS) limit was exceeded.

All *_allowance_exceeded metrics should ideally remain zero.

Any sustained non-zero value indicates a networking bottleneck at the instance level.
For all metrics except the link-local you can solve it by changing the instance size or type to get a higher network bandwidth or work on reducing the load on this instance.

Capturing Network Metrics in EKS

These metrics are exposed by the EC2 ENA driver via ethtool, collected by node exporter, scraped by Prometheus, and visualized in Grafana.

On Amazon Linux EKS nodes, ethtool is installed by default.

To collect these metrics, the ethtool collector must be enabled in node exporter.

Enable ethtool Collector in node exporter

Add the following arguments to the node exporter container.

containers:
- args:
  - --collector.ethtool
  - --collector.ethtool.device-include=(eth|em|eno|ens|enp)[0-9s]+
  - --collector.ethtool.metrics-include=.*

After applying this change, the metrics will become available in Prometheus and Grafana.

Building the Grafana Dashboard

All panels are time series panels, built per node, to help correlate network saturation with DNS errors or latency.

Available Connection Tracking Capacity

The metric exported by node exporter is:

node_ethtool_conntrack_allowance_available

It represents the current number of connections that can still be tracked on each node.
PromQL query:

node_ethtool_conntrack_allowance_available{job="node-exporter"}

Packets Dropped Due to Conntrack Exhaustion

The metric node_ethtool_conntrack_allowance_exceeded is a counter that increases over time.
To calculate packet drops per second, use the rate() function.

sum by (instance) (
  rate(
    node_ethtool_conntrack_allowance_exceeded{job="node-exporter"}[1m]
  )
)

and the panel will be like this

Other Network Allowance Exceeded Metrics

Add the following panels using the same counter-to-rate approach.

node_ethtool_bw_in_allowance_exceeded
node_ethtool_bw_out_allowance_exceeded
node_ethtool_pps_allowance_exceeded
node_ethtool_linklocal_allowance_exceeded

Each panel shows packets dropped per second per node.

Full Grafana dashboard JSON:

Network limits dashboard

Final Insight

All allowance exceeded metrics are tied to EC2 instance sizing, with one exception.

Link-local traffic has a fixed limit of 1024 packets per second, regardless of instance size.

This explains why DNS can fail even when CPU, memory, and pod-level metrics look healthy.

The bottleneck exists below Kubernetes, at the EC2 networking layer.

Takeaway

If you are debugging intermittent DNS failures on EKS, do not stop at CoreDNS metrics.

Always inspect instance-level network allowances.

🚀 The Hidden DNS Misconfiguration That Was Killing Performance in Our EKS Cluster (and How We Fixed it)

Ahmed Shendy — Sat, 06 Dec 2025 14:29:38 +0000

Why I Wrote This Article — The Incident That Sparked Everything

This article didn’t come from curiosity.

It came from pain.

One morning, I received a message from the DevOps team:

“Some services are failing to resolve hostnames again —

we’re getting Temporary failure in name resolution.”

And this wasn’t the first time.

It had happened before — randomly, unpredictably, quietly causing latency and connection failures.

As the new Cloud Architect, the responsibility landed on my desk:

“We need this fixed forever — no more band-aids.”

So I started investigating.

(NOTE: I’ll publish a second article soon with the full debugging journey.)

Nothing looked broken at first.

Pods healthy. Cluster stable. CoreDNS replicas are running.

No crashes. No alerts.

But something felt off — so I went deep into metrics.

And there it was:

CoreDNS wasn’t resolving —

it was drowning in NXDOMAIN.

Thousands per second.

It wasn’t an outage.

It was a storm — a silent performance killer - we have around 80% of the DNS queries with response code NXDOMAIN

And the storm had one surprising source…

🕵️ The Real Breakthrough — It Was One Hostname

When I traced DNS traffic volume by hostname,

The data made me stop.

It wasn’t many hostnames.

It wasn’t dozens.

It was one about from 80% to 90% of the DNS queries are related to only one host.

Our RabbitMQ endpoint — the heart of our event-driven system — contained only four dots:

rabbitmq.eu-west-1.aws.company.production

And with Kubernetes default ndots=5,

This meant the resolver didn’t treat it as a fully qualified domain.

Instead, Kubernetes expanded it through every search domain in the pod:

rabbitmq.eu-west-1.aws.company.production.default.svc.cluster.local ❌ NXDOMAIN
rabbitmq.eu-west-1.aws.company.production.svc.cluster.local ❌ NXDOMAIN
rabbitmq.eu-west-1.aws.company.production.cluster.local ❌ NXDOMAIN
rabbitmq.eu-west-1.aws.company.production. ❌ NXDOMAIN
rabbitmq.eu-west-1.aws.company.production ✅ finally correct

For each attempt:

A lookup ❌
AAAA lookup ❌

🟡 4 to 8 extra DNS queries for every single valid lookup

RabbitMQ is used everywhere — messaging, telemetry, queues, notifications.

So every millisecond meant more queries → more NXDOMAIN → more pressure.

We weren’t resolving DNS.

We were manufacturing DNS traffic.

⚡ The One-Character Fix That Saved Us

Under pressure and needing a fast mitigation,

I tried a tiny change that felt almost silly:

I added a trailing dot to the hostname.

Just one dot:

rabbitmq.eu-west-1.aws.company.production.

That trailing dot tells Linux resolver:

“This is a fully qualified domain.

Do not apply search paths.”

The effect was instant:

❌ NXDOMAIN flood dropped immediately
💡 CoreDNS CPU reduced by ≈50%
⚡ Lookup performance improved ~5x
🧘 Zero failures since
😊 Developers finally stopped pinging me about DNS issues

We didn’t scale DNS.

We didn’t tune CoreDNS.

We didn’t rewrite applications.

We removed unnecessary work.

One dot → stability restored.

We later applied additional DNS optimizations to handle even larger query loads

— more on that in the next article.

🔍 The Root Cause: `ndots` in `/etc/resolv.conf`

Every Kubernetes pod has a resolver config like:

search default.svc.cluster.local svc.cluster.local cluster.local eu-west-1.compute.internal
nameserver 172.20.0.10
options ndots:5

The ndots value controls:
How many dots must exist in a hostname before it is treated as an absolute FQDN.

If hostname dot-count < ndots → search domains appended

This Kubernetes default exists to support internal service discovery:

service → service.default.svc.cluster.local → resolves successfully
But for external hostnames?

🚩 Disaster waiting to happen.

🧪 Benchmark — Measured Results
I have used this python script to test the ndots effect

#!/usr/bin/env python3
import argparse
import asyncio
import time

import dns.asyncresolver


async def main() -> None:
    parser = argparse.ArgumentParser(
        description="Measure DNS lookup time for multiple queries using current resolv.conf (ndots/search)."
    )
    parser.add_argument("host", help="Hostname to resolve (bare name to exercise ndots/search)")
    parser.add_argument(
        "-n",
        "--queries",
        type=int,
        default=100,
        help="Number of concurrent queries to issue (default: 100)",
    )
    parser.add_argument(
        "-t",
        "--timeout",
        type=float,
        default=2.0,
        help="Per-query timeout in seconds (default: 2.0)",
    )
    args = parser.parse_args()

    resolver = dns.asyncresolver.Resolver()  # uses /etc/resolv.conf (ndots/search respected)
    resolver.timeout = args.timeout
    resolver.lifetime = args.timeout
    resolver.use_edns = False

    async def one_query() -> None:
        try:
            await resolver.resolve(args.host, "A", search=True)
        except Exception:
            # Ignore failures; we only care about timing behavior.
            pass

    tasks = [asyncio.create_task(one_query()) for _ in range(args.queries)]
    start = time.monotonic()
    await asyncio.gather(*tasks)
    elapsed = time.monotonic() - start
    print(f"{args.queries} queries for '{args.host}' in {elapsed:.3f}s ({elapsed/args.queries:.4f}s/query)")


if __name__ == "__main__":
    asyncio.run(main())

Python async DNS resolver test:

Before fix (no trailing dot)

python ndots_async_bench.py rabbitmq.eu-west-1.aws.xxxxx.production -n 100
100 queries for 'rabbitmq.eu-west-1.aws.xxxxx.production' in 2.207s (0.0221s/query) 

python ndots_async_bench.py rabbitmq.eu-west-1.aws.xxxxx.production -n 10
10 queries for 'rabbitmq.eu-west-1.aws.xxxxx.production' in 0.302s (0.0302s/query)

100 queries → 2.207s (0.0221 s/query)
10 queries → 0.302s (0.0302 s/query)

After fix (trailing dot → FQDN)
rabbitmq.eu-west-1.aws.company.production.

python ndots_async_bench.py rabbitmqc.eu-west-1.aws.xxxxx.production. -n 100
100 queries for 'rabbitmqc.eu-west-1.aws.xxxxx.production.' in 0.399s (0.0040s/query) 
python ndots_async_bench.py rabbitmqc.eu-west-1.aws.xxxxx.production. -n 10 
10 queries for 'rabbitmqc.eu-west-1.aws.xxxxx.production.' in 0.095s (0.0095s/query)

100 queries → 0.399s (0.0040 s/query)
10 queries → 0.095s (0.0095 s/query)
🚀 DNS became ~5x faster
🧨 NXDOMAIN traffic dropped nearly in half

🛠 Fixing the Problem (Two Options)

1️⃣ Use Fully Qualified Domain Names with a trailing dot

Examples:

api.company.internal.
googleapis.com.
rabbitmq.eu-west-1.aws.company.

✔ Easiest fix
✔ No Kubernetes changes
✔ Zero search-domain expansion
✔ Best performance

2️⃣ Reduce ndots for external workloads

spec:
  dnsConfig:
    options:
      - name: ndots
        value: "2"

AWS docs state:

You can reduce the number of requests to CoreDNS by lowering the ndots option of your workload or fully qualifying your domain requests by including a trailing . (e.g. api.example.com. ).

📘 References

Kubernetes Docs
https://kubernetes.io/docs/concepts/services-networking/dns-pod-service/

AWS EKS
https://docs.aws.amazon.com/eks/latest/best-practices/scale-cluster-services.html#:~:text=Reduce%20external%20queries%20by%20lowering%20ndots

Linux Resolver
https://man7.org/linux/man-pages/man5/resolv.conf.5.html

🎯 Final Thought

Sometimes the biggest reliability problems
come from the smallest defaults.

ndots=5 is perfect for Kubernetes internal services…
but for external hostnames it can quietly overwhelm DNS
and drag performance down across the entire cluster.

One dot fixed everything.

Fix it once → enjoy peace and performance forever.

💬 If you'd like Part 2 (the full debugging journey — how I traced and proved the root cause), comment below:

Show me the debugging story

AWS VPC From Zero to Hero Series: 2, Build, Secure, and Monitor Networks on AWS

Ahmed Shendy — Fri, 16 Feb 2024 14:25:22 +0000

This is a series of hands-on labs that start from beginner level to advanced level to give you a real understanding of AWS VPC.

Overview

In this interactive lab, you'll construct and peer three VPCs—labeled A, B, and C to illustrate the escalating complexity of configuration and routing that accompanies the addition of multiple VPCs, gateways, and associated resources. You will then transition the network architecture to a hub-and-spoke design using AWS Transit Gateway. Additionally, you'll set up network monitoring and implement security measures to safeguard your resources.

Topics covered

By the end of this lab, you will be able to do the following:
• Set up routing between VPCs.
• Select the appropriate connectivity options for an environment.
• Capture network traffic information (metadata) with VPC flow logs.
• Configure monitoring for networking statistics and metrics.
• Filter network traffic with Network Access Lists (NACLs), and Security Groups (SG).

Prerequisites

An active AWS account.
A user that has access to the AWS console with the needed permissions.
knowledge about basic networking concepts (such as IP Addressing, CIDR notation, and routing), an understanding of basic cloud operations, and familiarity with navigating in the AWS Management Console.

Duration

This lab requires approximately 120 minutes to complete.

Task 1: Build a Multi-VPC Architecture

you will provision three logically isolated sections of the AWS Cloud into VPCs. In addition to using multiple VPCs, you will span the environment across multiple Availability zones (AZ) within a Region. After creating these VPCs, you will launch Amazon Elastic Compute Cloud (Amazon EC2) instances into the virtual networks that you define.
Each of the VPCs you create requires an internet gateway (IGW). This is a horizontally scaled, redundant, and highly available VPC component that allows communication between instances in your VPC and the internet. It therefore imposes no availability risks or bandwidth constraints on your network traffic.
You will finish this task by observing that by default, EC2 instances in different VPCs are not able to communicate with each other using private IP addresses.

TASK 1.1: CREATE THREE VPCS WITH PRIVATE SUBNETS

In this task, you will be creating three VPCs with private subnets. Each VPC will have subnets in two Availability Zones within the Region.
Table 1. The below table shows IPv4 CIDR allocations for VPCs and AZs

Task 1.1.1: Create VPCs

Our first step is to create VPCs with non-overlapping CIDR blocks.

We will not create the VPC as we did in lab 1 using VPC and more option, In this lab we will use VPC only option to get more control on the created resources.
1- Go to VPC service
2- Choose Create VPC and configure the following:

3- Follow the same steps to create VPC B and VPC C; refer to Table 1.
4- Now we have these 3 VPCs

Task 1.1.2: Create Subnets

For each VPC, we will create two subnets - one per availability zone (AZs name could differ - depending on your chosen region).

5- On the left navigation pane, under Virtual private cloud, choose Subnets.
6- Choose Create subnet to create subnets for VPC A configure the following:
• VPC ID: VPC A

• Subnet name: VPC A - AZ1
• Availability Zone: Select the first Availability Zone in the list. (Do not choose No Preference)
• IPv4 CIDR block: 10.0.0.0/24
• Tags: Accept proposed Tags

7- Choose Add new subnet to add Subnet 2 of 2 into AZ2 and configure the following:
• Subnet name: VPC A - AZ2
• Availability Zone: Select the second Availability Zone in the list. (Do not choose No Preference)
• IPv4 CIDR block: 10.0.1.0/24
• Tags: Accept proposed Tags.

8- Choose Create subnet.

9- Repeat the steps above to create subnets for VPC B and VPC C; refer to the below table for copying subnet name and subnet CIDR allocations.

After you finish the task, six new subnets should be available.

Task 1.1.3: Deploy Internet Gateways

In this section, you will deploy one Internet Gateway (IGW) per VPC. You need an Internet Gateway to establish outside connectivity to EC2 instances in VPCs.
10- On the left navigation pane, under Virtual private cloud, choose Internet gateways.
11- Choose Create internet gateway and configure the following:
• Name tag: VPC A - IGW
• Tags: Accept proposed Tags

12- Choose Create internet gateway.
13- From Actions Select Attach to a VPC.

14- For Available VPCs, choose VPC A.

15- Choose Attach internet gateway.
16- Repeat the above steps to create and attach IGWs
VPC B - IGW to VPC B and
VPC C - IGW to VPC C.
You should now have an IGW for the default VPC and three newly created IGWs available.

Task 1.1.4: Update Routing Tables

To utilize newly created Internet Gateways, you need to update VPC routing tables to point the default routes to these IGWs.
17- On the left navigation pane, under Virtual private cloud, choose Route tables.
Please make sure that you DO NOT create a new Route table. Please follow the next steps to rename the existing Main Route table.
18- Select the Route table that belongs to VPC A.

19- Edit the name to be: VPC A Route Table

20- Make the same for the other 2 route tables
21- Select VPC A Route Table.
22- Choose the Routes tab in the lower half of the page.

There is one route in your route table that allows traffic within the 10.0.0.0/16 network to flow within the network, but it does not route traffic outside of the network.
23- Choose Edit routes.
24- Choose Add route and then configure the following:

25- Choose Save changes.
26- Repeat the above steps to add the default route 0.0.0.0/0 to the Route tables
VPC B Route Table for VPC B and
VPC C Route Table for VPC C.

TASK 1.2: DEPLOY EC2 INSTANCES IN VPC’S

You will deploy one EC2 instance per VPC and demonstrate that, by default, VPCs provide network isolation. The EC2 instances should not be able to reach each other using ping (a common diagnostic tool) before the next task is completed.
27- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for EC2 and then choose EC2 from the list.

28- On the left navigation pane, choose Instances.
29- Choose Launch instances and configure the following:

Name and tags: Under Name, enter EC2 VPC A - AZ1
Application and OS Images (Amazon Machine Image): -- Quick Start: Select Amazon Linux ○ Amazon Machine Image (AMI): Amazon Linux 2 AMI (HVM), SSD Volume Type ○ Architecture: 64-bit (x86) • Instance type: t2.micro • Key pair (login): Proceed without a key pair (Not recommended) You will use SSM session Manager to access the shell running on the EC2, so a key pair is not needed in the lab. 30- Scroll down to Network settings, choose Edit, and configure the following: • VPC - required: VPC A • Subnet: VPC A - AZ1 • Auto-assign public IP: Enable • Firewall (security groups): ○ Choose Create security group ○ Security group name - required: VPC A EC2 Security Group ○ Description - required: Allow ICMP Traffic • Inbound security groups rules: ○ Type: Select All ICMP - IPv4 from the dropdown instead of SSH ○ Source type: Custom ○ Source: 10.0.0.0/8 It is not a best practice to have wide open Security groups that allow anyone/everyone such as 0.0.0.0/0, to limit access to only what is required. 31- Scroll down to Advanced details and configure the following: • IAM instance profile: From the drop-down list, select AmazonSSMRoleForInstancesQuickSetup 32- Choose Launch instance. 33- Choose View all instances to display all the instances launched. 34- Occasionally choose the console refresh button and wait for Public Instance to display the Instance state as Running and wait for Status check to pass 2/2 checks passed. The Amazon EC2 instance named Public Instance is initially in a Pending state. The instance state then changes to Running indicating that the instance has finished booting. 35- Launch 2 more EC2 instances and assign them names accordingly: i. Second EC2 instance in VPC B with the following configuration • Name and tags: EC2 VPC B - AZ1 • Network settings: ○ VPC - required: VPC B ○ Subnet: VPC B - AZ1 • Firewall (security groups): ○ Security group name - required: VPC B EC2 Security Group ○ Description - required: Allow ICMP Traffic Except for the above details, keep all other configurations same as first EC2 instance. ii. Third EC2 instance in VPC C with the following configuration • Name and tags: EC2 VPC C - AZ1 • Network settings: ○ VPC - required: VPC C ○ Subnet: VPC C - AZ1 • Firewall (security groups): ○ Security group name - required: VPC C EC2 Security Group ○ Description - required: Allow ICMP Traffic Except for the above details, keep all other configurations same as first EC2 instance. For all EC2 instances, update the Security Group rules under the Security tab to allow ICMP traffic. There is no need to have a rule for SSH since SSM Session Manager will be used to connect to the terminal. After a few minutes, you should now have 3 EC2 instances in the Running state. 36- Copy the Private IPv4 addresses assigned to EC2 instances by choosing an Instance and navigating to the Details tab. 37- Copy the following table with IP information to your favorite notepad tool, and populate the private IP addresses that you copied in the previous step. Table 2. Populate the following table with EC2 instances' private IP Addresses:

TASK 1.3: TEST INTER-VPC COMMUNICATION BETWEEN EC2 INSTANCES

Now that the EC2 instances are available in each VPC, you will use ping (a common diagnostic tool) to verify that the instances cannot communicate demonstrating how VPCs provide isolation.

38- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for EC2 and then choose EC2 from the list.
39- On the left navigation pane, choose Instances.
40- Select the EC2 VPC A - AZ1 instance.
41- Choose Connect from the navigation bar.

42- With Session Manager tab selected, choose Connect.

A terminal session should open in a new browser tab.
43- From the EC2 VPC A - AZ1 instance in VPC A, try pinging the private IP addresses of:

EC2 VPC B - AZ1 instance in VPC B and
EC2 VPC C - AZ1 instance in VPC C. Check if those addresses are pingable. ping 10.1.0.x Refer to Table 2. for the private IP addresses of the instances.

TASK 1.4: SETUP VPC PEERING

The end of the previous task demonstrated that EC2 instances in different VPCs cannot reach each other on their private IP addresses. but, we want to ensure that all the traffic between VPCs is on the AWS Backbone, and not traversing the internet. You will achieve this using VPC peering. VPC Peering is a connection between two VPCs that enables you to route your traffic between them.
In this task, you will establish VPC peering links between VPC A and VPC B, as well as VPC A and VPC C. Note that all three VPCs have non-overlapping CIDRS. You cannot create a VPC peering connection between VPCs with matching or overlapping IPv4 CIDR blocks.

44- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for VPC and then choose VPC from the list.

45- On the left navigation pane, under Virtual private cloud, choose Peering connections.

46- Choose Create peering connection and configure the following:

Name: VPC A <-> VPC B
VPC ID (Requester): VPC A
Account: My account
Region: This Region (xx-xxxx-x) Make sure the region labeled AWSRegion in the left side of the instructions matches with the region where you are creating the VPC Peering connection.
VPC ID (Accepter): VPC B
Tags: Accept proposed Tags 47- Choose Create peering connection. The status transitions through Initiating Request to Pending Acceptance. 48- From the Actions button, choose Accept request. 49- Choose Accept request. 50- Repeat the above steps to create VPC A <-> VPC C peering connection with the following configuration:
Name: VPC A <-> VPC C
VPC ID (Requester): VPC A
Account: My account
Region: This Region (xx-xxxx-x)
VPC ID (Accepter): VPC C
Tags: Accept proposed Tags You should now have two active peering connections.

Task 1.4.1: Update Route Tables for VPCs

51- On the left navigation pane, under Virtual private cloud, choose Route Tables.

A. Update Route Table for VPC A

52- Select VPC A Route Table.
53- Choose Routes tab.
54- Choose Edit routes.
55- Choose Add route to add VPC B CIDR in Destination to enable VPC A to reach VPC B through the VPC peering connection:

Destination: 10.1.0.0/16
Target: Choose Peering Connection from drop-down list
Select pcx-xxxxxxxxxx (VPC A <-> VPC B) from the drop-down list

56- Choose Add route to add VPC C CIDR in Destination to enable VPC A to reach VPC C through the VPC peering connection:

Destination: 10.2.0.0/16
Target: Choose Peering Connection from drop-down list
Select pcx-yyyyyyyyyy (VPC A <-> VPC C) from the drop-down list 57- Choose Save changes. #### B. Update Route Tables for VPC B

58- On the left navigation pane, under Virtual private cloud, choose Route tables.
59- Select VPC B Route Table.
60- Choose Routes tab.
61- Choose Edit routes .
62- Choose Add route to add VPC A CIDR in Destination to enable VPC B to reach VPC A through the VPC peering connection:

Destination: 10.0.0.0/16
Target: Choose Peering Connection from drop-down list
Select pcx-xxxxxxxxxx (VPC A <-> VPC B) from the drop-down list

63- Choose Save changes.

C. Update Route Table for VPC C

64- On the left navigation pane, under Virtual private cloud, choose Route tables.
65- Select VPC C Route Table.
66- Choose Routes tab.
67- Choose Edit routes.
68- Choose Add route to add VPC A CIDR in Destination to enable VPC C to reach VPC A through the VPC peering connection:

Destination: 10.0.0.0/16
Target: Choose Peering Connection from drop-down list
Select pcx-yyyyyyyyyy (VPC A <-> VPC C) from the drop-down list

69- Choose Save changes.

D. Check EC2 connectivity in VPC A

70- At the top of the AWS Management Console, to the right of the Services menu, search for EC2 in the search bar and then choose EC2 from the list.
71- On the left navigation pane, choose Instances.
72- Select the EC2 VPC A - AZ1 instance.
73- Choose Connect from the navigation bar.
74- With the Session Manager tab selected, choose Connect.
If peering and routing are configured correctly, you can ping both instances.

75- Select the EC2 VPC B - AZ1 instance.
76- Choose Connect from the navigation bar.
77- With the Session Manager tab selected, choose Connect.

From the above image, you can find that EC2 VPC B - AZ1 ping to EC2 inside VPC A, but can't pint to EC2 inside VPC C, this is due that there is a peering between VPC B and VPC A, but there is no peering between VPC B and VPC C.

TASK 1.5: SIMPLIFY ROUTING USING AWS TRANSIT GATEWAY

In the previous section, you created VPC Peering Links to facilitate connectivity between the VPCs, without sending that traffic over the public internet. While this approach can be used to interconnect many VPCs, managing many point-to-point connections can be cumbersome as the number of VPCs you connect grows. A more scalable approach is to utilize AWS Transit Gateway.
In this task, you will remove point-to-point peering connections between VPC A and VPC B; VPC A and VPC C. You will set up Transit Gateway (TGW) and use it to interconnect VPC A, VPC B, and VPC C.

Task 1.5.1: Delete VPC Peering Connections

78- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for VPC and then choose VPC from the list.
79- On the left navigation pane, under Virtual private cloud, choose Peering connections.
80- Select VPC A <-> VPC B peering connection.
81- From the Actions button, choose Delete peering connection.
82- Select the option to Delete related route table entries to avoid traffic blackholing scenarios.
83- To confirm deletion, type delete in the field: delete.

84- Choose Delete.
85- Repeat the above steps to delete VPC A <-> VPC C peering connection.

Task 1.5.2: Setup Transit Gateway

A. Create Transit Gateway

86- On the left navigation pane, under Transit gateways, choose Transit gateways.
87- Choose Create transit gateway and configure the following:

Name tag: TGW
Description: Lab Transit Gateway TGW
Amazon side Autonomous System Number (ASN): 64512
The Amazon side ASN or Multicast support cannot be changed after the transit gateway is created.
DNS support: enable
VPN ECMP support: enable
Default route table association: enable
Default route table propagation: enable
Multicast support: enable
Keep the default settings for the rest of the parameters
88- Choose Create transit gateway.
A few moments later, TGW will transition from Pending to Available state.

B. Create Transit Gateway attachments subnets

According to best practices, it is recommended to use a separate small /28 subnet for each transit gateway VPC attachment.
Table 3. The below table shows IPv4 CIDR allocations for all Subnets in all the VPCs.

The availability zones in the above table are shown as an example. Please use the availability zones from the region where your lab is deployed.

89- On the left navigation pane, under Virtual private cloud, choose Subnets.
90- Choose Create subnet and configure the following:

VPC ID: VPC A
Subnet name: Create subnets with names that reflect VPC and AZ placement, such as VPC A - AZ1 TGW
Availability Zone: Select the first Availability Zone in the list. (Do not choose No Preference)
IPv4 CIDR block: 10.0.2.0/28
Tags: Accept proposed Tags
91- se Add new subnet and configure the following:
Subnet name: Create subnets with names that reflect VPC and AZ placement, such as VPC A - AZ2 TGW
Availability Zone: Select the second Availability Zone in the list. (Do not choose No Preference)
IPv4 CIDR block: 10.0.3.0/28
Tags: Accept proposed Tags
92- Choose Create subnet.
93- Repeat the steps above to create subnets for VPC B and VPC C; refer to Table 3. above for CIDR allocations.
After you finish the task, six new Transit Gateway attachment subnets should be available.

C. Create Transit Gateway attachments

94- On the left navigation pane, under Transit gateways, choose Transit gateway attachments.
95- Choose Create transit gateway attachment and configure the following:

Name tag: VPC A Attachment
Transit Gateway ID: TGW
Attachment type: VPC
VPC ID: VPC A
Subnet IDs: Choose both subnets from the drop-down list:
- VPC A - AZ1 TGW
- VPC A - AZ2 TGW

96- Choose Create transit gateway attachment .
97- Repeat these steps to create attachments

VPC B Attachment for VPC B
VPC C Attachment for VPC C Upon completion, you should see three Transit Gateway attachments.

Task 1.5.3: Check Transit Gateway route table

98- On the left navigation pane, under Transit gateways, choose Transit gateway route tables.
99- Select the route table you see with the route table ID starting with tgw-rtb-xxxxxxxxxxxxxx.
100- Choose the Routes tab in the lower half of the page.
Your routing table should be populated with VPC A, VPC B, VPC C routes.

Task 1.5.4: Update Route Tables of VPCs

101- On the left navigation pane, under Virtual private cloud, choose Route tables.
102- Select VPC A Route Table.
103- Choose Routes tab.
104- Choose Edit routes .
105- Choose Add route and configure the following:

Destination: Enter 10.0.0.0/8
Target: Choose Transit Gateway from the drop-down list and then choose the displayed Transit Gateway ID. You can do this because the existing local route for VPC A (10.0.0.0/16) is more specific and therefore, any traffic for 10.0.0.0/16 will traverse the more specific local route to the VPC. Anything other traffic for 10.0.0.0/8 will traverse the less specific route (10.0.0.0/8) to the transit gateway. 106- Choose Save changes . 107- Repeat these steps to create 10.0.0.0/8 route pointing to the TGW Transit Gateway in VPC B Route Table and VPC C Route Table Routing tables.

TASK 1.6: CHECK CONNECTIVITY BETWEEN VPCS USING THE TGW

Now connect to any EC2 as we did before and try to ping to the other 2 EC2s, it will work from any EC2 to other EC2s

Task 2: Configure Network Monitoring

In this task, you will set up a way to log network traffic using VPC Flow logs. You will also use Amazon CloudWatch to monitor and alarm based on predetermined conditions. Finally, you will explore dashboards where you can customize your experience.

You will utilize the three VPCs with Internet Gateways, Transit Gateway, and EC2 instances that were created in task 1. You will set up VPC Flow logs for VPC A, generate some traffic, and then view the logs in CloudWatch.

TASK 2.1: CREATE VPC FLOW LOGS

VPC Flow Logs is a feature that enables you to capture information (metadata) about the IP traffic going to and from network interfaces in your VPC. For example, if you have a content delivery platform, flow logs can profile, analyze, and predict customer patterns of the content access, and track down top talkers and malicious calls.
In this task, you will create a flow log for all traffic in VPC A and save it to the destination log group VPCFlowLog.

Task 2.1.1: Create CloudWatch Log group

108- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for CloudWatch and then choose CloudWatch from the list.

109- On the left navigation pane, under Logs, choose Log groups.

110- Choose Create log group and configure the following:

111- Choose Create.

Task 2.1.2: Create Create an IAM role for flow logs

112- At the top of the AWS Management Console, to the right of the Services menu, in the search bar, search for IAM and then choose IAM from the list.
113- On the left navigation pane, under Access management, choose Policies.

114- Choose Create policy group and configure the following:

Chose JSON
replace the content with this json { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Action": [ "logs:CreateLogGroup", "logs:CreateLogStream", "logs:PutLogEvents", "logs:DescribeLogGroups", "logs:DescribeLogStreams" ], "Resource": "*" } ] }

115- Choose Next.
116- Enter Policy name: VPCFlowLogPolicy
117- Choose Create policy

118- On the left navigation pane, under Access management, choose Roles.

119- Choose Create role group and configure the following:

Trusted entity type: Custom trust policy.
Custom trust policy: replace "Principal": {}, with the following. "Principal": { "Service": "vpc-flow-logs.amazonaws.com" }, 120- Choose Next 121- In Add permissions select the VPCFlowLogPolicy 122- Choose Next 123- Enter Role name: VPCFlowLogRole 124- Choose Create

Task 2.1.3: Create VPC Flow log

112- Go to VPC A
113- From Actions and choose Create flow log

114- On the Create flow log page, under Flow log settings, configure the following:

Filter: All
Maximum aggregation interval: 1 minute
Destination: Send to CloudWatch Logs
Destination log group: VPCFlowLog
IAM role: VPCFlowLogRole
Log record format: AWS default format 114- Choose Create flow log.

TASK 2.2: GENERATE NETWORK TRAFFIC BETWEEN VPCS

IP traffic going to and from network interfaces in VPC A is now being collected in through VPC Flow Logs and stored using an Amazon CloudWatch Log Group. You need to verify that your monitoring is setup properly. To accomplish this, you will generate some traffic between the Amazon EC2 instance in

VPC A
(EC2 VPC A – AZ1), and instance in

VPC B
(EC2 VPC B – AZ1) using

iperf
(a widely used tool for network performance measurement and tuning).

Task 2.2.1: Update Security Group for EC2 instance in VPC B

First, you will have to update the Security Group for the Amazon EC2 instance in VPC B to allow the iperf server to receive incoming traffic.
115- Go to EC2 VPC B - AZ1, and choose Security tab

116- Choose on the Security Group with ID starting with sg- (VPC B EC2 Security Group) in the lower pane to view and edit its rules.

117- On VPC B EC2 Security Group Security Group page, at the bottom of the page, choose the Inbound rules tab.
118- At the right side of the Inbound rules section, choose Edit inbound rules .

119- On the Edit inbound rules page, choose Add rule and configure the following:

Type: Custom TCP
Port range: 5201
Source type: Custom
Source: 10.0.0.0/16 You added a rule to allow traffic on TCP port 5201 from the Amazon EC2 instance in VPC A by allowing the VPC CIDR 10.0.0.0/16 in the source. 120- Choose Save rules.

Task 2.2.2: Install and run iperf3 server on EC2 instance in VPC B

121- Connect to EC2 VPC B - AZ1

122- Enter the following command to install iperf:
sudo yum install iperf3 -y

123- Enter the following command to start the iperf server on the EC2 instance in VPC B:
iperf3 -s

Task 2.2.3: Perform the following on the EC2 instance in VPC A

124- Connect to EC2 VPC A - AZ1 using the session manager as we did before.
125- Enter the following command to install iperf:
sudo yum install iperf3 -y

126- Enter the following command to set up a TCP transfer with 2 parallel streams for 30 seconds to the EC2 instance in VPC B. Make sure to replace

in the following command:
iperf3 -c <PRIVATE IP OF "EC2 VPC B- AZ1" INSTANCE 10.1.0.X> -P2 -t 30

127- Shutdown the iperf3 server running on EC2 VPC B - AZ1 instance in VPC B by switching back to Session Manager and by pressing CTRL+C.

TASK 2.3: VIEW FLOW LOGS IN CLOUDWATCH

VPC Flow logs can be sent to either an Amazon S3 bucket or CloudWatch. In this lab, you configured the flow logs from VPC A to be sent to CloudWatch. Navigate to CloudWatch to view the logs.
Anatomy of a flow log:

128- Go to VPC service and select VPC A.

129- In the Details pane at the bottom of the page, choose the Flow logs tab.

130- Under Destination name, choose VPCFlowLog to navigate to CloudWatch.
When publishing to CloudWatch, flow log data is published to a log group, and each network interface has a unique log stream in the log group. Log streams contain flow log records. You can create multiple flow logs that publish data to the same log group.
131- Select one of the network interface (ENI) log streams to see the flow records for that interface.
To find the appropriate log, use ENI from EC2 VPC A - AZ1 (on EC2 Instance - Networking Tab scroll to see Networking interfaces list).

TASK 2.4: QUERY FLOW LOG FOR INSIGHTS

CloudWatch Logs Insights enables you to interactively search and analyze log data in CloudWatch Logs, including VPC flow logs. You can perform queries to help you more efficiently and effectively respond to operational issues.
Run a query to show the top 10 talkers based on bytes transferred.

132- Navigate to the browser tab with CloudWatch service.
133- On the left navigation pane, under Logs, choose Logs Insights.
134- From the drop-down list, select VPCFlowLog.

135- On the right navigation pane, choose Queries.

136- Under Sample queries, expand VPC Flow Logs.
137- From the list, choose Top 10 byte transfers by source and destination IP addresses.

138- Choose Apply.
139- Choose Run query.
Review the query results. Do you recognize the top two IP addresses?
Hint: Take a look at the Primary private IPv4 address and Description columns in the EC2 - Network Interfaces and see what they are for.

TASK 2.5: USING AMAZON CLOUDWATCH TO SET AN ALARM

Amazon CloudWatch is a metrics repository. Amazon VPC publishes data points to Amazon CloudWatch for your transit gateways and transit gateway attachments. CloudWatch enables you to retrieve statistics about those data points as an ordered set of time series data, known as metrics. In this section, you will create a static route, set up an alarm on a threshold breach, and set up a dashboard to view a set of metrics on one page.

Task 2.5.1: Create a static route

You can create a static route for a VPC, VPN, or transit gateway peering attachment, or you can create a blackhole route. The blackhole state indicates that the route’s target isn’t available. For more details on the static route and blackhole route, check Additional Resources.
In this section, you will create a blackhole route in your transit gateway route table that drops traffic that matches the route. This task is necessary to view the CloudWatch Metrics in the later task.

140- Go to VPC page.
141- On the left navigation pane, under Transit gateways, choose Transit gateway route tables.
142- Select the route table you see with the route table ID starting with tgw-rtb-xxxxxxxxxxxxxx.
143- Choose the Routes tab in the lower half of the page.
144- Choose Create static route.
145- Under Details, configure the following:

CIDR: Enter 10.3.0.0/16
Type: Choose Blackhole 146- Choose Create static route.

Task 2.5.2: Create Alarm

You can create a CloudWatch alarm that monitors CloudWatch metrics for a given AWS service. CloudWatch will automatically send you a notification when the metric reaches a threshold you specify.
Create an alarm that monitors the number of packets dropped because they match a blackhole route of a Transit gateway. If the packets are dropped, an email notification is sent to the configured email address.

147- Go to CloudWatch service.
148- On the left navigation pane, under CloudWatch Alarms, choose All alarms.
149- Select Create alarm .
150- Choose Select metric .
151- On the Select metric page, under the Metrics section, select the TransitGateway card.
152- Select the Per-TransitGateway Metrics card.
153- Search for PacketDropCountBlackhole .
154- Select tgw-xxxxxxxxxxxxxx Transit Gateway.

155- Choose Select metric .
156- On the Metric page, change the Statistic and Period parameters to the following:

157- On the Conditions section, configure the following:

158- Choose Next .
159- On the Notification page, configure the following:

Alarm state trigger: In alarm
Send a notification to the following SNS topic: Create new topic
Create a new topic… :PacketDropCountBlackhole-Alarm
Email endpoints that will receive the notification… Enter your email address where you want to receive the alarm notification.

160- Choose Create topic.
161- Choose Next .
162- For the Alarm name, enter PacketDropCountBlackhole-Alarm.
163- Choose Next .
164- Review the settings on the next page and choose Create alarm.
The State of the alarm may show Insufficient data. This will happen until enough data points are received by the alarm.
Under Actions you may see Pending confirmation which means that you have not confirmed the subscription yet.
Amazon SNS will send a confirmation response to your email address. Be aware that this can take a few moments for it to be delivered.
165- Go into your email, open the new email from AWS Notifications and choose the Confirm subscription link.

A new browser page opens indicating the subscription status has changed to Subscription Confirmed! This will be required to ensure you receive the notification.

Task 2.5.3: Trigger Alarm

In this section, you will get the opportunity to trigger the alarm and test the notification that you configured.
166- Go to the EC2 service page
167- Using session manager to connect to EC2 VPC A - AZ1 instance.
168- From the EC2 VPC A - AZ1 instance in VPC A try pinging any private IP address belonging to 10.3.0.0/16 CIDR range for example 10.3.0.0

If the number of packets dropped is greater or equal to 1 because they matched a blackhole route of a Transit gateway, then it will activate the CloudWatch Alarm you created.
169- Go to CloudWatch service page.
170- On the left navigation pane, under CloudWatch Alarms, choose All alarms.
171- Confirm that the State of the PacketDropCountBlackhole-Alarm alarm is in alarm.

Please note that it may take a few minutes before you see the state change to In alarm.
The number of packets dropped crossed the set threshold. This will invoke the CloudWatch Alarm you created, which will send a message to the Amazon SNS topic. Amazon SNS will then send you an email message.
172- Check your email to confirm that you received a notification with the subject ALARM: “PacketDropCountBlackhole-Alarm” alerting you that your Amazon CloudWatch Alarm has entered the ALARM state.

Task 2.5.4: Create dashboard

Amazon CloudWatch dashboards are customizable home pages in the CloudWatch console that you can use to monitor your resources in a single view, even those resources that are spread across different Regions. You can use CloudWatch dashboards to create customized views of the metrics and alarms for your AWS resources.
In this section, you will create a CloudWatch Dashboard to view the PacketDropCountBlackhole metric for the Transit Gateway.

173- On the left navigation pane, choose Dashboards.
174- Select Create dashboard .

175- Enter PacketDropCountBlackhole-Dashboard for Dashboard name.
176- Choose Create dashboard .
177- Select Number on the Add widget page, then choose Next.

178- On the Add metric graph page, under the Metrics section, select the TransitGateway card.
179- Select the Per-TransitGateway Metrics card.
180- Search for PacketDropCountBlackhole .
181- Select tgw-xxxxxxxxxxxxxx Transit Gateway.

181- Choose Graphed metrics (1) tab.
182- Change the Statistic and Period parameters:

183- Scroll to the top of the Add metric graph page. Near the top of the screen, by Untitled graph, choose the Edit icon.
184- Enter a name for the graph, such as PacketDropCountBlackhole-Graph.

185- Choose Apply .
186- Choose Create widget at the bottom of the page.
187- At the top of the dashboard, choose Save.
You can resize the widget by dragging the lower right corner of the graph. Change the data time span to 1h for better visualization. After making these modifications, always remember to choose Save button to save your Dashboard changes.
Task complete: You have successfully generated traffic between instances, viewed the flow log in CloudWatch, and configured a CloudWatch Alarm, and a CloudWatch Dashboard.

Task 3: Security Controls

Recall that at the end of Task 1, you had provisioned 3 AWS VPCs and EC2 instances in each VPC. The VPCs were interconnected using Transit Gateway. The environment is depicted as below.

You will now be adding security to the environment you have built. You will use Network Access Lists (NACLs) and Security Groups, which are basic filtering mechanisms.

TASK 3.1: MODIFYING DEFAULT NETWORK ACLS TO BLOCK, ALLOW SUBNETS

Network ACLs are stateless access controls you configure at a subnet level, to allow or block a CIDR block on a particular port or range of ports. Network ACL rules are numbered list and evaluated top down, with a DENY ALL at the end. If a rule is matched, subsequent rules are not evaluated.
Both inbound and outbound traffic can be controlled with these rules. By default when you have created the above subnets, the default Network ACL attached to them will have an ALLOW ALL rule for both inbound and outbound traffic.
In this task, you will modify Network ACL on VPC A - AZ1 to allow only VPC B’s CIDR; and test connectivity using ping to send ICMP traffic from VPC B to VPC A, and from VPC C to VPC A as well.
According to best practices, it is recommended to use a separate subnet for each transit gateway VPC attachment in case of using NACLs with Transit Gateways. We keep the inbound and outbound NACL that is associated with the transit gateway subnets open, and will apply NACLs with filters to your workload subnets.

Task 3.1.1: TGW Subnets & NACL configuration

188- Go to the VPC service page.
189- On the left navigation pane, under Security, choose Network ACLs.
190- At the upper-right corner of the Network ACLs page, choose Create network ACL.
191- On the Create network ACL page, under Network ACL settings, configure the following:

Name: nacl-tgw-vpca
VPC: VPC A
Tags: Accept proposed Tags 192- Choose Create network ACL . 193- Repeat these steps to create network ACLs:
nacl-tgw-vpcb for VPC B.
nacl-tgw-vpcc for VPC C.

194- On the Network ACLs page, select nacl-tgw-vpca.
195- Choose the Subnet associations tab.
196- Choose Edit subnet associations .
197- From the Edit subnet associations page, select the following:

VPC A - AZ1 TGW
VPC A - AZ2 TGW 198- Choose Save changes . 199- Repeat these steps for VPC B and VPC C by choosing corresponding subnets with TGW in Subnet name.

200- On the Network ACLs page, select nacl-tgw-vpca.
201- On the nacl-tgw-vpca Network ACLs page, at the bottom of the page, choose the Inbound rules tab.
Notice there is only one inbound rule that denies all incoming traffic.

202- At the right side of the Inbound rules section, choose Edit inbound rules .
203- On the Edit inbound rules page, choose Add new rule and configure the following:

Rule number: 100
Type: All traffic
Source: 0.0.0.0/0
Allow/Deny: Allow 204- Choose Save changes . 205- On the nacl-tgw-vpca Network ACLs page, at the bottom of the page, choose the Outbound rules tab. Notice there is only one outbound rule that denies all incoming traffic.

206- At the right side of the Outbound rules section, choose Edit outbound rules .
207- On the Edit outbound rules page, choose Add new rule and configure the following:
Rule number: 100
Type: All traffic
Destination: 0.0.0.0/0
Allow/Deny: Allow

208- Choose Save changes .
209- Repeat the above steps to add the same Inbound and Outbound Rules in nacl-tgw-vpcb and nacl-tgw-vpcc Network ACLs.
210- Now we’re ready to proceed to the next section and configure NACLs for EC2 instances to allow the ICMP traffic, TGW NACLs that allow everything will not introduce any blocking for us.

Task 3.1.2: NACL configuration for EC2 subnets in AZ1

211- On the left navigation pane, under Security, choose Network ACLs.
212- In the Network ACLs page, in the Filter network ACLs search bar, search for VPC A .

213- Select the ACL marked as the Default with a value of Yes.
Check the default inbound rules in the Network ACL can be seen above. All traffic gets evaluated for Protocol, Port, and Source IP match. In this default Network ACL, all traffic is allowed into the VPC A-AZ1 Subnet by the first rule. The second rule which is a DENY ALL is not evaluated.
We will now modify the first rule (100) to allow only ICMP traffic from VPC B’s CIDR.

214- On the Network ACLs page, at the bottom of the page, choose the Inbound rules tab.
215- At the right side of the Inbound rules section, choose Edit inbound rules.
216- On the Edit inbound rules page, modify rule number 100 and configure the following:

Type: All ICMP - IPv4
Source: 10.1.0.0/16
Allow/Deny: Allow 217- Choose Save changes . We have now completed modifying the Network ACL of VPC A - AZ1 Subnet to allow ONLY ICMP traffic from VPC B’s CIDR and all other traffic will be denied by the catch-all DENY rule. Let us now test this from VPC B - AZ1 Subnet for ALLOW, and VPC C - AZ1 Subnet for DENY. Note that You have not modified outbound rules, and the default outbound rule allows ALL traffic to flow out of the subnet.

TASK 3.2: TESTING CONNECTIVITY THROUGH NACLS FROM VPC B TO VPC A

Here, you will login to EC2 VPC B - AZ1 using EC2 instance connect, and verify reachability to EC2 VPC A - AZ1 over ICMP (ping).
218- Go to the EC2 service page.
219- On the left navigation pane, choose Instances.
220- Select the EC2 VPC B - AZ1 instance.
221- Choose Connect from the navigation bar.
222- With the Session Manager tab selected, choose Connect .
A terminal session should open in a new browser tab.
223- From the EC2 VPC B - AZ1 instance in VPC B, try pinging the private IP addresses of the EC2 VPC A - AZ1 instance in VPC A.

The ICMP traffic should flow through and show as being successfully returned. Enter ‘CTRL + C’ to stop the ping.
You have now verified that the Network ACL on VPC A - AZ1 subnet is allowing ICMP traffic to flow in and out, from EC2 VPC B - AZ1.

TASK 3.3: TESTING CONNECTIVITY THROUGH NACLS FROM VPC C TO VPC A

Similarly, connect to EC2 VPC C - AZ1 from EC2 console, and ping EC2 VPC A - AZ1 as you did from VPC B above.

The ping command will freeze without progress because the Network ACL in VPC A - AZ1 subnet is DENYING all traffic from VPC C. Enter ‘CTRL + C’ to stop the ping see review the results. They should indicate that no replies were received.
Task complete: You have successfully modified the default Network ACL in VPC A to allow ICMP traffic only from VPC B; the only other rule is a DENY ALL. You verified that ICMP traffic flows through from EC2 VPC B - AZ1 to EC2 VPC A - AZ1 and DID NOT flow through from EC2 VPC C - AZ1.

TASK 3.4: MODIFYING SECURITY GROUP TO ALLOW ONLY ICMP TRAFFIC FROM VPC C TO VPC A

Security Groups are virtual, stateful firewalls attached to an instance or network interface. Both inbound and outbound rules can be defined to allow specific protocols, ports, and source/destination CIDR. A DENY is not possible with security groups.
With Security Groups, all rules are evaluated before a network packet is allowed or blocked, unlike Network ACLs where the rules are evaluated in order of rule number and once a rule matches subsequent rules are not evaluated.
In this section, you will modify the security group attached to EC2 VPC A - AZ1 to allow only ICMP traffic inbound from VPC C’s CIDR only. You will verify that EC2 VPC C - AZ1 can ping EC2 VPC A - AZ1, and EC2 VPC B - AZ1 is not able to ping EC2 VPC A - AZ1.
Prerequisite: Edit the Network ACL in VPC A - AZ1 subnet to revert the change to Rule 100, and set it to allow ALL TRAFFIC from all sources (0.0.0.0/0), because you want all traffic to flow past the Network ACL to the instance in VPC A to test the Security Group at the instance level.
224- Navigate to the browser tab with VPC service.
225- On the left navigation pane, under Security, choose Network ACLs.
226- On the Network ACLs page, in the Filter network ACLs search bar, search for VPC A.
227- On the Network ACLs page, at the bottom of the page, choose the Inbound rules tab.
228- At the right side of the Inbound rules section, choose Edit inbound rules.
229- On the Edit inbound rules page, modify rule number 100 and configure the following:

Type: All traffic
Source: 0.0.0.0/0
Allow/Deny: Allow

230- Choose Save changes .
231- Navigate to the browser tab with EC2 Instances.
232- On the left navigation pane, choose Instances and select EC2 VPC A - AZ1.
233- Navigate to the Security tab below and choose the Security Group with ID starting with sg- (VPC A EC2 Security Group) in the lower pane to view and edit its rules.
234- On the VPC A EC2 Security Group Security Group page, at the bottom of the page, choose the Inbound rules tab.
235- At the right side of the Inbound rules section, choose Edit inbound rules .

236- On the Edit inbound rules page, in the rule that is currently allowing All ICMP from 10.0.0.0/8 source, change it to allow only from VPC C’s CIDR:

Type: All ICMP-IPv4
Source type: Custom
Source: 10.2.0.0/16

237- Choose Save rules .

TASK 3.5: TESTING CONNECTIVITY FROM VPC B TO VPC A THROUGH SECURITY GROUPS

Now you have modified the Security Group on EC2 VPC A - AZ1 to allow ICMP traffic (ping traffic) only from instances in VPC C. You will now verify that you are NOT able to ping this instance from EC2 VPC B - AZ1, and you are ABLE to ping from EC2 VPC C - AZ1.
238- On the left navigation pane, choose Instances.
239- Select the EC2 VPC B - AZ1 instance.
240- Choose Connect from the navigation bar.
241- With the Session Manager tab selected, choose Connect.
A terminal session should open in a new browser tab.
242- From the EC2 VPC B - AZ1 ping to the private IP of the instance that is in VPC A.

It will not be able to reach out to the EC2 in VPC A.

TASK 3.6: TESTING CONNECTIVITY FROM VPC C TO VPC A THROUGH SECURITY GROUPS

Similarly connect to EC2 VPC C - AZ1 and try to ping EC2 VPC A - AZ1.
243- On the left navigation pane, choose Instances.
244- Select the EC2 VPC C - AZ1 instance.
245- Choose Connect from the navigation bar.
246- With the Session Manager tab selected, choose Connect.
A terminal session should open in a new browser tab.
247- From the EC2 VPC C - AZ1 instance in VPC C, try pinging the private IP addresses of EC2 VPC A - AZ1 instance in VPC A.

The ping will succeed and traffic will flow through. This is because the SG on EC2 VPC A - AZ1 is allowing ICMP traffic from VPC C’s CIDR range.
Task complete: You have successfully modified the Security Group on EC2 VPC A - AZ1 to allow only ICMP traffic from VPC C. You tested and verified that you cannot ping this instance from VPC B, but You can ping it from VPC C confirming the behavior of the Security Group.

Conclusion
Congratulations! You now have successfully:

Demonstrate intra versus inter-VPC routing.
Select the appropriate connectivity options for an environment.
Capture network traffic information (metadata) with VPC flow logs.
Configure monitoring for networking statistics and metrics.
Filter network traffic with Network Access Lists (NACLs), and Security Groups (SG).

End lab
1- Delete the EC2s

2- Go to the VPC service page to delete the TGW and VPCs

3- Select Transit gateway attachments under Transit gateways
4- You need to delete the attachments one by one

5- Select Transit gateways under Transit gateways.
6- Select TGW and from Actions select Delete transit gateway.

7- Select Your VPCs under Virtual private cloud.
6- Select VPC A, then from Actions select delete VPC

7- In Delete VPC page, Enter delete to confirm the delete then Select Delete.
8- Make the same with VPC B, and VPC C to delete them.

Additional Resources
For more information about the topics covered in this lab, see:

AWS VPC From Zero to Hero Series: 1- Introduction to Amazon Virtual Private Cloud (VPC)

Ahmed Shendy — Tue, 06 Feb 2024 08:09:49 +0000

This is a series of hands-on labs that start from beginner level to advanced level to give you a real understanding of AWS VPC.

Overview

In this adventure, you will use the Amazon VPC wizard to create a VPC, attach an Internet gateway, add a subnet, and then define routing for the VPC so traffic can flow between the subnet and the Internet gateway.

Topics covered

Upon completion of this adventure, you will be able to:

Create an Amazon VPC Using the VPC Wizard
Explore the basic components of a VPC including: 1- Public and private subnets 2- Route tables and routes 3- NAT gateways 4- Network ACLs

What is Amazon Virtual Private Cloud (VPC)?

Amazon Virtual Private Cloud (Amazon VPC) lets you provision a logically isolated section of the Amazon Web Services (AWS) cloud where you can launch AWS resources in a virtual network that you define. You have complete control over your virtual networking environment, including selection of your own IP address range, creation of subnets, and configuration of route tables and network gateways. You can use both IPv4 and IPv6 in your VPC for secure and easy access to resources and applications.

Prerequisites

An active AWS account.
A user that has access to the AWS console with the needed permissions.

ask 1: Create an Amazon VPC

In this task you will create an Amazon VPC using the VPC wizard. The wizard automatically creates a VPC based upon parameters you specify. Using the VPC Wizard is much simpler than manually creating each component of the VPC.
Here is an overview of the VPC you will create:

1- Open the AWS console, and select the region that you want to create the VPC in it.

2- At the top of the AWS Management Console, to the right of Services menu, in the search bar, search for "VPC" and then choose VPC from the list

3- Click VPC Dashboard in the top-left corner.
In every region, a default VPC with CIDR 172.31.0.0/16 has already been created for you. So, even if you haven’t created anything in your account yet, you will see some pre-existing VPC resources already there.
4- Click
5- On Create VPC page, under VPC settings section, choose VPC and more.

6- Configure the following fields:

7- Click
Your VPC will now be created. A status window displays progress. When the VPC completes, a status window confirms that your VPC has been successfully created. This may take a few minutes to create.
8- Click
9- Copy the VPC IP value and keep it in your text editor.

Task 2: Explore your VPC

In this task, you will explore the VPC components created by the VPC Wizard.
10- Select Your VPCs from the left panel.
11- In Filter by VPC search about the created VPC.
12- Locate Your VPCs’ Name column, your VPC is created with the name adventure_1-vpc.

13- In the left navigation pane, choose Internet gateways.

The Internet gateway for your VPC will be displayed.
An Internet gateway connects your VPC to the Internet. If the Internet gateway was not present, then the VPC would have no connectivity to the Internet.
An Internet gateway is a horizontally scaled, redundant, and highly available VPC component. It therefore imposes no availability risks or bandwidth constraints on your network traffic.
14- In the left navigation pane, choose Subnets.

A Subnet is a subset of a VPC. A subnet:
• Belongs to a specific VPC
• Exists in a single Availability Zone (while a VPC can span multiple Availability Zones)
• Has a range of IP addresses (known as a CIDR range, which stands for Classless Inter-Domain Routing)
Two subnets will be displayed for your VPC: a Public subnet and a Private subnet.
15- Select the Public subnet which starts with adventure_1-subnet-public in the Name column.

Examine the information displayed in the lower window pane:
• Each subnet is assigned a unique Subnet ID.
• The IPv4 CIDR of 10.0.25.0/24 means that the subnet contains the range of IP addresses from 10.0.25.0 to 10.0.25.255. (IPv6 is also supported, but is not part of this lab.)
• The subnet only has 250 Available IPs out of 256 possible addresses. This is because there are several reserved addresses in each subnet and one IP address has been consumed by the NAT gateway.
Why is this subnet considered to be a Public subnet? The answer lies in the Subnet Routing.

16- Choose the Route table tab.

Each subnet is associated with a Route table, which specifies the routes for outbound traffic leaving the subnet. Think of it like an address book that lists where to direct traffic based on its destination.

Two routes in the route table are associated with your public subnet:
• Route 10.0.0.0/16 | local directs traffic destined for elsewhere in the VPC (which has a range of 10.0.0.0/16) locally within the VPC. This traffic never leaves the VPC.
• Route 0.0.0.0/0 | igw- directs all traffic to the Internet gateway.
Routing rules are evaluated from the most restrictive (with the bigger number after the slash) through to the least restrictive (which is 0.0.0.0/0 since it refers to the entire Internet). Thus, traffic is first sent within the VPC if it falls within the range of the VPC, otherwise, it is sent to the Internet. The rules can further be edited based on your particular network configuration.
The fact that this subnet is associated with a Route Table that has a route to an Internet gateway makes it a Public Subnet. That is, it is reachable from the Internet.

17- Choose the Network ACL tab.

A network access control list (ACL) is an optional layer of security for your VPC that acts as a firewall for controlling traffic in and out of subnets. Network ACLs are normally left with their default settings that allow all traffic in and out of subnets:
• Rule 100 Inbound allows all inbound traffic into the Public Subnet.
• Rule 100 Outbound allows all traffic out of the Public Subnet.
• The second line in each ruleset shows an asterisk (*) that acts as a catch-all rule in case traffic does not match any of the earlier rules.

18- In the left navigation pane, choose Subnets.
19- At the top of the window, select Private subnet which starts with adventure_1-subnet-private in the Name column, and ensure that it is the only line selected.

20- Choose the Tags tab.

The subnet has been tagged with the key of Name starting with the value of Lab-subnet-private. Tags help you to manage and identify your AWS resources.

21- Choose the Route table tab.

The Route Table for the Private subnet has the configuration:
• Route 10.0.0.0/16 | local is the same as the Public subnet.
• Route 0.0.0.0 | nat- directs traffic to the NAT gateway.
This subnet does not have a route to the Internet gateway. Therefore, it is a Private Subnet.

22- In the left navigation pane, choose NAT gateways.

A Network Address Translation (NAT) gateway allows resources in a private subnet to connect to the Internet and other resources outside the VPC. This is an outbound-only connection, which means that the connection must be initiated from within the private subnet. Resources on the Internet cannot initiate an inbound connection. Therefore, it is a means of keeping resources private and improving security for VPC resources.

23- In the left navigation pane, choose Security groups, and select the security group that matches with your VPC ID.

24- Choose the Inbound rules tab.

Security groups act as virtual firewall for your instances to control inbound and outbound traffic. When you launch an Amazon EC2 instance into a VPC, you can assign up to five security groups to the instance. Security groups act at the instance level and not the subnet level. Your VPC automatically comes with a default security group. If you do not specify a different security group when you launch an Amazon EC2 instance, it will use the default security group.
The default security group permits ALL traffic to access associated resources, but only if the Source is the default security group. This self-reference might appear strange, but this configuration simply means that any EC2 instance associated with the default security group can communicate with any other EC2 instance that is associated with the default security group. All other traffic is denied. This is a very safe default setting because it limits any access from other resources.
When adding resources to the VPC, you can create additional security groups to permit desired access to resources such as web servers, application servers, and database servers.
Launching Amazon EC2 instances in this lab is out of the scope of the lab. Please do not attempt to launch an Amazon EC2 instance. This lab will not allow you to launch EC2 instances.

Delete the created resources

Now let's end the first adventure by deleting the created resources.

25- Select Nat gateways from left navigation pane.
26- From the NAT gateways list, select the one that matches your VPC ID, then from Actions click on Delete NAT gateway.

27- In the Delete Nat gateway window, write delete in the textbox, then click Delete

28- Wait until the state of the NAT gateway became Deleted

29- Select Your VPCs from left navigation pane.
30- From the VPCs list, select your VPC, then from Actions click on Delete VPC.

31- In the Delete VPC window, write delete in the textbox, then click Delete

How to choose the right EC2 for your AWS workloads

Ahmed Shendy — Mon, 24 Oct 2022 01:09:48 +0000

Before we start choosing the right EC2, there are currently more than 550 instance types, so we will look at the current available instance types.

Instance naming

The name of the EC2 is separated into two main parts by a dot, as shown in the below image; the second part is the size, the first part is the family, and which consists of letters and a digit; before this digit, there are one or more than one letters and this part is defining the category, the digit is for the generation, then after the digit, there are one or more letters each one defines a specific capability.

The available capabilities are :

The bellow table shows the instance types for the current generations

Instance type selection

Now let's try to select the right instance, there are 2 rules that you can start with them.

1- Select the latest generation, usually latest generation is better in performance and cheaper.

2- If you don't have a specific requirement, start with T families.

What attributes do you need?
try to get answer to these questions:

Do you need a specific processor type?
Can you take advantage of burstable performance?
Can you take advantage of ARM processors?
How much total memory do you need (min/max)?
What is your required vCPU:memory ratio?
What are your network performance requirements?
Do you need accelerated computing options? GPU? FPGA?

After getting answers for the above questions, then you can find you candidate instance from the above table.

EC2 Instance Discovery

AWS provides an easy way to discover, locate, and compare EC2 instances.

Go to the console then EC2 > Instances Types > Instance Types

How to choose the right load balancer for your AWS workloads

Ahmed Shendy — Tue, 04 Oct 2022 15:35:04 +0000

Before talking about how to choose the right load balancer, let's take a look about what are the available load balancers in AWS.

When do we need to use the Application LB?

When we have one of the below targets combined with one or more from the below requirements.

When do we need to use the Network LB?

When we have one of the below targets combined with one or more from the below requirements.

When do we need to use the Gateway LB?

When we have one of the below targets combined with one or more from the below requirements.

When do we need to use the AWS Global Accelerator?

When we have one of the below targets combined with one or more from the below requirements.

To get more details on how to select the LB got to this session https://www.youtube.com/watch?v=p0YZBF03r5A

Introducing OpenShift Container Platform 4

Ahmed Shendy — Mon, 15 Nov 2021 18:55:29 +0000

OpenShift Container Platform 4 Architecture

Red Hat OpenShift Container Platform 4 (RHOCP 4) is a set of modular components and services built on top of Red Hat CoreOS and Kubernetes.
One of the main advantages of using OpenShift is that it uses several nodes to ensure the resiliency and scalability of its managed applications. OpenShift forms a cluster of node servers that run containers and are centrally managed by a set of master servers.

The following diagram illustrates the high-level logical overview of the OpenShift Container Platform 4 architecture.

The following diagram illustrates the OpenShift Container Platform stack.

From bottom to top, and from left to right, this shows the basic container infrastructure, integrated and enhanced by Red Hat:

Red Hat CoreOS is the base OS on top which OpenShift runs. Red Hat CoreOS is a Linux distribution focused on providing an immutable operating system for container execution.
CRI-O is an implementation of the Kubernetes Container Runtime Interface (CRI) to enable using Open Container Initiative (OCI) compatible runtimes
Kubernetes manages a cluster of hosts, physical or virtual, running containers. It uses resources that describe multicontainer applications composed of multiple resources, and how they interconnect
etcd is a distributed key-value store, used by Kubernetes to store configuration and state information about the containers and other resources inside the Kubernetes cluster
Custom Resource Definitions (CRDs) are resource types stored in etcd and managed by Kubernetes. These resource types form the state and configuration of all resources managed by OpenShift.
Containerized services fulfill many PaaS infrastructure functions, such as networking and authorization. RHOCP uses the basic container infrastructure from Kubernetes and the underlying container runtime for most internal functions.
Runtimes and xPaaS are base container images ready for use by developers, each preconfigured with a particular runtime language or database.
DevOps Tools and User Experience: RHOCP provides web UI and CLI management tools for managing user applications and RHOCP services.

The following table lists some of the most commonly used terminology when you work with OpenShift.

Term	Definition
Node	A server that hosts applications in an OpenShift cluster
Master Node	A node server that manages the control plane in an OpenShift cluster. Master nodes provide basic cluster services such as APIs or controllers.
Worker Node	Worker nodes execute workloads for the cluster. Application pods are scheduled onto worker nodes.
Resource	Resources are any kind of component definition managed by OpenShift.
Controller	A controller is an OpenShift component that watches resources and makes changes attempting to move the current state towards the desired state.
Label	A key-value pair that can be assigned to any OpenShift resource.
Namespace or Project	A scope for OpenShift resources and processes, so that resources with the same name can be used in different contexts.
Console	A web UI provided by OpenShift that allows developers and administrators to manage cluster resources.

New Features in RHOCP 4

RHOCP 4 includes new features, such as:

CoreOS as the default operating system for all nodes, offering an immutable infrastructure optimized for containers
A new cluster installer, which simplifies the process of installing and updating the masters and worker nodes in the cluster
A self-managing platform, able to automatically apply cluster updates and recoveries without disruption.
A redesigned web console based on the concept of "personas", targeting both platform administrators and developers
An Operator SDK to build, test, and package Operators.

Describing OpenShift Resource Types

Pods (pod) are the basic unit of work for OpenShift.
Services (svc) Specific IP/port combinations that provides access to a pool of pods. By default, services connect clients to pods in a round-robin fashion.
Replication Controllers (rc) OpenShift resources that define how pods are replicated (horizontally scaled) to different nodes.
Persistent Volumes (pv) Storage areas to be used by pods.
Persistent Volume Claims (pvc) Requests for storage by a pod. A pvc links a pv to a pod so its containers can make use of it, usually by mounting the storage into the container's file system.
Config Maps (cm) A set of keys and values that can be used by other resources. ConfigMaps and Secrets are usually used to centralize configuration values used by several resources. Secrets differ from ConfigMaps maps in that Secrets are used to store sensitive data (usually encrypted), and their access is restricted to fewer authorized users.
Deployment Configs (dc) A set of containers included in a pod, and the deployment strategies to be used. A dc also provides a basic but extensible continuous delivery workflow.
Build Configs (bc) A process to be executed in the OpenShift project. The OpenShift Source-to-Image (S2I) feature uses BuildConfigs to build a container image from application source code stored in a Git repository. A bc works together with a dc to provide a basic but extensible continuous integration and continuous delivery workflows.
Routes DNS host names recognized by the OpenShift router as an ingress point for various applications and microservices deployed on the cluster.
Image Streams (is) An image stream and its tags provide an abstraction for referencing container images from within OpenShift Container Platform. The image stream and its tags allow you to track what images are available and ensure that you are using the specific image you need even if the image in the repository changes. Image streams do not contain actual image data, but present a virtual view of related images, similar to an image repository.

DEV Community: Ahmed Shendy

Aurora Serverless v2: When "I/O Optimized" Actually Costs You More

The Request

The Difference: It’s a Billing Flag, Not Hardware

The "CloudWatch Trap"

Why? Granularity.

The Solution: The "10-Day Snapshot" Method

Step 1: The Source of Truth (Cost Explorer)

Step 2: The Variable Costs (CloudWatch)

Step 3: The Comparison

The Verdict

Conclusion

DNS Failures in EKS? The Real Bottleneck Was AWS Network Limits

AWS Network Allowance Metrics

Capturing Network Metrics in EKS

Enable ethtool Collector in node exporter

Building the Grafana Dashboard

Available Connection Tracking Capacity

Packets Dropped Due to Conntrack Exhaustion

Other Network Allowance Exceeded Metrics

Final Insight

Takeaway

🚀 The Hidden DNS Misconfiguration That Was Killing Performance in Our EKS Cluster (and How We Fixed it)

Why I Wrote This Article — The Incident That Sparked Everything

CoreDNS wasn’t resolving —

it was drowning in NXDOMAIN.

🕵️ The Real Breakthrough — It Was One Hostname

⚡ The One-Character Fix That Saved Us

🔍 The Root Cause: ndots in /etc/resolv.conf

🛠 Fixing the Problem (Two Options)

1️⃣ Use Fully Qualified Domain Names with a trailing dot

2️⃣ Reduce ndots for external workloads

📘 References

🎯 Final Thought

💬 If you'd like Part 2 (the full debugging journey — how I traced and proved the root cause), comment below:

AWS VPC From Zero to Hero Series: 2, Build, Secure, and Monitor Networks on AWS

Overview

Topics covered

Prerequisites

Duration

Task 1: Build a Multi-VPC Architecture

TASK 1.1: CREATE THREE VPCS WITH PRIVATE SUBNETS

Task 1.1.1: Create VPCs

Task 1.1.2: Create Subnets

Task 1.1.3: Deploy Internet Gateways

Task 1.1.4: Update Routing Tables

TASK 1.2: DEPLOY EC2 INSTANCES IN VPC’S

TASK 1.3: TEST INTER-VPC COMMUNICATION BETWEEN EC2 INSTANCES

TASK 1.4: SETUP VPC PEERING

Task 1.4.1: Update Route Tables for VPCs

A. Update Route Table for VPC A

C. Update Route Table for VPC C

D. Check EC2 connectivity in VPC A

TASK 1.5: SIMPLIFY ROUTING USING AWS TRANSIT GATEWAY

Task 1.5.1: Delete VPC Peering Connections

Task 1.5.2: Setup Transit Gateway

A. Create Transit Gateway

B. Create Transit Gateway attachments subnets

C. Create Transit Gateway attachments

Task 1.5.3: Check Transit Gateway route table

Task 1.5.4: Update Route Tables of VPCs

TASK 1.6: CHECK CONNECTIVITY BETWEEN VPCS USING THE TGW

Task 2: Configure Network Monitoring

TASK 2.1: CREATE VPC FLOW LOGS

Task 2.1.1: Create CloudWatch Log group

Task 2.1.2: Create Create an IAM role for flow logs

Task 2.1.3: Create VPC Flow log

TASK 2.2: GENERATE NETWORK TRAFFIC BETWEEN VPCS

Task 2.2.1: Update Security Group for EC2 instance in VPC B

Task 2.2.2: Install and run iperf3 server on EC2 instance in VPC B

Task 2.2.3: Perform the following on the EC2 instance in VPC A

TASK 2.3: VIEW FLOW LOGS IN CLOUDWATCH

TASK 2.4: QUERY FLOW LOG FOR INSIGHTS

TASK 2.5: USING AMAZON CLOUDWATCH TO SET AN ALARM

Task 2.5.1: Create a static route

Task 2.5.2: Create Alarm

Task 2.5.3: Trigger Alarm

Task 2.5.4: Create dashboard

Task 3: Security Controls

TASK 3.1: MODIFYING DEFAULT NETWORK ACLS TO BLOCK, ALLOW SUBNETS

🔍 The Root Cause: `ndots` in `/etc/resolv.conf`