DEV Community: Manoj

The Mystery of 3/3 Checks on EC2—Solved

Manoj — Thu, 18 Sep 2025 08:24:12 +0000

Most engineers who’ve worked with AWS Cloud will have launched an EC2 instance (a virtual machine). But only a few of them really know what the “3/3 checks passed” message on the console actually means.

This post explains what these status checks are, why they matter, and what you should do if one of them fails. This is also something that might come up in an cloud interview question 👀.

When an instance is launched or started, AWS runs three types of status checks:

System Status Check
Instance Status Check
Instance Reachability Check Let’s go through each in detail.

1. System Status Check

✅ What it checks: The AWS infrastructure hosting your instance — hardware, networking, and power.
⚠️ If it fails: It’s AWS’s problem. This status check is most unlikely to fail among the three.

Stop & start your instance (it will migrate to a new host).
If the issue persists, you should probably speak to aws support team through raising a support ticket.

2. Instance Status Check

✅ What it checks: Your instance’s operating system( kernel, boot process, responsiveness).
⚠️ If it fails: That’s your responsibility.

Check system logs in the EC2 console.
Use EC2 Serial Console or SSM Session Manager if SSH/RDP isn’t working.
Fix issues like kernel panics, misconfigured filesystems, or firewall lockouts.
Reboot after fixing.

3. Reachability Check

✅ What it checks: Whether the instance is accessible over the network (SSH, RDP, or service ports).
⚠️ If it fails: Likely a config issue, it’s again our responsibility.

Check security group rules and NACLs(basically you will have to check whether you have created the instance in right subnet with right routes)
Confirm route tables & internet gateway for public subnets.
Verify the instance firewall (ufw, iptables, Windows Firewall).

Think of it this way - if your EC2 instance passes all three checks, it’s like your VM just cleared a cloud-level health checkup. Green lights all the way means you can focus on building, not babysitting.

Until next time, Saiyonara. 👋

When Private DNS Just Won’t Resolve: Debugging VPC-to-Hosted Zone Issues in AWS

Manoj — Wed, 23 Jul 2025 16:51:38 +0000

If you’ve worked with AWS VPCs and Route 53, I bet you’ve used or at least come across Private Hosted Zones. They’re incredibly useful when you want to create internal-only DNS records like resolving db.internal to an internal IP.

But what happens when your EC2 instance refuses to resolve the domain, even though everything seems to be wired up correctly?

That’s exactly what I faced recently, and this post is a quick walkthrough of what went wrong and how to fix it.

🔍 The Problem

I had:

A Private Hosted Zone in Route 53 (say, myinternal.local)
A VPC attached to that hosted zone
An EC2 instance launched inside that VPC
DNS record like api.myinternal.local created in the zone But inside the EC2 instance:

ping api.myinternal.local

It gave me:

ping: unknown host api.myinternal.local

⚙️ Root Cause (What Was Missing)

After debugging, the issue boiled down to two missing or misconfigured pieces:

DNS Hostnames or DNS Support Disabled

If your VPC doesn't have DNS resolution and DNS hostnames enabled, the private DNS resolution will silently fail.

You can check and fix this from the VPC → Actions → Edit DNS Resolution/Hostnames in the AWS Console or use the CLI:

aws ec2 modify-vpc-attribute --vpc-id vpc-xxxxxx --enable-dns-support "{\"Value\":true}" aws ec2 modify-vpc-attribute --vpc-id vpc-xxxxxx --enable-dns-hostnames "{\"Value\":true}"

Instance Launched in Wrong Subnet or VPC

Make sure your EC2 instance is in the same VPC that’s attached to the private hosted zone.

Route 53 private hosted zones don’t resolve from any instance — they only resolve from the specific VPC(s) they're associated with.

🧪 How I Debugged It

Here's how I approached the problem:

Tested with dig and nslookup:

dig api.myinternal.local

This gave me no answer section, confirming DNS resolution was failing completely.

Verified /etc/resolv.conf on the EC2 instance:
It was using AmazonProvidedDNS (i.e., .2 address), so the default resolver was correct.

Checked VPC settings:
DNS support and DNS hostnames were disabled by default on this VPC.

Re-enabled those flags and restarted networking on the EC2 instance:

 sudo systemctl restart network

DNS resolution magically started working.

check this out

Manoj — Fri, 18 Jul 2025 08:38:12 +0000

Manoj

Jul 15 '25

AWS Network Firewall: A DevOps Guide in Simple Words

#aws #devops #network #security

Comments

2 min read

AWS Network Firewall: A DevOps Guide in Simple Words

Manoj — Tue, 15 Jul 2025 12:05:48 +0000

I bet most of you who’ve worked on AWS even a little, chances are you’ve built a VPC — added subnets, configured route tables, attached internet gateways, and locked things down with network access control lists and security groups.

But there comes a moment when you realise that security groups and NACLs aren’t enough. Maybe an application is reaching out to sketchy IPs. Maybe you need to block traffic to specific IP addresses, or log every packet leaving your VPC. That’s where AWS Network Firewall comes into the picture — and to be honest, not enough people use it to its full potential. In this article, I’ll walk you through what it is and why it’s required.

What is AWS Network Firewall?

AWS Network Firewall is a managed firewall service that helps you control network traffic at a much more granular level that security groups and NACLs.

To help you understand better, think like this:

Security Groups: Act like a bouncer at EC2 level
NACL: Watch traffic at subnet border
Network Firewall: It is a smart customisable checkpoint inside your VPC for deep inspection, filtering and rule-based control

And the best part? You don’t have to manage infrastructure. AWS will do all the necessary weight lifting for you.

What Can It Do?

Block/Allow/Alert traffic based on IP, Port and Protocol.
Detect and stop traffic to known bad/malicious IPs.
Block access to an entire country/countries.
Log and analyse traffic using VPC Flow Logs, Cloudwatch or S3.
Integrate with Suiracata rules ( an open source threat detection engine).

Why Use IT?

Block outbound traffic to malicious IPs
Ensure your app(s) only connect to approved APIs / IPs / Domains
Monitor for unusual connection patterns
You can use suricata rules to detect intrusion attempts( for example: unauthorised sql access attempt, unauthorised access to exploit port to name a few)
We all know this kind of control is hard to build using only NACL/Security Group.

Basic Setup Overview

Create a firewall policy: Define what should be allowed/blocked.
Add rule groups- like: stateless (quick filters) and stateful (connection aware, deeper checks)
Create a firewall - attach rule groups and firewall policy to your firewall, attach the firewall to your vpc
Configure routing: Traffic must flow through firewall (typically via a dedicated subnet)

Thanks for reading! If you’re using Network Firewall or curious about understanding it better, hit reply — happy to discuss or write a follow up post.

cheers

Custom AMI without cloud-init? Here's how it broke my EC2-Instance

Manoj — Mon, 07 Jul 2025 08:49:11 +0000

So here's a quick post on something that cost me a good bit of time.

I had launched an EC2 instance from a custom AMI, and right after the boot it started failing EC2 instance reachability checks. The instance was running, but I couldn't SSH into it. So I checked system logs via EC2 console and found this:

network: Bringing up interface ens5: ERROR : [/etc/sysconfig/network-script/ifup-eth] Device ens5 has different MAC address than expected, ignoring

what actually happened?

I had created an AMI from a running instance without cloud-init installed. So the image had the original MAC address hardcoded in /etc/sysconfig/network-scripts/ifcfg-ens5. When I launched a new ec2 instance from this AMI, AWS assigned a new MAC address to the network interface but the OS was still looking at the old one. It was a classic mismatch and therefore network failed to initialise, and so did the reachability check.

How I debugged it?

Since I couldn't SSH into the instance, here's what i did:

Stopped the instance, detached it's root volume and attached it another working instance in same availability zone as a secondary volume(e.g., /dev/xvdf).
Mounted the volume to a temporary directory:
sudo mkdir /mnt/temp
sudo mount /dev/xvdf1 /mnt/temp
Edit the network config file and delete the line with HWADDR compeletely.
vi /mnt/temp/etc/sysconfig/
Unmounted the volume, detached it and attached back to original instance as root volume and started it.

Hurray! I was able to connect succeessfully

How to prevent this from happening?

Install cloud-init before creating your custom AMI