Originally published on Naveed Ahmed Tech Blog.
The 3:00 AM Wake-Up Call You Never Want
It started with an automated email that makes every on-call engineer's stomach drop:
From: AWS Trust & Safety
Subject: Abuse Report – Suspicion of Unauthorized Vulnerability Scanning from Account 896xxxxxxxxx
Activity: Outbound TCP connections completing three-way handshakes to port 23 (Telnet) and port 2323 against remote hosts.
Our first reaction was disbelief.
We weren't running sketchy scripts. We weren't operating internet scanners. In fact, our application was running on AWS App Runner—a fully managed container PaaS. There were no raw EC2 instances to patch, no Kubernetes control planes to defend, and no open SSH ports.
Yet, third-party honeypots across the globe were registering full TCP handshakes originating from our AWS infrastructure.
Our container had been weaponized into a botnet scanner probing WordPress sites and network hardware across the web.
Here is the step-by-step investigation of how an unauthenticated framework vulnerability bypassed our cloud defenses, why our initial network fix caused a cascading database outage, and the vital DevSecOps lessons every engineering team needs to learn about managed cloud services.
Act I: The Red Herring (Chasing Ephemeral IP Ghosts)
The abuse notice reported two specific source IP addresses: 52.76.184.184 and 13.228.212.140.
Naturally, our incident response playbook began with inventory cross-referencing:
# Checking all allocated Elastic IPs in the region
aws ec2 describe-addresses \
--region ap-southeast-1 \
--query 'Addresses[*].[PublicIp, AllocationId, InstanceId]' \
--output table
Result: Zero matches. Neither IP belonged to our production EC2 instances, bastion hosts, or load balancers.
The App Runner Egress Trap
If you deploy containers on AWS App Runner with Default Egress, your traffic does not originate from your VPC. Instead, outbound requests route through AWS’s regional pool of shared, dynamic public IPs.
[App Runner Container]
│
▼ (Default Egress)
[Shared Regional Public IP Pool (52.76.x.x / 13.228.x.x)]
│
▼ (Port 23 Handshake)
[Third-Party Honeypots & External Hosts]
The IPs reported in the abuse complaint belonged directly to that regional Singapore pool. Two containerized services were actively running on that compute engine: our primary web portal and a micro-service frontend built on Next.js.
The call was coming from inside the house.
Act II: The Anatomy of Patient Zero
We pulled CloudWatch application log groups across both services for the exact timestamps cited in the abuse report.
In the micro-service logs, we found the smoking gun:
[Error: RCE_RES]
at Object.runInternalAction (/app/.next/server/chunks/...)
at async eval (eval at <anonymous> ...)
TypeError: Cannot read properties of undefined (reading 'j')
Repeated execution traces aligned to the second with the port 23 probing.
The Attack Vector: Unauthenticated RCE in React Server Components
Our micro-service was pinned to next@15.1.7.
Recent critical vulnerabilities in React Server Components (RSC) and unauthenticated action dispatching allowed remote attackers to craft malicious HTTP payloads containing serialized gadget chains.
Because the endpoint was publicly accessible:
- Inbound Exploitation: The attacker fired an HTTP POST request carrying an unauthenticated payload directly to the public URL.
- Deserialization & Execution: The vulnerable Next.js runtime deserialized the payload, spawning a detached shell process inside the container.
- Payload Download: The background process fetched a compact scanning tool designed to hunt for exposed WordPress plugins, Telnet services, and insecure IoT devices.
[Attacker] ─── (1) Malicious POST with RSC Gadget ───▶ [Next.js Container]
│
(2) Unauthenticated Deserialization
│
▼
[Detached Child Shell]
│
[External Targets] ◀── (3) Outbound Port 23 Scans ────────────┘
Act III: The Band-Aid & The Cascading Outage
Our first impulse was to stop the bleeding immediately by severing the outbound scanning path.
Because default App Runner egress bypasses your VPC, Security Groups and Network ACLs do not apply. To block port 23, we had to reconfigure App Runner to route traffic through our VPC.
1. Enforcing VPC Egress & Blocking Ports
We updated the VPC Network ACL (acl-053e6828260f5af57):
-
Rule 90:
DENYoutbound TCP destination port 23 (0.0.0.0/0) -
Rule 91:
DENYoutbound TCP destination port 2323 (0.0.0.0/0) -
Rule 100:
ALLOWall other outbound traffic
Then, we reconfigured the App Runner service to attach to our custom VPC Connector:
aws apprunner update-service \
--service-arn "arn:aws:apprunner:ap-southeast-1:896919141349:service/app-service/xxxx" \
--network-configuration '{"EgressConfiguration":{"EgressType":"VPC","VpcConnectorArn":"arn:aws:apprunner:ap-southeast-1:xxxx:vpcconnector/Main-VPC-connector"}}'
2. The Unintended Outage: PostgreSQL Drops Dead
Minutes after deploying the change, our service monitors alerted:
SequelizeConnectionError: connect ETIMEDOUT 18.138.36.160:5432
at ConnectionManager.connect (/app/node_modules/sequelize/...)
Why did this happen?
Under default egress, the container connected to the database's public endpoint over the open internet. But once forced inside the VPC connector:
- Outbound traffic was routed through our subnets toward a NAT Gateway.
- The traffic reached the database coming from the NAT Gateway's Elastic IP, not the old dynamic IP.
- The RDS Security Group dropped the packets because the NAT Gateway IP wasn't whitelisted.
We had stopped the outbound scanning, but in doing so, we knocked our own database offline.
3. The Realization: Outbound Blocking is a Band-Aid
Even if our network block stopped the container from scanning WordPress hosts, the front door was still unlocked.
Any attacker visiting our public domain could still submit the exploit payload, execute arbitrary code inside our container, dump environment variables, or attempt lateral movement within our private VPC subnets.
The only true remediation was at the code level.
Act IV: Recovery & Permanent Hardening
To bring the service back safely without architectural regressions, we implemented a 4-step hardening plan:
Step 1: Framework Patching (The Root Cause)
We bumped next and eslint-config-next out of the vulnerable range:
"dependencies": {
- "next": "15.1.7",
+ "next": "^15.1.8",
"react": "^19.0.0",
"react-dom": "^19.0.0"
}
We verified the fix by inspecting lockfiles and running automated security audits:
npm audit --audit-level=high
Step 2: Proper Database Routing via Private Endpoints
Instead of hair-pinning traffic out to the internet through a NAT Gateway to reach our own RDS instance, we routed PostgreSQL traffic through the VPC's internal private CIDR block:
# Authorizing RDS access directly from the VPC CIDR
aws ec2 authorize-security-group-ingress \
--group-id sg-0123456789abcdef0 \
--protocol tcp \
--port 5432 \
--cidr 10.0.0.0/16
Step 3: Secret Invalidation & Rotation
Because remote code execution grants process memory access, we assumed any secret present in the container runtime was compromised:
- Invalidated and re-issued database credentials.
- Rotated API tokens and session signing secrets.
- Audited IAM roles for unauthorized CloudTrail events during the dwell window.
Act V: Key Takeaways for DevSecOps Teams
1. The Managed Cloud Illusion
Cloud providers manage the underlying infrastructure, operating system, and container hypervisor. They do not manage your application runtime or dependencies. An RCE inside a container running on App Runner, ECS Fargate, or Google Cloud Run is just as dangerous as an RCE on bare metal.
2. Network ACLs Cannot Cure Application Insecurity
Firewalls, WAF rules, and egress restrictions are secondary layers of defense. If your application parses unauthenticated input insecurely, perimeter security only limits where the attacker can pivot—it does not stop the compromise.
3. Shift-Left Must Be Enforced, Not Recommended
Vulnerabilities like framework-level RCEs are disclosed and patched rapidly. If your CI/CD pipeline does not fail builds on critical CVEs (npm audit, Snyk, or Trivy), vulnerable dependencies will quietly make their way into production containers.
💬 Discussion
Have you ever faced an incident where a managed cloud service was exploited due to an application-layer dependency? How does your team handle automated CVE gates in CI/CD?
Drop your thoughts in the comments below!
Connect with Naveed Ahmed
- 🌐 Portfolio & Production Systems: https://naveedkumbhar.com
- ✍️ Technical Architecture Blog: https://blog.naveedkumbhar.com
- 💼 LinkedIn Profile: https://pk.linkedin.com/in/naveedkumbhar
- 🐙 GitHub Projects: https://github.com/naveedkumbhar
- 🐦 X / Twitter: https://x.com/naveedkumbhar
- 🎮 DevOps Arcade & Drills: https://games.naveedkumbhar.com
- 🛠️ 970+ SRE Interview Scenarios: https://interview.naveedkumbhar.com
- ☸️ Kubernetes Mastery Path: https://k8s.naveedkumbhar.com
- 📬 DevOps Dispatch Newsletter: https://news.naveedkumbhar.com
Top comments (0)