A real-world VPC + Lambda DNS puzzle that'll make you double-check your security groups forever.
🧩 The Scenario
Your team deploys a Lambda function that needs to read and write to a DynamoDB table. The setup looks solid:
- ✅ Lambda is attached to a VPC
- ✅ A VPC Endpoint for DynamoDB is configured (Gateway type)
- ✅ Lambda's IAM role has the correct DynamoDB permissions (
dynamodb:GetItem,dynamodb:PutItem, etc.) - ✅ The Lambda's Security Group has an outbound rule:
All TCP → 0.0.0.0/0 - ✅ DNS resolution and DNS hostnames are enabled on the VPC
- ✅ An EC2 instance in the same VPC can successfully resolve the DynamoDB DNS name AND access the table
Yet, Lambda invocations keep failing with a connection/timeout error. It simply cannot resolve the DynamoDB DNS name.
🤔 Take a Moment — What Do You Think Is Wrong?
You've verified IAM. You've verified the VPC endpoint. DNS is enabled at the VPC level. Even an EC2 in the same VPC works perfectly.
The Lambda function is not in a public subnet issue. It's not a missing route table entry.
Seriously — take 30 seconds. What's different between how an EC2 instance handles DNS vs. a Lambda function locked down by a Security Group?
...
...
...
...
...
...
Think about the OSI model. 🤔
...
...
...
...
...
...
What protocol does DNS primarily use?
...
...
...
...
...
...
💡 The Answer
The Lambda's Security Group was missing an outbound rule for UDP traffic.
Specifically, this rule was absent:
| Type | Protocol | Port Range | Destination |
|---|---|---|---|
| Custom UDP | UDP | 53 | 0.0.0.0/0 |
Or more broadly (to cover both TCP and UDP for DNS):
| Type | Protocol | Port Range | Destination |
|---|---|---|---|
| All traffic | All | All | 0.0.0.0/0 |
🔬 Why This Happens — The Root Cause Explained
DNS Uses UDP (Primarily)
DNS queries almost always go out over UDP port 53. TCP port 53 is used only for larger responses (like zone transfers or responses exceeding 512 bytes). For typical hostname resolution — like resolving dynamodb.us-east-1.amazonaws.com — UDP is the default protocol.
The Security Group had:
Outbound: All TCP → 0.0.0.0/0 ✅
Outbound: All UDP → 0.0.0.0/0 ❌ MISSING
So when Lambda tried to resolve the DynamoDB endpoint DNS name before establishing a connection, the UDP DNS query was silently dropped by the Security Group. No DNS resolution = no connection = timeout.
Why Did EC2 Work But Lambda Didn't?
This is the clever part. The EC2 instance likely had a broader Security Group — perhaps All traffic → 0.0.0.0/0 — or one that explicitly included UDP. Since nobody usually thinks about restricting outbound on EC2 instances during testing, it just worked.
Lambda, being more of a "locked-down by default" environment in VPCs, often gets a tighter security group — and TCP-only outbound is a very common mistake.
Why Didn't the VPC DNS Setting Matter Here?
enableDnsResolution on the VPC controls whether instances can use the Route 53 Resolver (at the VPC base IP + 2, e.g., 10.0.0.2). But that resolver is still accessed over UDP port 53 from the Lambda's network interface. If the Security Group blocks outbound UDP, the Lambda ENI (Elastic Network Interface) can never reach the resolver — regardless of the VPC-level DNS settings.
✅ The Fix
Update the Lambda's Security Group outbound rules to include UDP:
Option 1 — Minimal / Precise (recommended for production):
| Direction | Type | Protocol | Port | Destination |
|---|---|---|---|---|
| Outbound | Custom UDP | UDP | 53 | VPC CIDR or 0.0.0.0/0 |
| Outbound | Custom TCP | TCP | 443 | 0.0.0.0/0 (for HTTPS to DynamoDB endpoint) |
Option 2 — Simple / Permissive (fine for dev/test):
| Direction | Type | Protocol | Port | Destination |
|---|---|---|---|---|
| Outbound | All traffic | All | All | 0.0.0.0/0 |
🗝️ Key Takeaways
- DNS runs on UDP port 53 — always allow outbound UDP 53 in Lambda security groups when using VPC.
- "All TCP" ≠ "All Traffic" — a very easy mistake to make in the AWS console.
- EC2 working ≠ Lambda will work — EC2 and Lambda often have different Security Groups with different outbound rules.
- VPC Endpoints don't bypass DNS resolution — Lambda still needs to resolve the DynamoDB hostname before the VPC endpoint routing kicks in.
- Silence is not golden in Security Groups — blocked UDP traffic produces no error, just a timeout, making this particularly tricky to debug.
🛠️ Quick Debugging Checklist for Lambda-in-VPC DNS Issues
- [ ] Does the Security Group allow outbound UDP 53?
- [ ] Does the Security Group allow outbound TCP 443 (for AWS service endpoints)?
- [ ] Is DNS Resolution enabled on the VPC?
- [ ] Is DNS Hostnames enabled on the VPC?
- [ ] Is there a route in the subnet's route table pointing to the VPC Endpoint?
- [ ] Does the VPC Endpoint policy allow the Lambda's IAM role?
Found this useful? Drop a 💬 comment with your own sneaky AWS networking gotchas — there are plenty of them out there!
Follow for more real-world AWS and Linux troubleshooting posts in this series.
Top comments (0)