DEV Community: Knirl Amboy

No More Clicking: Moving My AWS Architecture to Terraform

Knirl Amboy — Thu, 22 Oct 2026 10:47:07 +0000

What was I trying to solve?

We can build cloud architectures by clicking through a console. Yes, that works, but only once. It leaves you something that is difficult to review, reliably repeat and the sequence only exists in your memory. I have already built a highly available AWS architecture manually, but what if I need to replicate the build multiple times? This brings up a problem.

How do you make an architecture reproducible, reviewable, and safe to change, instead of a one-time sequence of console clicks?

What did I build?

Taking the web architecture I built manually on the AWS console, I reimplemented it using an Infrastructure as Code tool Terraform that enables me to automate the creation, management and updates of the architecture.

This project took on two stages/forms: first one mainly served as my knowledge and understanding of Terraform then the second was the final form which shows improvements in a lot of areas that practices the standards of Terraform usage or in production environments, which will be talked about in another article post.

Being a new user of this tool, Terraform, I took things easy at the start. It was important for me to understand first the basics and fundamentals of Terraform, how it works.

So, these are what the project intentionally does not include yet:

No variables
No variables.tf
No outputs.tf
No modularization

Everything was hardcoded directly into the resource blocks and every resource lives in one folder instead of being split into reusable modules.

Here are the steps I took

Set up remote state. A small one-time state-bootstrap project created the S3 bucket that holds Terraform’s state.
Put guard rails in first. I created the .gitignore and ran git init before writing any real resources, so state files and secrets could never be committed by accident.
Build in layers. Each layer got a plan and then an apply:

Networking: VPC, subnets, gateways, route tables
Security groups
Database: Multi-AZ RDS
Compute: launch template and Auto Scaling Group
Traffic: target group and load balancer
Storage and monitoring: S3, CloudWatch, SNS

Test in the console and browser. Terraform stops at apply, so load balancing, auto-healing, and database failover were verified by hand.
Tear it all down with one command. terraform destroy
Commit and push the code, with documentation, to GitHub.

Where Terraform made it easier

I never wrote “create the VPC first.” A reference was enough, and Terraform worked out the order:

Security rules stayed correct as the infrastructure changed. Each security group trusts the previous one by reference, not by IP address:

Secrets never appeared in the code. The database password is generated, stored in Secrets Manager, and passed to RDS by reference. It exists nowhere as plaintext.

Every change was previewable. terraform plan showed exactly what would be added or destroyed before anything was touched. Reading plan output became my safety check.

Teardown was one command. terraform destroy removed everything in the correct order. In the console, I had to remember that order myself.

The setup is documented and repeatable. The whole architecture lives in version-controlled files that anyone can read, review, and rebuild.

The Challenges I encountered

1. Deprecated backend parameter**

While setting up remote state for the project, I configured Terraform’s S3 backend using a DynamoDB table for state locking — the standard approach I’d read about in most tutorials. Running terraform init, though, returned a deprecation warning: dynamodb_table was being phased out in favor of a newer parameter, use_lockfile, which relies on S3’s own native conditional-write locking instead of requiring a separate DynamoDB table.

It wasn’t a breaking error — my original setup still worked — but it was a clear signal that the “standard” approach I’d learned was already on its way out. Since I was still early in the build, I switched to use_lockfile immediately rather than build the rest of the project on a pattern Terraform had already flagged for removal.

2. Overnight cost uncertainty

Partway through the build, I had to stop for the night with a NAT Gateway and a Multi-AZ RDS instance already live — two of the most expensive, non-free-tier resources in the whole architecture. I didn’t want to tear everything down and lose my progress, but I also didn’t want to wake up to a billing surprise.

Rather than guess, I worked out the actual math: NAT Gateway at roughly $0.045/hour, multi-AZ RDS at roughly $0.068/hour, multiplied by a ~10-12 hour overnight window — landing at under $1.50 total. That small bit of arithmetic turned an anxious guess into a confident decision.

Key Takeaways from this project:

Tutorials and guide material can lag behind a tool’s actual current best practices — it’s worth paying attention to deprecation warnings even when they don’t block you, since they’re often a preview of what “wrong” will look like in the future.
Cloud cost awareness isn’t just knowing what’s expensive in the abstract — it’s being able to do quick, back-of-envelope math on your specific running resources to make an informed call, instead of either panicking or ignoring the risk entirely.
Missed opportunity or mistake I realized:
I wasn't aware that I can just import my existing AWS resources to terraform state. If I did that, I didn't need to destroy my architecture then recreate it using Terraform.

Doing this approach in reimplementing my web architecture through Terraform, I learned about how Terraform works and how quite helpful it is to make building AWS projects much quicker and efficiently.

Onto the next article, I will discuss what were the improvements for this project to practice production standards.

How I Built a Self-Healing, Multi-AZ Architecture on AWS — And What Broke Along the Way

Knirl Amboy — Thu, 17 Sep 2026 05:29:40 +0000

The Problem I Set Out to Solve

One such example of an AWS project/build can be as simple as setting up one EC2 instance and then installing a web server. The moment that instance crashes or traffic spikes, the whole thing falls over.

I wanted to build something that answers the question when it comes to "high availability": what happens when something fails, and does the system recover without a human stepping in

What did I build?

I set out to build a web architecture with no single point of failure — spanning two Availability Zones, load-balanced, auto-scaling, with a self-healing database — and then to actually break it on purpose to prove it recovers, instead of just trusting the console said everything was configured correctly.

The Architecture

A VPC spanning 2 Availability Zones, with public subnets for the load balancer and private subnets for everything else
1 NAT Gateway for both AZ. Note: This should've been 1 NAT Gateway per AZ, so it should be a total of 2 NAT Gateways The reason for this is explained in this section, where I discussed the trade-offs I did.
An Application Load Balancer distributing traffic to an Auto Scaling Group of EC2 instances
RDS running MySQL in Multi-AZ mode, with a live standby ready to take over
CloudWatch monitoring the whole thing, with an alarm and an SNS to notify me.

The Challenges I encountered

Incident 1: The ALB That Wouldn't Respond

After wiring up the VPC, security groups, target group, and load balancer, I hit the ALB's DNS name in my browser and got:

My first assumption was a security group misconfiguration. So I worked through it methodically:

Checked the ALB's security group — inbound HTTP 80 from 0.0.0.0/0 was correctly set
Checked subnet placement — the ALB was correctly deployed across both public subnets
Checked the ALB's scheme — confirmed "Internet-facing," not "Internal"
Checked the listener — HTTP:80 was correctly forwarding to my target group
Checked Network ACLs at the subnet level — default allow-all rules were untouched

Every single AWS-side configuration was correct. That's the moment it clicked that the problem probably wasn't in AWS at all — it was somewhere between my browser and AWS. I tried the same URL from my phone on cellular data, and then just paid closer attention to what was actually in my address bar.

The issue: I'd pasted the bare DNS name into Edge, and the browser had silently prepended https://. My ALB only had an HTTP listener configured — no HTTPS/443 listener existed at all. The browser was trying to reach a port that wasn't listening for that protocol, and quietly timing out instead of giving a clear "connection refused."

The fix: type the URL explicitly as http://....

What this taught me: when every layer of your infrastructure checks out, stop re-checking the same layer and start questioning your assumptions about the client side. I spent close to 20 minutes re-verifying security groups I'd already verified, when the actual bug was in how I was typing a URL.

Incident 2: The Page That Loaded, But Told Me Nothing

Once the ALB was reachable, my page loaded — but it was supposed to display the serving instance's ID and Availability Zone, and both fields were blank.

My EC2 instances pull this data from AWS's internal instance metadata service, via a request like:

curl -s http://169.254.169.254/latest/meta-data/instance-id

This is a well-documented pattern, and I'd used it based on older references. Apache was running fine — the page loaded — so my instinct was that the problem was somewhere in my HTML generation logic. It wasn't. The metadata request itself was silently failing.

The root cause: Amazon Linux 2023 requires IMDSv2 by default — a token-based authentication flow for metadata requests. My script was using the older tokenless (IMDSv1-style) request pattern, which gets rejected without an explicit error visible in the page output — curl just returned empty, and my script happily wrote empty values into the HTML.

The fix required requesting a session token first, then passing it as a header on every subsequent metadata call:

TOKEN=$(curl -s -X PUT "http://169.254.169.254/latest/api/token" -H "X-aws-ec2-metadata-token-ttl-seconds: 21600")
INSTANCE_ID=$(curl -s -H "X-aws-ec2-metadata-token: $TOKEN" http://169.254.169.254/latest/meta-data/instance-id)

But here's the part that actually tests whether you understand Auto Scaling Groups: fixing the script wasn't enough. My existing instances were already running the old script — updating a Launch Template only affects instances launched after the change, not ones already running. I had to publish the corrected script as a new Launch Template version, set it as default, and then trigger an Instance Refresh — which gradually and safely replaced my running instances with new ones, one at a time, keeping at least one healthy throughout the process. That refresh took about 12–15 minutes for two instances, which is worth knowing going in so you don't assume something's stuck.

Now, load balancer distributes traffic and the instances are spread across two AZs

What this taught me: "no visible error" doesn't mean "no failure." A script can execute successfully from the OS's perspective while still failing at the thing it was actually meant to do. And infrastructure-as-config (Launch Templates) has a subtlety that infrastructure-as-code makes more obvious later: changing the definition doesn't retroactively change what's already running.

Incident 3: Auto Scaling Group "flapping" risk

Initially considered a manual step-scaling setup (scale out at 70% CPU / scale in at 40%). Switched to target tracking at a 50% target instead — reduces configuration surface and lets AWS manage thresholds dynamically rather than risking two closely-tuned static alarms causing rapid scale in/out cycles.

Verification & Testing

Configuration alone isn't proof — each piece of this architecture was actually tested:

Load balancing: confirmed by refreshing the ALB's URL repeatedly and observing the displayed Instance ID alternate between the two running instances.

Auto healing / monitoring: intentionally removed the ec2-sg inbound rule to cut ALB→EC2 connectivity, then confirmed:

Target Group flipped both instances to unhealthy
The CloudWatch alarm transitioned to In alarm

An SNS email notification was recieved

The ASG attempted to replace the unhealthy instances (visible in its Activity log)

Then reverted the rule and confirmed full recovery (targets back to healthy, alarm back to OK).

Database failover: using RDS's "Reboot with failover" option, forced the Multi-AZ database to promote its standby in the second Availability Zone to primary. Verified via the console that the instance's Availability Zone changed before vs. after — confirming automatic failover actually works, not just that the checkbox was enabled.

Trade-Offs I Had to Reason Through

Cost vs. Genuine Fault Tolerance: Multi-AZ RDS

AWS's Free Tier RDS template doesn't offer Multi-AZ as an option at all — it's locked behind the "Production" template, which roughly doubles the database's hourly cost. I had a choice: stay on Free Tier and only be able to describe automatic failover in an interview, or spend a small amount of account credit to actually build and test it.

I chose to spend the credit. The reasoning: the entire point of this project was proving fault tolerance, not just listing services on a resume. Being able to say "I triggered a forced failover and verified the database moved to a different Availability Zone" is a fundamentally different claim than "I know RDS supports Multi-AZ." For a project meant to demonstrate engineering competence, the second claim doesn't hold up well under a follow-up question.

Cost vs. Architecture Purity: NAT Gateways per AZ

The "textbook correct" production pattern is one NAT Gateway per Availability Zone, so that a NAT Gateway failure in one AZ doesn't take down internet access for the private subnet in that same AZ. Doing this at scale, for a real company, is the right call.

For this project, I made a deliberate trade-off in the other direction: I chose to deploy a single NAT Gateway shared across both AZs to manage hourly AWS costs. I recognize that a production-grade environment requires a NAT Gateway per AZ to prevent cross-AZ dependencies and ensure high availability. However, I got concerned on the adding another NAT Gateway as it can quite stack up costs plus, I wasn't fully aware yet if my free credits on the AWS account would cover it. So, I decided to keep costs low for now.

What Actually Mattered Here

If I'm honest, the value of this project wasn't in successfully clicking through the AWS console in the right order. It was in the hours I spent chasing down a browser protocol quirk and a metadata API version mismatch. Production systems don't fail in the ways tutorials describe; they fail in small, unglamorous, easy-to-miss ways, and the skill that matters is having a methodical process for narrowing down the cause instead of guessing.

Next, I'm rebuilding this same architecture in Terraform — taking it from a manual console build to something version-controlled, reviewable, and deployable in minutes, then adding on some improvements.