From FYP to Production AWS: How I Built My First Real Cloud Project (And How Many Times It Broke)

#aws #terraform #devops #microservices

My final year project was done.

90/100. Top 8 at the exhibition. Supervisor happy. Committee happy.

And I had absolutely no idea if I could actually get a job.

The FYP had AWS in it Lambda, DynamoDB, API Gateway. But let's be honest. Zappa does most of the heavy lifting. You're not really doing cloud engineering when a library handles your entire deployment. I knew how to follow a tutorial. I didn't know if I could build something real from scratch.

That gap bothered me.

So I opened an empty folder and decided to find out.

Why Event-Driven

I spent a week just researching what to build before writing a single line of code.

Every job description I read - cloud engineer, DevOps engineer, backend engineer - kept mentioning the same things. Microservices. Event-driven architecture. Decoupled systems. SQS. SNS. EventBridge.

Not because it's trendy. Because it's how real production systems are actually built. Direct service-to-service API calls don't scale. They create tight coupling. One service goes down and it takes everything with it.

Event-driven architecture solves that. Services don't talk to each other directly - they fire events and let the messaging layer handle delivery. The order service doesn't care whether inventory or notifications are running. It fires an event and moves on.

That was the pattern I wanted to learn properly. So I built around it.

What OrderFlow Does

OrderFlow is an order management platform with three microservices:

Order Service - accepts incoming orders, fires an order.created event
Inventory Service - consumes that event, updates stock
Notification Service - consumes that event, sends confirmation

Simple domain. Complex plumbing.

The entire event flow looks like this:

Client Request
      │
      ▼
Application Load Balancer
      │
      ▼
ECS Fargate Cluster
      │
order.created event fired
      │
      ▼
Amazon EventBridge
      │
      ├──► SQS inventory-queue (direct)
      │
      └──► SNS orders-topic
                  │
                  └──► SQS notification-queue (fan-out)

Order service fires one event. EventBridge routes it. SNS fans it out. Both downstream services get what they need without knowing anything about each other.

That's the theory. The reality was messier.

The Part Nobody Puts in Their README

I want to be honest about what building this actually looked like.

This was my first time building something at this level of complexity without a tutorial holding my hand. No template. No starter repo. Just AWS documentation, Stack Overflow, and a lot of CloudWatch logs telling me things were broken.

The Terraform errors came first.

I had used Terraform before on smaller projects but never with a modular structure across networking, ECS, messaging, and observability layers all wired together. The dependency graph alone took me two days to get right. Terraform would plan successfully and then fail mid-apply because a security group wasn't ready before the ECS task definition tried to reference it. I learned more about depends_on in those two days than in any course.

Then ECS refused to run anything.

Services registered as ACTIVE but tasks immediately stopped. No obvious error in the console. I dug into CloudWatch logs and found the container was crashing on startup - environment variables weren't being injected correctly into the task definition. The container was starting, hitting a missing config value, and dying before it could even serve a health check. The ALB kept marking targets unhealthy. The fix was straightforward once I found it, but finding it took hours.

Then EventBridge events were disappearing.

I'd fire an order.created event and nothing would show up in the SQS queue. No error. No DLQ message. Just silence. Turns out my EventBridge rule pattern wasn't matching the event structure I was sending - a JSON field name mismatch between what I was publishing and what the rule was filtering on. One character difference. Two hours of debugging.

Then GitHub Actions kept failing on ECR push.

OIDC setup between GitHub and AWS is genuinely fiddly. The trust policy on the IAM role has to exactly match the GitHub repository path and branch conditions. I had the role ARN right but the trust relationship was slightly wrong - it was accepting tokens from the right GitHub org but not scoping to the right repo. Every push was failing at the assume-role step with a cryptic access denied. Fixed it by reading the IAM trust policy documentation more carefully than I've ever read any documentation.

Each one of these felt like a wall when I hit it. In hindsight they were the whole point.

What the Final Architecture Looks Like

Three Flask microservices running on ECS Fargate behind an Application Load Balancer with path-based routing:

/orders/*     → order-service     (port 5000)
/inventory/*  → inventory-service (port 5001)
/notify/*     → notification-service (port 5002)

Every AWS resource - VPC, subnets, route tables, security groups, ECS cluster, task definitions, ECR repos, ALB, SQS queues, SNS topic, EventBridge rules, CloudWatch alarms, X-Ray group - provisioned through modular Terraform. Zero manual console clicks after the initial setup.

CI/CD via GitHub Actions with OIDC - no AWS access keys stored anywhere. On push, the pipeline assumes an IAM role via web identity token, builds the Docker image, pushes to ECR, and forces a rolling ECS redeployment.

Observability through CloudWatch dashboards tracking ECS CPU and memory across all three services, ALB request count and response time, and SQS queue depth. Alarms fire if CPU crosses 80% or queue depth crosses 100 messages. X-Ray distributed tracing wired across the full request path.

The whole thing runs for ~$22-25/month.

What Event-Driven Actually Taught Me

Before this project, event-driven architecture was a concept I understood intellectually.

After this project, I understand it in my hands.

The difference between calling inventory service directly from order service versus firing an EventBridge event is not just architectural elegance. It's operational resilience. During testing I deliberately stopped the inventory service container while firing order events. The orders kept processing. The events sat in SQS. When inventory came back up it consumed the backlog and caught up. Nothing was lost.

Try that with a direct REST call. The order fails. The user gets an error. You have a support ticket.

That moment - watching the system absorb a failure and recover without intervention - was when cloud engineering stopped feeling like infrastructure configuration and started feeling like system design.

What I Built Next

OrderFlow taught me the patterns. But it was missing something real - actual users, actual tenants, actual production concerns like multi-tenancy, authentication, rate limiting, and cost optimization under real traffic.

So I built StatusNest next. A production multi-tenant SaaS status page platform on AWS, with everything OrderFlow taught me plus the things OrderFlow couldn't teach me.

That story is in my next post.