DEV Community: Suleiman Abdulkadir

I built a pipeline that rolls itself back when production breaks

Suleiman Abdulkadir — Fri, 05 Jun 2026 22:04:55 +0000

Deployments that break silently at night bother me. By the time someone checks Slack in the morning, users have been hitting 502s for hours. I built ShipGuard because I wanted the infrastructure to fix itself before I even knew something was wrong.

It's a CodePipeline that does blue/green deployment with canary traffic shifting to EC2. If the new version starts returning 5xx errors, CodeDeploy shifts traffic back to the old version and kills the broken instances. I don't have to do anything.

Three CloudFormation templates. Everything in source control. Nothing configured by hand.

The pipeline flow

Push to main. Everything after that is automatic:

Pull source from GitHub
npm audit, Trivy, git-secrets run first. High or critical vuln? Build dies.
Tests run, TypeScript compiles, artifact gets packaged
Deploy to staging (one instance, in-place)
Email lands asking me to approve
I approve. Blue/green starts on production.
10% of traffic routes to the new version for 5 minutes
CloudWatch alarm stays quiet? Remaining traffic shifts over.
Old instances terminated. Done.

If the alarm fires during steps 7 or 8, traffic goes 100% back to the previous version. Green instances die. I get an email explaining which alarm triggered the rollback.

Things that bit me

TimeBasedCanary doesn't work for EC2

I spent an afternoon trying to configure TimeBasedCanary in a custom deployment config. CloudFormation accepted the template at lint time and then failed at deploy time with "Traffic routing configuration should be null for Server deployment configuration."

Turns out canary percentage configs only exist for ECS and Lambda. For EC2, the ALB and target group weights handle traffic shifting, not a CodeDeploy config. Nowhere in the docs does it say this clearly; you just find out when it breaks.

IAM role chain from hell

Four roles. They all need to trust different AWS services, and they all need slightly different permissions:

Pipeline role needs iam:PassRole to hand off to CodeDeploy
CodeBuild role needs S3 access to the artifact bucket plus CloudWatch Logs
CodeDeploy role needs EC2, ASG, ALB, S3
EC2 instance profile needs to pull from S3 and push CloudWatch metrics

Miss one permission and you get "Access Denied" with no indication of which call failed or which role is the problem. I iterated on this more times than I'd like to admit.

You need an AMI in your Launch Template

This one's embarrassing. cfn-lint doesn't catch a missing ImageId in a Launch Template. CloudFormation doesn't catch it either, until the ASG actually tries to spin up an instance and fails. The fix is an SSM parameter that resolves to the latest Amazon Linux 2023 AMI:

LatestAmiId:
  Type: AWS::SSM::Parameter::Value<AWS::EC2::Image::Id>
  Default: /aws/service/ami-amazon-linux-latest/al2023-ami-kernel-default-x86_64

CodeStarNotifications has different tag syntax

I deployed the pipeline stack three times before figuring this out. Every other resource in CloudFormation uses Tags as a list:

Tags:
  - Key: Name
    Value: my-rule

AWS::CodeStarNotifications::NotificationRule wants a map:

Tags:
  Name: my-rule

CloudFormation gives you "Properties validation failed" with no hint about what's wrong. I found the answer buried in a GitHub issue after 20 minutes of searching.

Security scanning

Three scans run before tests. If any exit non-zero, the build stops, and nothing reaches staging:

pre_build:
  commands:
    - npm audit --audit-level=high
    - trivy fs --severity HIGH, CRITICAL --exit-code 1
    - git secrets --scan

No third-party service needed. npm audit is already there, Trivy downloads in a few seconds during install, and git-secrets is an AWS open source tool that's one clone away. The pipeline sends a notification identifying which scan killed the build.

The rollback alarm

Production5xxAlarm:
  Type: AWS::CloudWatch::Alarm
  Properties:
    MetricName: HTTPCode_Target_5XX_Count
    Namespace: AWS/ApplicationELB
    Statistic: Sum
    Period: 60
    EvaluationPeriods: 1
    Threshold: 10
    ComparisonOperator: GreaterThanThreshold
    TreatMissingData: notBreaching

10 or more 5xx errors in 60 seconds trigger the alarm. CodeDeploy has DEPLOYMENT_STOP_ON_ALARM in its rollback config, so it catches the alarm and reverses the deployment.

TreatMissingData: notBreaching is worth noting. Without it, the alarm fires during periods with zero traffic (nights, weekends) because "no data" defaults to "assume breach." That caused a false rollback the first time I tested this on a weekend.

What I'd change next time

I'd probably use ECS Fargate. CodeDeploy's blue/green for ECS actually supports TimeBasedCanary properly, so you can do true 10% -> 50% -> 100% shifts with observation windows between each step. EC2 blue/green is coarser. You get "new instances with traffic control," but the fine-grained percentage steps aren't natively there.

I stuck with EC2 because I wanted to learn how the instance-level deployment mechanics work. Worth it for the education. Probably not what I'd pick for a real production system in 2026.

Repo

Public: github.com/suletetes/ShipGuard

Three templates, one buildspec, one appspec, four shell scripts. Deploy staging first, then production, then pipeline. Push code. Pipeline picks it up.

Costs about $45/month with everything running. ALBs are most of that ($16 each). Tear down staging when you're not testing.

Stack

CodePipeline, CodeBuild, CodeDeploy
EC2 Auto Scaling behind ALBs
CloudWatch alarms
SNS
S3
CloudFormation
TypeScript/Express (the app being deployed)

If you've done the "deploy a Lambda" tutorials and want something closer to what production infrastructure actually looks like, this hits the right problems. Cross-stack references, IAM chains, blue/green mechanics, alarm-driven automation. The stuff that takes four tries to get right and nobody warns you about in advance.

The deployed infrastructure

Here's what the stacks actually create in AWS:

CloudFormation stacks

VPC subnets (multi-AZ)

Security groups

Load balancers

Target groups (blue and green)

Auto Scaling groups

EC2 instances

S3 artifact bucket

SNS topics

Migrating a MERN app to AWS serverless (and what broke)

Suleiman Abdulkadir — Thu, 28 May 2026 21:02:34 +0000

I built Taskly about a year ago. Standard MERN stack, ran on a $10/month VPS with PM2 and nginx. It worked fine. Nobody was complaining.

I migrated it to AWS serverless anyway. Partly to learn, partly because I was mass applying to DevOps roles and needed something real to talk about in interviews. "I deployed a hello world Lambda" doesn't cut it.

The app

Task management for small teams. Tasks, projects, teams, calendar, notifications, avatar uploads, productivity stats. About 15 API routes, 6 Mongoose models. React frontend with Context API nothing fancy but enough moving parts that the migration wasn't trivial.

Original stack: Express, session auth, MongoDB, Cloudinary, Resend for emails.

Where I ended up

Request flow: users hit CloudFront for the React app, WAF-filtered API Gateway for the backend. Lambda runs Express via serverless-express, talks to DocumentDB in a private VPC, pushes events to EventBridge, and queues emails through SQS to SES.

Network layer: VPC with public and private subnets across two AZs. Security groups restrict DocumentDB to Lambda only (port 27017). VPC endpoints for Secrets Manager. NAT gateway for outbound.

Deployment: GitHub Actions authenticates via OIDC (no stored keys), packages Lambda, does canary traffic shifting, monitors error rate, auto-rolls back if anything breaks.

VPC with private subnets. Security groups. NAT gateway. Secrets Manager. 12 Terraform modules, about 2000 lines of HCL. GitHub Actions CI/CD with OIDC auth and canary deployments.

Took about 3 weeks of evenings.

Sessions broke immediately

First thing. My Express app used express-session with a MongoDB store. Lambda spins up new instances per request. Sessions were just gone.

I ended up with dual mode auth. Sessions for local dev (easy to debug, familiar), Cognito JWT for production (stateless, works with Lambda). The middleware checks which environment it's running in:

if (process.env.COGNITO_USER_POOL_ID && process.env.COGNITO_CLIENT_ID) {
  return validateCognitoToken(req, res, next);
} else {
  return req.isAuthenticated() ? next() : res.status(401).json({...});
}

Not elegant. Works.

DocumentDB is almost MongoDB

95% compatible. The other 5% shows up at the worst times.

Some aggregation stages behave differently. The connection needs a TLS certificate bundle you have to download from AWS and ship with your Lambda zip. I only caught these because I tested against the actual cluster, not just local Mongo.

If I'd only tested locally, these would have been production bugs discovered at 2 am.

Terraform got out of hand fast

Started with one main.tf. Lasted a day.

Split into modules: vpc, lambda, iam, s3, documentdb, waf, ses, api-gateway, cloudfront, secrets, monitoring, disaster-recovery. State in S3 with DynamoDB locking.

The thing about Terraform: plan says "2 to add, 0 to destroy" and you feel safe. Then apply takes 15 minutes because NAT gateways are slow. And if it fails halfway, you get to learn about terraform state rm.

Security groups

The mental model that clicked:

Lambda SG: egress all (needs DocumentDB, NAT, VPC endpoints)
DocumentDB SG: ingress port 27017 from Lambda SG only
VPC Endpoints SG: ingress port 443 from Lambda SG only

Three groups. Database unreachable from internet. Lambda can reach database. Done.

Canary deploys

The CI/CD pipeline packages the code, uploads to S3, publishes a new Lambda version, shifts 10% of traffic to it, waits 5 minutes watching CloudWatch error metrics, and either promotes to 100% or rolls back.

Saved me twice. Once from a missing env var, once from a dependency that worked locally but not in the Lambda runtime.

What I'd change

Skip the VPC for Lambda if possible. The ENI attachment adds cold start latency, and NAT gateways cost $32/month each. DocumentDB forces you into a VPC though, so I was stuck.

Write smaller Terraform modules. My IAM module has 8 policies in one file. Should be separate.

Set up CI/CD first, not last. I did manual deploys for weeks. Dumb.

Cost

Old VPS: $10/month
AWS serverless: ~$45/month (mostly NAT gateway and DocumentDB)

More expensive. But I actually understand VPCs, IAM, security groups, and Terraform now. That's worth more than $35/month to me.

Code

github.com/suletetes/taskly

Infrastructure in infrastructure/, Lambda handler in backend/lambda/handler.js, CI/CD in .github/workflows/.

If you're doing something similar, start with VPC and DocumentDB. They take the longest to provision and have the most surprises. Get those working, then add Lambda and API Gateway on top.

I Built a SaaS Platform From Scratch. Here's How I Architected It on AWS.

Suleiman Abdulkadir — Tue, 28 Apr 2026 00:03:29 +0000

So I've been working on something for a while now. It's called TechVerse. It's a SaaS e-commerce platform, and I built the whole thing from the ground up using the MERN stack.

I want to talk about the cloud architecture side of things. Not the textbook version. The real version. The one where you're staring at your screen at 2am trying to figure out why your WebSocket connections keep dropping, or why your API response times just spiked to 4 seconds.

Let me walk you through it.

The Problem I Was Trying to Solve

Here's the situation. There are thousands of tech retailers in Nigeria who sell laptops, phones, accessories, all kinds of stuff. Most of them run their entire business from a physical shop. No website. No online store. Nothing.

Why? Because getting a custom e-commerce site built costs a fortune. And the international platforms? They charge in dollars. That's a dealbreaker when your local currency fluctuates every other week.

So I thought, what if I built a SaaS platform that lets these businesses spin up a professional online store for a fraction of the cost? Pricing in local currency. Optimized for local internet speeds. Local payment gateways baked right in.

That was the idea. Now I had to figure out how to actually build and deploy it.

Choosing the Stack

I went with what I know best. MongoDB, Express, React, Node. The classic MERN stack. But I made some specific choices that matter for production.

React 19 with Vite 7 on the frontend. Vite is ridiculously fast for builds. The dev experience alone is worth it, but more importantly, the production bundles are tiny. That matters when your users are on 3G connections.

Node.js 20 with Express on the backend. Nothing fancy here. It works. It scales. The ecosystem is massive. I added Socket.io for real-time features like order notifications and live inventory updates.

MongoDB Atlas for the database. I considered self-hosting on EC2, but honestly, managed databases save you so much headache. Automated backups, point-in-time recovery, monitoring. All handled. I went with an M10 cluster to start.

Redis through ElastiCache for caching and session management. This was a game changer for performance. More on that later.

The Architecture (And Why Each Piece Exists)

Alright, let's break down the actual AWS setup. I'll explain why I chose each service, not just what it does.

The Entry Point: Route 53 and CloudFront

Every request starts at Route 53 for DNS resolution. Simple enough. But the real magic is CloudFront.

CloudFront is AWS's CDN, and it has edge locations in Lagos. That's huge. It means my users in Nigeria are hitting a server that's physically close to them, not one sitting in Ireland or Virginia.

I configured CloudFront to do two things. Static file requests go to an S3 bucket where my React build lives. API requests get forwarded to my backend through the Application Load Balancer. One domain, two destinations. Clean and simple.

I also attached an ACM certificate here. Free SSL. No reason not to use it.

The Frontend: S3

The React app gets built by Vite, and the output goes straight into an S3 bucket. No servers involved. S3 serves static files incredibly well, and combined with CloudFront caching, my frontend loads in under 2 seconds on a 3G connection.

I set up error page redirects so that 404s go back to index.html. That's essential for single-page apps with client-side routing. Without it, refreshing any page that isn't the root would give you a blank screen.

The Backend: EC2 with Auto Scaling

Here's where it gets interesting. My Node.js API runs on EC2 instances inside a public subnet. I have two instances behind an Application Load Balancer, with an Auto Scaling Group that can spin up to four instances based on CPU utilization.

Why not Fargate or Lambda? Honestly, for a WebSocket-heavy application, EC2 gives you more control. Lambda has cold starts that would kill the real-time experience. Fargate is great but adds complexity I didn't need yet. EC2 with a good Auto Scaling policy hits the sweet spot.

The ALB distributes traffic evenly and handles health checks. If one instance goes down, traffic automatically routes to the healthy ones. No manual intervention needed.

The Data Layer: MongoDB Atlas and ElastiCache

MongoDB Atlas sits in a private subnet. It's peered with my VPC, so the connection is fast and secure. No public internet involved.

ElastiCache Redis handles three things for me. Session storage, so users stay logged in across multiple EC2 instances. Response caching, so repeated database queries don't hit MongoDB every time. And rate limiting, so I can throttle abusive requests without adding load to my application servers.

Before I added Redis, my average API response time was around 400ms. After? Under 200ms. That's the kind of improvement that users actually feel.

Monitoring and Email: CloudWatch and SES

CloudWatch collects logs and metrics from everything. EC2 instances, the load balancer, Redis, all of it. I set up alarms for CPU spikes, memory usage, and error rates. If something breaks at 3am, I get a notification.

Amazon SES handles transactional emails. Order confirmations, password resets, shipping updates. It's cheap and reliable. Way better than trying to manage your own SMTP server.

Backups

Everything gets backed up to S3. MongoDB Atlas handles its own backups, but I also dump snapshots to S3 for extra safety. CloudWatch logs go there too. Storage is cheap. Losing data is not.

The CI/CD Pipeline

This part I'm actually proud of. GitHub Actions handles everything.

When I push to the main branch, here's what happens. The pipeline runs tests. If they pass, it builds the React frontend and syncs it to S3. Then it deploys the backend to EC2 through the load balancer. Zero downtime. The whole process takes about 4 minutes.

I also have separate workflows for staging and production. Feature branches deploy to staging automatically. Production requires a manual approval step. That one extra click has saved me from shipping broken code more than once.

Stripe Integration

Payments go through Stripe. The integration is bidirectional. My EC2 instances send payment requests to Stripe's API, and Stripe sends webhook events back for things like successful charges, refunds, and subscription updates.

I handle webhooks on a dedicated endpoint with signature verification. Never trust incoming data without verifying it. That's a lesson you only need to learn once.

What This Actually Costs

Here's the part everyone wants to know. My monthly AWS bill for this setup is roughly $90 to $110. That breaks down to about:

EC2 instances (t3.small): $25-30
MongoDB Atlas (M10): $57
ElastiCache Redis (t3.micro): $12
S3 and CloudFront: $5-10
Route 53 and misc: $2-3

For a production SaaS platform with auto-scaling, CDN, managed database, caching, monitoring, and automated deployments, that's pretty reasonable. It can comfortably handle hundreds of concurrent users and thousands of daily requests.

Lessons I Learned the Hard Way

Let me share a few things that bit me during this process.

WebSocket connections through CloudFront need specific configuration. You have to set up the right cache behaviors and forward the Upgrade header. I spent an entire weekend debugging why Socket.io worked locally but not in production. The fix was three lines of CloudFront config.

Don't skip the VPC design. I initially put everything in a public subnet because it was easier. Then I realized my database was exposed to the internet. Moved it to a private subnet immediately. Take the time to set up your network properly from day one.

Redis connection pooling matters. My first implementation created a new Redis connection for every request. Under load, I was hitting connection limits within minutes. Connection pooling fixed it instantly.

Auto Scaling needs a cooldown period. Without it, your instances will scale up and down like a yo-yo. I set a 5-minute cooldown, and the scaling became smooth and predictable.

Environment variables are not optional. I had a brief moment where I accidentally committed a JWT secret to GitHub. Rotated it immediately and moved everything to AWS Systems Manager Parameter Store. Use it. It's free.

What's Next

The architecture I have now works well for the current stage. But I'm already thinking about what comes next as the platform grows.

I want to add a message queue. Probably SQS. Right now, some of my background tasks like sending emails and processing images run synchronously. That's fine with 50 users. It won't be fine with 500.

I'm also looking at moving to containers eventually. ECS with Fargate would give me better resource utilization and simpler deployments. But that's a migration I'll do when the current setup starts showing strain, not before.

And I need better observability. CloudWatch is good for basics, but I want distributed tracing. Probably AWS X-Ray or something like Datadog. When you have multiple services talking to each other, you need to see the full picture of a request's journey.

Final Thoughts

Building a SaaS platform is one thing. Making it production-ready on AWS is a completely different challenge. It forces you to think about things you never consider during development. Network security. Scaling behavior. Cost optimization. Disaster recovery.

But here's what I've realized. You don't need a perfect architecture on day one. You need one that works, that you understand, and that you can evolve. Start simple. Add complexity only when you have a real reason to.

If you want to dig into the code, the full project is on GitHub: TechVerse on GitHub