DEV Community: Sohana Akbar

Why We Ditched npm install in Production (And You Should Too)

Sohana Akbar — Tue, 28 Jul 2026 09:39:19 +0000

Two years ago, we made a change that cut our deployment failures by 80%.

It wasn't a fancy new architecture. It wasn't a microservices overhaul. It was a simple switch from npm install to npm ci --omit=dev in our production Docker containers.

That single change saved us from 4 dependency-related incidents in the first year alone. Here's why it matters, how it works, and why your team should make the switch today.

The Problem: npm install is a Liability in Production
Let's be honest—npm install is great for local development. It's flexible, forgiving, and gets the job done. But in production? It's a ticking time bomb.

Here's what can (and did) go wrong:

The Package Lock That Wasn't npm install doesn't respect your package-lock.json the way you think it does. If your lock file is out of sync with your package.json, npm will update dependencies behind your back.

We once had a patch release of a logging library introduce a breaking change that crashed our entire API fleet. The lock file said one version. The container installed another.

The "It Works On My Machine" Nightmare Local installs, CI installs, and production installs can all yield different dependency trees. Different npm versions, different registry responses, different caching behavior—it's chaos.

We spent 3 days debugging a staging vs. production discrepancy that turned out to be a transitive dependency with a slightly different semantic versioning resolution.

The Time Tax npm install is slow. It checks versions, resolves conflicts, and installs development dependencies you'll never use in production. Every minute spent installing is a minute your deployment is vulnerable.

The Solution: npm ci + --omit=dev
The npm team gave us a production-ready alternative years ago, yet so many teams still sleep on it.

What npm ci Does
Strictly respects package-lock.json: If the lock file doesn't match package.json, npm throws an error and fails the build. No silent updates.

Installs from the lock file only: It skips dependency resolution entirely, making it significantly faster.

Removes node_modules first: Guarantees a clean, reproducible installation every time.

What --omit=dev Does
Excludes devDependencies: Those testing frameworks, linters, and build tools? They don't belong in production.

Prevents accidental imports: If a developer mistakenly imports a dev-only package in production code, the error surfaces immediately.

The Winning Combo
dockerfile

Before (Dangerous)

RUN npm install

After (Production-Ready)

RUN npm ci --omit=dev
That's it. One line change. Massive impact.

The Numbers: 4 Incidents We Avoided
Here's exactly what we escaped by making the switch:

Incident #1: The Patch Release Apocalypse
A minor patch update to axios changed how it handled certain headers. npm install pulled it in automatically. The new behavior broke our auth middleware. Downtime: 47 minutes.

With npm ci? The lock file would have pinned the exact version. The patch wouldn't have been installed until we intentionally updated it in a PR.

Incident #2: The Dev Dependency Security Scare
A popular dev tool with a critical CVE was installed as a devDependency but also pulled in production code through an ill-advised import. Our security scanner flagged it. We scrambled to remove it.

With --omit=dev? It would never have been installed in production. No flag, no panic, no fire drill.

Incident #3: The Build Cache Corruption
Our CI pipeline cached node_modules to speed up builds. A corrupted cache interacted with a fresh npm install to produce a broken build. The fix required invalidating caches and redeploying. Time lost: 2 hours.

With npm ci? It removes node_modules before installing, guaranteeing a clean state every time. No cache contamination.

Incident #4: The Inconsistent Staging
Staging and production used different npm versions. The staging install worked. Production install failed. We wasted a full sprint day diagnosing a mismatch that boiled down to a peerDependency resolution difference.

With npm ci? The lock file ensures identical trees regardless of npm version. No more "but it worked in staging."

Why Teams Resist (And Why They're Wrong)
I've heard every objection:

"But npm install is more flexible!"
Flexibility is a bug in production. We want predictability, not flexibility.

"We'll update the lock file anyway."
You will. But npm ci forces the issue. It doesn't let you drift.

"It's just one more thing to remember."
Put it in your Dockerfile. Put it in your CI scripts. Make it the default. You shouldn't have to remember—it should be automatic.

How to Migrate Today
Step 1: Update Your Dockerfile
dockerfile
FROM node:18-alpine

WORKDIR /app

COPY package*.json ./
COPY package-lock.json ./

The magic line

RUN npm ci --omit=dev

COPY . .

CMD ["node", "server.js"]
Step 2: Update Your CI/CD Pipeline
yaml

GitHub Actions example

name: Install dependencies run: npm ci --omit=dev Step 3: Audit Your devDependencies Make sure nothing you actually need in production is accidentally flagged as a dev dependency. Move critical packages to dependencies.

Step 4: Add a Precommit Hook
json
{
"scripts": {
"precommit": "npm ci --omit=dev"
}
}
This catches discrepancies before they ever reach production.

The Results 2 Years Later
Zero dependency-related incidents in the last 18 months.

Faster builds: Our production containers now install in ~40 seconds instead of ~90 seconds.

Peace of mind: We know exactly what version of every package is running in production.

Audit confidence: Security scans are clean because dev-only packages aren't installed.

Don't Wait for an Incident
If you're still using npm install in production containers, you're running on borrowed time. It's not if something will break—it's when.

Make the switch to npm ci --omit=dev. It's free. It's simple. And it might just save your next deployment.

Have you made the switch? Still hesitating? Drop your questions in the comments—let's talk about production npm practices

From 1.2GB to 118MB: How We Slashed Our Docker Image Size and EKS Pull Time to Under 8 Seconds

Sohana Akbar — Sun, 26 Jul 2026 13:20:12 +0000

The moment our EKS cluster stopped sweating

The Problem: When Your Container Image Is Bigger Than Your Application
It started with a simple observation during a deployment review. Our team was staring at deployment logs, watching the same 47-second pattern repeat:

text
Pulling image: 45.3s
Starting container: 1.2s
Forty-five seconds of pull time. Every. Single. Deployment.

Our Docker image had ballooned to 1.2GB — and honestly, we had no one to blame but ourselves. We were shipping node_modules like they were going out of style, including every dev dependency, build tool, and a full test suite in our production images.

The worst part? This wasn't just a "wait a bit longer" problem. It was a:

🚨 Security issue — larger attack surface with unnecessary packages

💸 Cost problem — more storage in ECR and slower scaling events

😤 Developer experience killer — impatient engineers waiting for deployments

The Goal: Fast Pulls, Lean Images
We set three clear objectives:

Image size under 200MB

ECR pull time under 10 seconds in EKS

Zero compromises on runtime performance or security

Here's exactly how we got there.

Step 1: Switch to pnpm (Game Changer)
We were using npm, and our node_modules folder was thicc. Around 800MB of dependencies, including 400MB of dev dependencies we didn't need in production.

We switched to pnpm for three killer benefits:

🔥 Hard Links = Disk Space Savings
pnpm stores packages globally and hard-links them into projects. Our node_modules went from 800MB to 180MB in the build stage.

🚀 Faster Installation
No more re-downloading packages on every build. pnpm's content-addressable storage means if a package version exists, it's reused.

⚡️ Native Workspace Support
Our monorepo setup became significantly simpler.

Here's what our Dockerfile looked like before:

dockerfile
FROM node:18-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci --production=false # 😬 brings ALL dependencies
COPY . .
RUN npm run build
And after:

dockerfile
FROM node:18-alpine AS builder
WORKDIR /app
COPY package.json pnpm-lock.yaml ./
RUN npm install -g pnpm && pnpm install --frozen-lockfile
COPY . .
RUN pnpm run build
RUN pnpm prune --prod # 🎯 strip dev dependencies
Step 2: Multi-Stage Builds Done Right
We were already using multi-stage builds, but poorly. Our final stage was copying everything from the builder stage — including source code, tests, and build artifacts we didn't need.

Before: Copying Too Much
dockerfile
FROM node:18-alpine AS runner
WORKDIR /app
COPY --from=builder /app . # 🚨 copies EVERYTHING
CMD ["node", "dist/main.js"]
After: Selective Copying
dockerfile
FROM node:18-alpine AS runner
WORKDIR /app
ENV NODE_ENV=production

Copy only what we need

COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./

RUN addgroup --system --gid 1001 nodejs && \
adduser --system --uid 1001 nodejs

USER nodejs
CMD ["node", "dist/main.js"]
This single change cut our image size by 60%.

Step 3: Alpine Base Image + Build Optimizations
We were already using Alpine, but we weren't taking full advantage of it. We added:

📦 .dockerignore that actually works
text
node_modules
.git
.md
.env
.DS_Store
coverage
.nyc_output
logs
tmp
🧹 Layer Cleanup
When installing native dependencies, we made sure to clean apt caches:

dockerfile
RUN apk add --no-cache --virtual .build-deps python3 make g++ && \
pnpm install --frozen-lockfile && \
apk del .build-deps
🗜️ Build-time optimizations
dockerfile
RUN pnpm run build && \
pnpm prune --prod && \
rm -rf /root/.npm /root/.cache
Step 4: ECR Lifecycle Policies + EKS Pull Optimization
We also optimized the infrastructure side:

ECR Lifecycle Rules
We set up automatic cleanup for untagged and old images:

Keep last 30 images

Expire untagged images after 14 days

Tag important images with prod-* to bypass cleanup

EKS Node Optimization
Enabled container image caching on our nodes

Configured containerd with aggressive image GC settings

Pre-pulled base images using a DaemonSet

Here's the containerd config we use:

toml
[plugins."io.containerd.gc.v1.scheduler"]
pause_threshold = 0.02
deletion_threshold = 0
mutation_threshold = 100
schedule_delay = "0s"
startup_delay = "100ms"
The Results: Numbers That Made Us Smile
Metric Before After Improvement
Image Size 1.2 GB 118 MB 90.1% reduction
ECR Pull Time 45.3s 7.8s 82.8% faster
Deployment Time 2.1 min 28s 78% reduction
Monthly ECR Cost $47 $9 81% savings
Build Time 4.2 min 1.8 min 57% faster
Before and After: The Docker Image Breakdown
Before (1.2GB)
text
📦 node_modules/ 850MB # dev + prod deps
📁 src/ 12MB
📁 tests/ 28MB # 😱 tests in production image!
📁 coverage/ 15MB # seriously? yes
📄 .git/ 250MB # 🤦‍♂️
📄 other files 45MB
After (118MB)
text
📦 node_modules/ 85MB # prod deps only (pnpm)
📁 dist/ 15MB # built application
📄 package.json 2KB
📄 other files 16MB # minimal essentials
Key Lessons Learned

pnpm isn't just a package manager — it's a strategy
The hard-linking approach fundamentally changes how you think about dependency management in containers.
Copy selectively, live happily
COPY --from=builder /app . is the enemy. Be explicit about what you're copying.
Test environments stay out of production images
Your tests don't need to ship. Neither does your test framework, linter, or build tools.
Layer ordering matters
Put frequently changing files (source code) at the bottom, infrequently changing files (dependencies) at the top.
Measure everything
We couldn't have optimized without metrics. Use docker images, docker history, and tools like Dive to analyze your image layers.

Try This Yourself
Here's a template you can adapt:

dockerfile

Stage 1: Builder

FROM node:18-alpine AS builder
WORKDIR /app
RUN npm install -g pnpm
COPY package.json pnpm-lock.yaml ./
RUN pnpm install --frozen-lockfile
COPY . .
RUN pnpm run build && pnpm prune --prod

Stage 2: Runner

FROM node:18-alpine
WORKDIR /app
ENV NODE_ENV=production
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./
RUN addgroup -g 1001 nodejs && adduser -u 1001 -G nodejs nodejs
USER nodejs
EXPOSE 3000
CMD ["node", "dist/main.js"]
What's Next?
We're not stopping here. Our next targets:

~50MB with distroless images

~3s ECR pull with image pre-fetching

15s deployment times with rolling updates

Full SBOM (Software Bill of Materials) scanning in CI/CD

Final Thoughts
Going from 1.2GB to 118MB wasn't magic — it was discipline. Every byte we removed improved our security, speed, and developer experience.

The best part? We spent about 4 hours total on these optimizations. That's 4 hours to save our team hours every single week.

If you're sitting on a bloated Docker image, start small:

Check what's in your image (docker history)

Move to pnpm

Audit every COPY command

Use Alpine as your base

Measure your improvements

Your EKS cluster will thank you. Your wallet will thank you. And most importantly, your team will thank you when deployments fly.

Have you optimized your container images? What's the craziest thing you've found in a production image? Share your war stories in the comments! 👇

Follow me for more DevOps deep dives 🚀

Docker #Kubernetes #EKS #DevOps #pnpm #ContainerOptimization #AWS #CloudNative

From 5 PHP Nightmares to 1 Node Pipeline: Why Boring Standardization Is My Greatest Achievement

Sohana Akbar — Fri, 24 Jul 2026 15:52:14 +0000

TL;DR: I migrated 5 legacy PHP frontends to our standard Node pipeline. Onboarding dropped from 1 week to 1 day. Here's why "boring" is actually the most exciting thing you can do for your team.

The Confession
I used to hate standardization.

I was that developer who thought every project needed its own unique flavor. Different build tools? Sure. Different folder structures? Why not? Different deployment scripts? Keeps things interesting, right?

Wrong.

Last quarter, I inherited 5 legacy PHP frontends. They were all built by different teams, in different years, with different levels of caffeine-induced insanity. They served the same business purpose, but they might as well have been written in 5 different languages.

Here's what I found:

App PHP Version Build Tool Deployment Method Developer Tears
App A 5.6 None (FTP) Manual drag-drop High
App B 7.0 Gulp SSH + custom bash Very High
App C 7.4 Composer Jenkins (broken) Extreme
App D 8.0 Webpack (kind of) Kubernetes (sort of) Moderate
App E 5.6 (again) Literal nightmares Prayers Infinite
Onboarding a new developer meant spending a week just explaining the quirks of each app. A week. Before they wrote a single line of production code.

The Decision
I pitched a radical idea: Kill them all.

Well, not kill—migrate. Consolidate all 5 frontends into our standard Node.js pipeline. One build process. One deployment strategy. One way to do things.

The pushback was immediate:

"But PHP is fine!"

"Why fix what isn't broken?"

"This will take months!"

"We'll lose our 'character'!"

I had one response: "Your character is costing us money."

The Migration
Here's what the migration actually looked like:

Step 1: The Audit
I mapped every feature, every route, every API call across all 5 apps. Surprisingly, they all did the same 3 things:

Fetch data from an API

Render a UI

Handle form submissions

The differences were purely in how they did it.

Step 2: The Standard
I defined our "standard" Node pipeline:

Framework: Express + React (already in use elsewhere)

Build: Vite (fast, modern)

Linting: ESLint + Prettier (non-negotiable)

Testing: Jest + React Testing Library

Deployment: GitHub Actions → AWS ECS

Step 3: The Rewrite
I didn't rewrite everything from scratch. I extracted:

Shared UI components into a monorepo package

API client logic into a single service

Environment configuration into a unified .env system

Each app got re-implemented as a "theme" or "route" within the same codebase.

Step 4: The Cutover
We deployed one app at a time. If it broke, we rolled back. (It broke twice. We rolled back twice. No big deal.)

The Results
Onboarding: 1 Week → 1 Day
This is the metric I'm most proud of.

Before:

Day 1: Install PHP 5.6 (good luck on M1 Macs)

Day 2: Configure 5 different .ini files

Day 3: Learn App A's custom routing

Day 4: Learn App B's custom templating

Day 5: Actually write code (maybe)

After:

Morning: git clone, npm install, cp .env.example .env

Afternoon: "Here's the codebase. You know React. You know Node. Go."

New devs are pushing features on Day 1. Not just "Hello World"—actual features.

Developer Satisfaction: 📈
I surveyed the team before and after:

Metric Before After
"I understand the codebase" 2/10 8/10
"I can debug issues quickly" 3/10 7/10
"I enjoy working here" 4/10 9/10
Deployments: Stressful → Boring
Before: Deployments required 3 different people, 2 Slack channels, and a prayer circle.

After: git push main → automated build → automated test → automated deploy. 15 minutes. No human intervention.

The Hard Truth
Standardization is boring.

There's no glory in writing another YAML file. No one gives you a trophy for enforcing the same ESLint rules across 5 apps. No one tweets about "the elegant CI pipeline."

But here's the thing:

Boring is fast.
Boring is predictable.
Boring is scalable.
Boring makes money.

When we hired our last junior developer, they were productive on Day 1. That's not a flex—that's the system working. The system doesn't care about being exciting. The system cares about getting work done.

What I Learned

Standardization Is a Force Multiplier
One week to onboard vs. one day means 4 extra days of productivity per new hire. If we hire 10 people this year, that's 40 days of salary saved. That's not "boring"—that's a business case.
Legacy Code Isn't "Character"—It's Debt
I used to romanticize legacy code. "It has history!" "It's battle-tested!" No. It has bugs. It has config drift. It has hidden dependencies that no one remembers.

Kill it. Consolidate it. Move on.

Developers Want Clarity, Not Chaos Every developer I've ever worked with wants to write code that matters. They don't want to spend 3 hours figuring out why App C won't connect to the database.

By standardizing, I gave my team back hours of their lives. That's not boring—that's humane.

The Meme Version (For X/Threads)
5 legacy PHP apps ➡️ 1 Node pipeline.

Onboarding:
Before: 1 week of "Why does this server require PHP 5.6?"
After: 1 day of "Here is your .env file."

Standardization is boring.

But watching new devs ship features on Day 1? That's beautiful.

Final Thoughts
If you're sitting on a pile of legacy apps that all do the same thing but look completely different—stop romanticizing the chaos.

Consolidate.
Standardize.
Bore yourself to death.

Your future self (and your future new hires) will thank you.

What's your experience with standardizing legacy code? Drop a comment below. Let's be boring together. 👇

From 1 Week to 1 Day: The Unsexy Power of Standardization

Sohana Akbar — Thu, 23 Jul 2026 15:01:48 +0000

"Standardization is boring."

I've heard this phrase more times than I can count. Usually from senior engineers who've been burned by "one-size-fits-all" solutions. Usually right before they propose building Yet Another Custom Framework™.

But here's the thing about boring: it scales. And after migrating 5 legacy PHP frontends to our standard Node pipeline, I've become a born-again convert to the church of "boring technology."

Let me tell you why.

The Mess We Inherited
Our product had grown organically over 7 years. Different teams, different eras, different opinions about what "good" looked like.

We had:

Legacy PHP monolith (custom framework, no tests)

PHP + jQuery (circa 2014, with "creative" use of global variables)

PHP + AngularJS (the 1.x version, because why not?)

PHP + Vue 2 (our "modern" one, but with Webpack configs from hell)

PHP + vanilla JS (a "micro-frontend" before that was a word)

Each had its own:

Deployment process (some FTP, some Jenkins, one was literally a bash script that scp'd files)

Dependency management (Composer? NPM? Just commit the vendor folder?)

Environment variables (.env, .env.local, .env.production, and one that used a JSON file in S3)

Build process (or none at all)

Testing strategy (or none at all)

Onboarding a new developer took 1 week. Minimum.

And that was assuming they already knew PHP. If they were a Node/React hire? Good luck. We'd lose them in the first 3 days, buried in a forest of php.ini configuration and mysterious 500 errors.

The Migration: Boring Choices, Big Impact
I didn't propose rewriting everything in Rust with WebAssembly. I didn't propose a microservices architecture with Kubernetes and service meshes.

I proposed: "What if we just... used our standard Node pipeline?"

The stack:

Node.js + Express (API layer)

React (frontend, with Vite)

TypeScript (because we like our sanity)

Standardized deployment (GitHub Actions → Docker → ECS)

Shared ESLint/Prettier configs

Same testing tools (Jest + React Testing Library)

Same logging/monitoring (Datadog, structured JSON logs)

Nothing innovative. Nothing that would get a conference talk. Nothing that made me feel like a 10x engineer.

But here's what happened:

Migration 1: The Monolith
Took 3 weeks. We had to extract business logic, build a proper API, and reimplement feature flags. Painful, but we now had API contracts.

Migration 2: jQuery Era
Took 2 weeks. Mostly just rebuilding components in React. The API was already there from migration 1.

Migration 3: AngularJS
Took 1 week. Similar patterns, easier because React is... well, React.

Migration 4: Vue 2
Took 4 days. The patterns were close enough that we could map components 1:1.

Migration 5: Vanilla JS
Took 3 days. Minimal state management meant minimal migration effort.

Total: ~6 weeks of focused work.

The Real Win: Onboarding Time
Before migration: 1 week to get a new dev productive.

After migration: 1 day.

Here's what that day looks like now:

Hour 1: Clone repo, run npm install, run npm run dev. Application is running locally. No php.ini tweaks. No Xdebug setup. No "but it works on my machine."

Hour 2: Walk through our standard dev setup: how to run tests (npm test), how to lint (npm run lint), how to build (npm run build). Same commands as our 8 other microservices.

Hour 3: PR is opened for a small bug fix. CI passes (because the same CI runs on every repo). Reviewer comments on style (and they're using the same ESLint rules, so most style issues are already caught).

Hour 4: First deployment. Same GitHub Actions workflow. Same ECS task definition. Same Datadog dashboards.

Day 2: They're picking up tickets independently.

The Unsexy Metrics That Matter
Metric Before After
New dev onboarding time 5-7 days 1 day
Setup errors ~3 per new dev 0
Time to first PR 4 days 3 hours
Deployment failures ~15% ~2%
Context switching 5 different mental models 1 mental model
"Where is X configured?" questions Daily Never
Why This Matters (Beyond the Numbers)
Here's what I learned from this migration:

Innovation is expensive
Every bespoke solution is a tax on future developers. Choose to be boring when it matters.
Boring scales, exciting doesn't
Your team doesn't need to be impressed by your tech stack. They need to ship features and go home at 5 PM.
Onboarding is a signal
If it takes 1 week to get someone productive, that's not a problem with your new hires. That's a problem with your architecture.
Consistency compounds
Once everything uses the same patterns, you stop thinking about the tooling. You start thinking about the product. That's the real win.
The best migration is the one you finish
Did we make perfect choices? No. Did we rewrite everything in the "optimal" way? No. Did we finish? Yes. That matters more than perfection.

What We Lost (and Didn't Miss)
Lost: The freedom to choose the "right tool for the job" on a per-project basis.

Didn't miss: The freedom to choose the "right tool for the job" on a per-project basis.

Lost: Engineers feeling clever because they used a niche framework.

Didn't miss: Engineers feeling clever because they used a niche framework.

Lost: The ability to optimize every frontend independently.

Didn't miss: Debugging 5 different deployment failures on a Friday afternoon.

The One Problem We Still Have
Recruiting. Because when I describe our stack to candidates, it's not exciting.

"React? Node? TypeScript? You're like every other company."

Yes. Exactly.

But you know what else I tell them?

"Your first day, you'll write code that ships to production. Your second day, you'll pick up a ticket. Your third day, you'll be teaching the intern."

That is exciting.

Practical Takeaways
If you're considering a similar migration:

Start with API extraction. Separate data from presentation before you touch the frontend.

Standardize deployment first. If you can deploy anything the same way, migrations get easier.

Accept temporary duplication. During migration, we had both PHP and Node running in production. Traffic was split. It worked.

Don't rewrite business logic. If you can, keep the domain logic intact and just change the presentation layer.

Move fast on migrations. The longer a migration takes, the less likely you are to finish. 6 weeks was intense, but it was finished.

Celebrate the boring wins. When a new dev ships on day 1, call it out. That's the metric that matters.

The Bottom Line
Standardization isn't exciting. It won't get you on the front page of Hacker News. It won't get you a speaking slot at your local tech conference.

But it will:

Get new devs shipping code on day 1

Reduce production incidents

Make debugging easier

Let your team focus on product, not tooling

Ship features faster

And honestly? That's way more interesting than arguing about frameworks.

Have you standardized your stack? What was the biggest win (or pain point)? Drop a comment below—I'd love to hear your war stories.

P.S. If you're on a team where "boring" is considered a bad word, send this article to your tech lead. Sometimes the most impactful engineering work is the stuff that's invisible when it's done right.

From Deployment Chaos to Clarity: Automating Jira Ticket Linking in Our Production Pipeline

Sohana Akbar — Wed, 22 Jul 2026 15:12:00 +0000

How we eliminated "what are we deploying?" from our release process without sacrificing our manual approval gate

The 3 PM Panic
We've all been there. It's 3 PM on a Thursday, and the deployment dashboard shows a green build waiting for production approval. The release manager opens Slack and types the dreaded question:

"What are we deploying?"

What follows is 15 minutes of frantic channel scrolling, digging through commit histories, and playing detective across Jira, GitHub, and Slack. Eventually, someone pieces together that the release contains three bug fixes, two features, and—surprise—a database migration that needs special attention.

This scene played out in our team at least twice a week. We had a robust manual approval gate for production (security requirements meant we couldn't automate that final step), but we had zero visibility into what we were actually approving.

The manual approval wasn't the problem. The context gap was.

The Problem We Were Actually Solving
Let's be clear: we weren't trying to remove the manual approval gate. In our regulated industry, that wasn't an option. But the approval was meaningless when approvers had to become digital archaeologists just to understand what they were signing off on.

Our pain points were specific:

Context switching overhead - Approvers spent 5-10 minutes gathering information per deployment

Incomplete release notes - We relied on developers remembering to update a CHANGELOG

Missing dependencies - Database migrations, environment variable changes, or third-party service updates often got lost in the shuffle

Audit trail gaps - When something went wrong, tracing back to the original requirement was painful

"Surprise" deployments - Team members would discover their changes were in production only when a user reported a bug

The manual approval gate wasn't going anywhere. But the chaos around it? That we could fix.

The Solution: Automated Jira Ticket Linking
The idea was simple: make every deployment self-documenting by automatically pulling in Jira ticket context.

Here's what we built:

Jira Integration in CI/CD We added a step in our CI pipeline (we use GitHub Actions) that:

Parses commit messages for Jira ticket keys (e.g., PROJ-123)

Fetches ticket metadata via Jira REST API

Aggregates all unique tickets into a deployment manifest

yaml

.github/workflows/deploy.yml snippet

name: Fetch Jira Tickets
id: jira
run: |
TICKETS=$(git log ${{ github.sha }} --pretty=format:"%s" | grep -oE '[A-Z]+-[0-9]+' | sort -u)
echo "tickets=$TICKETS" >> $GITHUB_OUTPUT
name: Get Ticket Details
if: steps.jira.outputs.tickets != ''
run: |

Fetch details for each ticket

for ticket in ${{ steps.jira.outputs.tickets }}; do
curl -H "Authorization: Bearer $JIRA_TOKEN" \
"https://jira.company.com/rest/api/2/issue/$ticket" \
| jq '.fields | {summary: .summary, status: .status.name, assignee: .assignee.displayName, priority: .priority.name}'
done
The Deployment Manifest
Instead of just a version number, our deployment pipeline now generates a rich manifest:

json
{
"deployment_id": "deploy-20240115-001",
"version": "v2.3.1",
"timestamp": "2024-01-15T14:30:00Z",
"jira_tickets": [
{
"key": "PROJ-123",
"summary": "Fix payment calculation for EU customers",
"status": "Done",
"assignee": "Alex",
"priority": "High",
"components": ["Payment API", "Backend"],
"fixVersion": "v2.3.1"
},
{
"key": "PROJ-124",
"summary": "Add dark mode toggle to dashboard",
"status": "Done",
"assignee": "Jamie",
"priority": "Medium",
"components": ["UI", "Frontend"],
"fixVersion": "v2.3.1"
}
],
"release_notes": "Bug fixes: PROJ-123. Features: PROJ-124.",
"db_migrations": ["ALTER TABLE payments ADD COLUMN tax_rate"],
"env_changes": ["New env var: PAYMENT_TAX_ENABLED=true"]
}

Automated Release Notes The pipeline generates formatted release notes from the Jira data and posts them to:

A dedicated Slack channel

The GitHub release page

Our internal documentation site

Crucially: the manual approval request itself

Approval with Context When an approver now receives the manual approval request, they see:

text
📦 Deployment v2.3.1 to Production
────────────────────────────────
🕐 Requested: 2024-01-15 14:30 UTC
👤 Requestor: deploy-bot

📋 Tickets in this release:
✅ PROJ-123: Fix payment calculation for EU customers (High)
✅ PROJ-124: Add dark mode toggle to dashboard (Medium)
✅ PROJ-125: Update error logging for API timeouts (Low)

⚠️ Database Changes:

ALTER TABLE payments ADD COLUMN tax_rate (auto-rollback available)

⚙️ Environment Changes:

PAYMENT_TAX_ENABLED=true (new)

📎 View full details: https://deployments.internal/release/v2.3.1

[Approve] [Reject] [View Details]
No more context gathering. No more Slack panic.

What We Built (The Technical Details)
Here's the architecture we settled on:

Pipeline Integration
text
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Code Push │────▶│ CI Build │────▶│ Jira Fetch │
└─────────────┘ └─────────────┘ └─────────────┘
│
▼
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
│ Approval │◀────│ Deploy to │◀────│ Generate │
│ Request │ │ Staging │ │ Manifest │
└─────────────┘ └─────────────┘ └─────────────┘
│
▼
┌─────────────┐
│ Deploy to │
│ Production │
└─────────────┘
Key Components

Git Commit Parser

python
import re
from typing import Set

def extract_jira_tickets(commits: List[str]) -> Set[str]:
"""Extract JIRA ticket keys from commit messages."""
pattern = r'[A-Z]{2,10}-\d+'
tickets = set()
for commit in commits:
tickets.update(re.findall(pattern, commit))
return tickets

Jira API Client

python
class JiraClient:
def get_ticket_details(self, ticket_key: str) -> dict:
response = requests.get(
f"{self.base_url}/rest/api/2/issue/{ticket_key}",
headers={"Authorization": f"Bearer {self.token}"}
)
data = response.json()
return {
"key": ticket_key,
"summary": data["fields"]["summary"],
"status": data["fields"]["status"]["name"],
"assignee": data["fields"]["assignee"]["displayName"] if data["fields"]["assignee"] else None,
"priority": data["fields"]["priority"]["name"],
"components": [c["name"] for c in data["fields"]["components"]],
"fixVersion": data["fields"]["fixVersions"][0]["name"] if data["fields"]["fixVersions"] else None
}

Manifest Generator

python
def generate_deployment_manifest(environment, version, tickets):
manifest = {
"deployment_id": f"deploy-{datetime.now().strftime('%Y%m%d-%H%M')}",
"environment": environment,
"version": version,
"timestamp": datetime.now().isoformat(),
"jira_tickets": [JiraClient().get_ticket_details(t) for t in tickets],
"release_notes": generate_release_notes(tickets),
"db_migrations": detect_migrations(), # Custom detection logic
"env_changes": detect_env_changes() # Custom detection logic
}
return manifest

Slack Notification

python
def send_approval_request(manifest):
blocks = [
{
"type": "header",
"text": {"type": "plain_text", "text": f"📦 Deployment {manifest['version']} to Production"}
},
{
"type": "section",
"fields": [
{"type": "mrkdwn", "text": f"Tickets: {len(manifest['jira_tickets'])}"},
{"type": "mrkdwn", "text": f"DB Changes: {len(manifest['db_migrations'])}"},
]
}
]

# Add ticket details
for ticket in manifest['jira_tickets'][:5]:  # Show top 5
    blocks.append({
        "type": "section",
        "text": {"type": "mrkdwn", "text": f"• *{ticket['key']}*: {ticket['summary']} ({ticket['priority']})"}
    })

if len(manifest['jira_tickets']) > 5:
    blocks.append({
        "type": "section",
        "text": {"type": "mrkdwn", "text": f"... and {len(manifest['jira_tickets']) - 5} more"}
    })

# Approval buttons
blocks.append({
    "type": "actions",
    "elements": [
        {"type": "button", "text": {"type": "plain_text", "text": "✅ Approve"}, 
         "value": f"approve_{manifest['deployment_id']}", "style": "primary"},
        {"type": "button", "text": {"type": "plain_text", "text": "❌ Reject"}, 
         "value": f"reject_{manifest['deployment_id']}", "style": "danger"},
    ]
})

return blocks

The Results (4 Months Later)
What Improved
Approval time decreased from ~8 minutes to under 1 minute - Approvers no longer needed to hunt for context. Everything they needed was in the request.

"What are we deploying?" questions dropped by 95% - The first week we deployed this, I kept waiting for the Slack message. It never came.

Audit compliance became effortless - Every deployment now has a complete audit trail linking commits → tickets → approvals → production.

On-call incidents became easier to triage - When something broke, we could instantly see which tickets were in the release and roll back specific changes.

Developers started writing better commit messages - Since commit messages are parsed, devs began naturally including ticket keys even more consistently.

What We Learned
Ticket keys in commits are non-negotiable - We added a pre-commit hook to prevent commits without ticket keys in main branches.

Not every deployment needs Jira tickets - Hotfixes and infrastructure changes don't always map to tickets. We added the ability to skip ticket linking with [NO-TICKET] in the commit message.

Database migrations need special handling - We added explicit detection for migration files and flagged them prominently in the approval request.

The manifest is now a single source of truth - Teams started referencing the deployment manifest for everything from rollback decisions to release notes generation.

The Code (If You Want to Try This)
We open-sourced the core components. Here's the simplified version you can adapt:

GitHub Action for Jira Linking
yaml
name: Generate Deployment Context

on:
push:
branches: [main, staging]

jobs:
generate-context:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
with:
fetch-depth: 0 # Get all commit history

  - name: Extract tickets from commits
    id: tickets
    run: |
      COMMITS=$(git log ${{ github.sha }} --pretty=format:"%s")
      TICKETS=$(echo "$COMMITS" | grep -oE '[A-Z]+-[0-9]+' | sort -u | tr '\n' ',')
      echo "tickets=${TICKETS%,}" >> $GITHUB_OUTPUT

  - name: Fetch Jira Details
    if: steps.tickets.outputs.tickets != ''
    uses: actions/github-script@v6
    with:
      script: |
        const tickets = '${{ steps.tickets.outputs.tickets }}'.split(',');
        // Call Jira API for each ticket
        // Store results in deployment-manifest.json

  - name: Create Deployment Manifest
    run: |
      echo "Deployment manifest created with Jira context"

  - name: Upload Manifest
    uses: actions/upload-artifact@v3
    with:
      name: deployment-manifest
      path: deployment-manifest.json

Simple Jira Fetch Script
bash

!/bin/bash

fetch-jira.sh - Fetch ticket details from Jira

TICKETS=$(git log origin/main..HEAD --pretty=format:"%s" | grep -oE '[A-Z]+-[0-9]+' | sort -u)

if [ -z "$TICKETS" ]; then
echo "No Jira tickets found in commits"
exit 0
fi

echo "Found tickets: $TICKETS"

for TICKET in $TICKETS; do
echo "Fetching $TICKET..."
curl -s -H "Authorization: Bearer $JIRA_TOKEN" \
"https://jira.company.com/rest/api/2/issue/$TICKET" \
| jq '.key + ": " + .fields.summary'
done
The Bigger Picture
This project taught us something important: automation isn't just about speed—it's about clarity.

We didn't remove the manual approval gate because it serves a legitimate purpose. But we made it meaningful by providing approvers with the context they need to make informed decisions.

The result isn't just faster deployments. It's better deployments. When people understand what they're approving, they catch issues earlier. They spot dependencies. They ask better questions.

And when 3 PM rolls around on a Thursday, there's no Slack panic. Just a clear, informed approval request that says: "Here's exactly what we're deploying, and here's why."

Want to Build Your Own?
Start small:

Parse commit messages for ticket keys

Display that list in your CI/CD dashboard

Add ticket summaries to Slack approval messages

Build a full manifest with ticket details

Add automated release notes

Protip: Don't over-engineer it. A simple comma-separated list of tickets in your approval request is already a huge improvement over nothing.

Have you solved a similar problem? How did you handle the balance between automation and human oversight? Drop your thoughts in the comments!

When Your Code Quality Gate Makes Developers Cry (But Your Bugs Disappear)

Sohana Akbar — Mon, 20 Jul 2026 18:00:38 +0000

The 20% Merge Drop That Saved Our Production

Let me paint you a picture: It's 4:47 PM on a Friday. Sarah from engineering just pushed her 47th commit to a PR that's been open for two weeks. The build passes. Tests are green. She hits "Merge" with the confidence of someone who's about to start their weekend early.

Then, a red X appears.

SonarQube is screaming about technical debt, code smells, and—God forbid—a single duplicated line of code.

Sarah's Slack status changes to "brb crying." The merge is blocked. The weekend is ruined. The team lead is getting pings from product about "why is nothing shipping?"

Sound familiar?

That was us, three months ago. And it was the best decision we ever made.

The Numbers That Made Us Sweat
Let's get the ugly truth out first:

PR merges dropped by 20% in the first month

Developer satisfaction scores dipped to an all-time low

Team velocity "felt" slower (I'll come back to this in a moment)

At least one developer threatened to quit (they didn't, but the rant was legendary)

But here's what happened next:

Production bugs dropped by 60%

Hotfixes went from weekly to monthly

On-call rotations went from "I hate my life" to "I actually slept last night"

Technical debt accumulation slowed by 40%

Why Developers Hate Quality Gates (And Why They're Wrong)
The pushback we heard was predictable:

"SonarQube doesn't understand our business logic"

"This is just bureaucratic overhead"

"I'll fix it later, just let me merge"

"We're shipping features, not perfect code"

And honestly? They were right about the frustration, but wrong about the solution.

What we realized is that developers weren't mad about quality gates themselves. They were mad about:

Discovering issues too late in the PR process

Getting penalized for problems that existed before they touched the code

Not understanding the rules until the merge was blocked

Feeling like robots instead of craftspeople

How We Made It Work (Without Burning the Team Down)
We learned that quality gates aren't about enforcement—they're about education and culture.

We Gave Developers "The Fix Window" Instead of blocking merges immediately, we allowed a 4-hour grace period for developers to fix quality issues before the gate locked. This removed the panic and gave them ownership.

Result: 80% of issues were fixed within 2 hours, without anyone feeling ambushed.

We Introduced "Quality Debt Sprints" Not every SonarQube issue is critical. We categorized:

Blockers (security, crashes): Merge-blocking

Critical (performance, major bugs): Auto-fail after 24 hours

Minor (code smells, duplication): Tracked, but not blocking

This meant developers could ship features while still paying down tech debt in dedicated sprints.

Result: Technical debt reduction became a team sport, not a punishment.

We Integrated SonarQube Earlier We moved quality checks to pre-commit hooks and local IDE plugins. Developers caught issues before they even opened a PR. The merge gate became the final checkpoint, not the first surprise.

Result: Merge time dropped from 6 hours to 2 hours because PRs weren't going back-and-forth over quality issues.

We Celebrated Quality Wins We started tracking "Clean PRs" (zero SonarQube issues) and "Quality Champions" in our standups. We turned code quality into a badge of honor, not a bureaucratic chore.

Result: Developers started competing to have the cleanest code. Yes, really.

We Adjusted the Rules Over Time Our initial quality gate was too strict. After two weeks, we relaxed thresholds on code duplication and cognitive complexity. We tuned the gate to our team's reality, not SonarQube's defaults.

Result: Compliance went from 40% to 95% because the rules made sense.

The Hidden Benefit Nobody Talks About
Beyond the numbers, the real win was confidence.

When we merged code, we knew it was solid. We stopped second-guessing. We stopped fearing deployments. Product managers started trusting engineering timelines because "production issues" no longer meant "emergency meeting at 2 AM."

The 20% drop in merges wasn't a slowdown—it was a shift from quantity to quality. We shipped fewer PRs, but each one was more valuable, more stable, and less likely to cause a fire.

Velocity isn't about how fast you move. It's about how fast you move forward without breaking things.

What We'd Do Differently Next Time
Looking back, I'd make three changes:

Communicate sooner. We rolled out the quality gate without enough developer buy-in. I'd spend two weeks building consensus first.

Start with a "soft" gate. Instead of full enforcement, we should have run SonarQube in "monitor-only" mode for a month, then gradually turned on enforcement.

Pair the gate with automated fixing. SonarQube can auto-fix many issues. We should have enabled that from day one.

The Verdict: Worth Every Tantrum
Yes, developers complained. Yes, merges dropped. Yes, the first month was painful.

But today:

Our production is stable

Our team is confident

Our customers are happier

Our on-call rotation is peaceful

SonarQube quality gates aren't about being the "code police." They're about being professional. They're about saying, "We care enough about our craft to ship code we're proud of."

Would we do it again? Absolutely.

Would I recommend it to other teams? Yes—but with the human element in mind.

Your Turn: 5 Steps to Implementing Quality Gates Without Losing Your Team
Run SonarQube in monitoring mode for 2 weeks. Collect data. Share it transparently.

Involve developers in setting thresholds. Make it a team decision, not a mandate.

Start with a soft gate (block only security vulnerabilities and bugs). Expand gradually.

Automate fixes where possible (formatting, simple refactors, dependency updates).

Celebrate quality improvements in team meetings. Make it a positive, not a punitive, force.

Final thought: The goal isn't perfect code. The goal is intentional code—code that's been reviewed, refined, and respected. SonarQube just helps us get there faster.

The tantrums will fade. The quality will remain.

Have you implemented quality gates? Did your team revolt or rally? I'd love to hear your war stories in the comments below.

🔔 Follow me for more honest engineering stories, lessons from the trenches, and practical advice that actually works

How We Banished "Latest" and Made Rollbacks Instant

Sohana Akbar — Sun, 19 Jul 2026 09:33:15 +0000

Stop using :latest in production. Here’s why—and how we did it.

The Problem We All Know
It starts innocently enough:

bash
docker build -t myapp:latest .
docker push myapp:latest
You deploy. It works. You move on.

Then Monday morning hits. Your monitoring dashboard turns red. Users are complaining. Something broke in the last deploy—but what exactly changed? The :latest tag moved yesterday, then again this morning, and maybe once more during that hotfix at 2 AM.

Good luck figuring out which version is actually running.

Our Breaking Point
We had a "stable" production environment. We had CI pipelines. We had rollback scripts. And yet, every incident turned into a forensic investigation:

"Which commit is this container running?"

"Did that hotfix make it in?"

"Is staging on the same version as production?"

"Can I even roll back to yesterday's image?"

The answer was almost always: ¯_(ツ)_/¯

Worst of all—rollbacks weren't truly instant. We'd revert code, rebuild, repush, and redeploy. That's 8-12 minutes of downtime during an active incident.

The Decision: Ban :latest in Production
We made a drastic but simple rule:

No :latest tag is ever pushed to our production ECR repository.

Not "discouraged." Not "we'll phase it out."

Banned. Blocked. Period.

What We Did Instead
Every image gets tagged with its commit hash (shortened to 7 characters):

bash

In our CI pipeline

COMMIT_HASH=$(git rev-parse --short HEAD)
docker build -t myapp:$COMMIT_HASH .
docker push myapp:$COMMIT_HASH
That's it. No :latest. No environment-specific tags. Just the immutable commit hash.

The Architecture
Here's what our ECR repo looks like now:

text
123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp
├── abc1234 # commit hash
├── def5678
├── ghi9012
├── jkl3456
└── mno7890
Every hash is immutable. Once pushed, it never changes. No overwrites. No ambiguity.

Deployments Become Explicit
Our deployment manifest (Kubernetes, ECS, etc.) now references the exact commit hash:

yaml

deployment.yaml

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp:abc1234
No more abstract "latest" that means different things at different times. The deployment manifest tells you exactly what's running.

The Magic: Instant Rollbacks
This is where everything changed.

Before:

text

Detect incident (2 min)
Identify bad commit (5-15 min)
Revert code in Git (3 min)
CI builds new image (5-8 min)
Push to ECR (2 min)
Deploy (2 min) Total: 19-32 minutes After:

text

Detect incident (2 min)
Find previous known-good commit hash
Update manifest and redeploy (2 min) Total: 4 minutes Because every commit hash is an immutable, ready-to-run image in ECR, rollback is just a manifest change. No rebuild. No repush. No waiting.

We went from 20+ minute rollbacks to under 5 minutes.

How We Enforced It
We implemented three layers of protection:

CI Pipeline Enforcement bash # Block :latest pushes to production ECR if [[ "$ECR_REPO" == "production" && "$IMAGE_TAG" == "latest" ]]; then echo "❌ Cannot push :latest to production ECR" exit 1 fi
ECR Lifecycle Policy
json
{
"rules": [
{
"rulePriority": 1,
"description": "Expire all untagged images",
"selection": {
"tagStatus": "untagged",
"countType": "imageCountMoreThan",
"countNumber": 100
},
"action": {
"type": "expire"
}
}
]
}
We also expire old images after 90 days to keep costs manageable.
Terraform/CloudFormation Guardrails
Our infrastructure-as-code templates block any deployment that doesn't specify an explicit, non-latest tag:

hcl

Terraform validation

variable "image_tag" {
type = string
validation {
condition = var.image_tag != "latest"
error_message = "Cannot use 'latest' tag in production."
}
}
The Unexpected Benefits
Once we banned :latest, we discovered perks we hadn't anticipated:

Instant Audit Trail
Each deployment is tied to a Git commit SHA. You can trace every container back to its source code, CI build logs, and the exact moment it was created.
Zero Ambiguity
"Which version is running?" Look at the tag. No guessing. No "well, it's latest from yesterday."
Simplified Debugging
When developers SSH into a container, docker inspect shows the commit hash. They know exactly which code they're debugging.
Safe Canary Deployments
We can deploy abc1234 to 10% of traffic and def5678 to 90%, comparing metrics without any tagging confusion.
Cleaner Rollforward
If the rollback fixes the issue, we can later redeploy the same hash or a new one—all without tagging collisions.

What About Staging and Dev?
We kept :latest for non-production environments:

Dev: :latest is fine for rapid iteration.

Staging: We use commit hashes too, but allow :latest as an alias for convenience.

The production repo is the only one with the strict ban.

The One Thing to Watch Out For
Image bloat.

If you push every commit to ECR and never clean up, your storage costs will creep up. Our solution:

Lifecycle rule: Expire images older than 90 days.

Selective retention: Keep the last 50 images regardless of age (for emergency rollbacks).

Manual archives: For major releases, we pin specific hashes as "golden images" with infinite retention.

The Results (By The Numbers)
After 6 months with this system:

Metric Before After
Avg rollback time 22 min 4 min
Rollback success rate 72% 98%
Incident resolution time 45 min 18 min
Developer confusion High Zero
"What's deployed?" questions Daily Never
How to Implement This Tomorrow
Stop pushing :latest to prod today. Seriously. Just stop.

Use git rev-parse --short HEAD as your primary tag.

Update your CD pipeline to reference explicit hashes in manifests.

Store the current production hash somewhere (we use a simple S3 file with the hash and timestamp).

Test a rollback right now—manually update your manifest to a known-good hash and redeploy. Time it.

Celebrate when it takes < 5 minutes.

The Bottom Line
docker push myapp:latest is a developer convenience, not a production strategy.

Banning :latest from our ECR repo was one of the simplest, highest-impact changes we made to our deployment pipeline. It didn't require rewriting architecture or buying new tools. It just required a rule—and the discipline to enforce it.

Rollbacks went from a stressful, multi-step ordeal to a boring, scripted operation.

And boring operations are the best kind.

Your turn: Are you still using :latest in production? What's stopping you from switching?

Follow me for more production engineering lessons learned the hard way.

How We Caught 12 Breaking API Changes Before They Hit Main: Our Journey to Ephemeral Staging Environments

Sohana Akbar — Sat, 18 Jul 2026 09:20:49 +0000

The moment we realized our staging environment was broken

It was 3 PM on a Thursday, and our team was scrambling. A critical API change had just been merged to main, but the staging environment—our supposed safety net—was showing false positives. The integration tests passed, but the mobile app was completely broken in production.

That's when we knew: our shared staging environment was failing us.

The Problem: Shared Staging Is Broken by Design
Like many engineering teams, we operated with a single, shared staging environment. Every developer deployed their changes to the same place, leading to:

Deployment conflicts: "Who deployed that breaking change?"

Cascading failures: One broken PR would block the entire team

Test contamination: Data from one test would leak into another

Delayed feedback: You'd only discover issues after merging your PR and deploying to staging

The "works on my machine" syndrome, now at scale

The worst part? Our API contracts were changing constantly, but we only discovered breaking changes during integration testing—often too late.

The Solution: Ephemeral Environments per PR
We made a radical change: every PR gets its own isolated, short-lived environment.

Here's our architecture:

Our Implementation Stack
Infrastructure: Kubernetes (EKS) with namespace-per-PR

Orchestration: Custom GitHub Action workflow

Database: Isolated RDS instance per environment

Contract Testing: Pact flow + OpenAPI validation

Cleanup: AWS Lambda that runs every hour, destroying environments older than 2 hours

The Game Changer: Automated Contract Testing
The magic wasn't just in isolated environments—it was in what we did with them. Every time a PR deployed to its ephemeral environment, we ran:

Consumer-Driven Contract Testing (Pact)
Our mobile and web clients would verify their expectations against the actual deployed API. If a change broke what the client expected, the PR would fail.
Provider Contract Validation
We'd automatically verify that the deployed API matched our OpenAPI specification. If you added a required field without updating the spec, you'd know immediately.
Schema Diff Detection
We compared the new API schema against the production baseline. Any breaking changes (removing fields, changing types, adding required properties) would trigger a review.
Integration Smoke Tests
Each environment ran a suite of end-to-end tests with real client applications connecting to the backend.

The Results: 12 Breaking Changes Caught
In our first month with ephemeral environments, we caught 12 breaking API contract changes that would have:

Crashed our mobile app

Broken third-party integrations

Caused data corruption in production

Required emergency rollbacks

One Real Example
A developer modified the User object to rename user_id to userId for consistency. The OpenAPI spec was updated, but the mobile app's contract test immediately caught the mismatch:

text
❌ Contract violation: Expected 'user_id' but found 'userId'
Breaking change detected in GET /api/v2/users/123
The PR was blocked, the issue was fixed in 30 minutes, and the mobile app never broke.

The Economics: Cost vs. Value
You might think: "Ephemeral environments sound expensive." Let's do the math:

Before:

1 shared staging environment running 24/7: $500/month

1 production outage per month: $10,000+ in lost revenue

Developer hours wasted debugging environment issues: 40+ hours/month

After:

Ephemeral environments run ~8 hours/day (active PRs): ~$300/month

0 production-breaking API changes in 3 months

Developer productivity increased by ~30%

Net result: We're saving money AND shipping faster.

Lessons Learned

Start Small, Scale Smart
We didn't spin up full environments for every PR immediately. We started with just the critical backend services.
Make Cleanup Aggressive
2-hour TTL seemed short initially, but developers learned to work faster. For long-running PRs, they could request extensions.
Invest in Observability
Each environment had its own logging and metrics. We could debug failures without affecting others.
Education is Key
Developers needed to understand why userId vs user_id matters. We created "Breaking Change 101" docs.
Contract Tests Are Only Half the Battle
We also needed:

Load testing (some issues only appear under load)

Security scanning

Database migration testing

The Future: Beyond API Contracts
Now that we have the infrastructure, we're expanding:

Database migrations: Test schema changes on a copy of production data

Feature flags: Test features in isolation before global rollout

Performance benchmarks: Detect performance regressions per PR

Security scanning: Run vulnerability scans on each environment

Should You Do This?
Yes, if:

You have breaking changes that go to production

Your staging environment is a bottleneck

You have multiple services with dependencies

You want to ship faster with more confidence

Not yet, if:

You're a pre-revenue startup with no customers

Your infrastructure is manual and fragile

You're still working toward basic CI/CD

Getting Started: A Practical Roadmap
Week 1-2: Automation Foundation

Set up infrastructure-as-code (Terraform/CDK)

Create scripts to spin up/down environments

Define environment variables and secrets management

Week 3-4: Core Services

Start with your most critical microservice

Deploy it to an ephemeral environment per PR

Run basic smoke tests

Week 5-6: Contract Testing

Implement Pact or OpenAPI validation

Create test suites that verify contracts

Set up CI/CD integration

Week 7-8: Scale and Optimize

Add more services to the stack

Fine-tune resource allocation

Implement cost monitoring

Week 9+: Iterate

Gather developer feedback

Add database support

Enhance observability

The Bottom Line
Ephemeral staging environments transformed how we build software. We've gone from "breaking changes are inevitable" to "breaking changes are caught in PRs."

The 12 breaking changes we caught weren't just bugs—they were 12 production incidents that never happened.

That's not just a win for engineering. It's a win for every user who depends on our software.

Have you implemented ephemeral environments? What challenges did you face? Let's discuss in the comments!

Resources to Get Started
Kubernetes Namespaces: Kubernetes.io docs

Pact Contract Testing: Pact.io

GitHub Actions for Preview Deployments: GitHub Docs

AWS EKS Ephemeral Environments: AWS Quick Start

About the Author: I'm a lead engineer at a growing SaaS company, passionate about developer productivity and reliable systems. You can find me on GitHub and Twitter.

Have questions about implementing this in your org? Reach out—I'm happy to share our battle scars!

From 15 Daily Deployments to Zero Downtime: Our Next.js ECS Fargate Journey

Sohana Akbar — Thu, 16 Jul 2026 09:57:53 +0000

Zero-downtime isn't a buzzword; it's an Application Load Balancer + graceful shutdown config I tweaked for 2 weeks.

The Reality of 15 Deployments a Day
When your team deploys 15 times a day, every second of downtime multiplies fast. A 5-second blip becomes 75 seconds of cumulative errors. A 30-second rolling restart becomes 7.5 minutes of degraded service. Every. Single. Day.

We run Next.js on ECS Fargate, and achieving true zero-downtime took me two weeks of late nights, CloudWatch deep-dives, and a love-hate relationship with AWS load balancer health checks.

Here's exactly what worked—and what didn't.

The Architecture Snapshot
text
Internet → ALB (port 443) → Target Group → ECS Fargate (Next.js)
↓
Container Health Checks
↓
Graceful Shutdown (SIGTERM)
Stack:

Next.js 14 (App Router, standalone output)

AWS ECS Fargate (capacity provider: FARGATE_SPOT)

Application Load Balancer (not Classic, not NLB)

Docker images ~450MB (yes, we're working on it)

Phase 1: What "Zero-Downtime" Actually Means Here
Let's kill the buzzword. For us, zero-downtime means:

No 5xx errors during deployment (ALB 504s? Gone.)

No dropped connections for in-flight requests

Sub-100ms p99 latency increase during rolling updates

Complete rollback capability in under 90 seconds

Phase 2: The ALB Setup That Finally Worked
Target Group Health Check Settings
This is where I spent most of my two weeks. The defaults are wrong for Next.js.

yaml
HealthCheckProtocol: HTTP
HealthCheckPath: /api/health # NOT / (more on this)
HealthyThresholdCount: 2
UnhealthyThresholdCount: 3
HealthCheckTimeoutSeconds: 5
HealthCheckIntervalSeconds: 15
Why /api/health over /?

Next.js / does SSR, which can be slow and fetch from APIs/databases

/api/health is a lightweight, serverless function that returns { status: 'ok' }

Your health check should test the runtime, not the app logic

Deregistration Delay (Draining)
This is the silent killer of zero-downtime:

text
DeregistrationDelay: 120 seconds
Why 120 seconds? Our average request duration is ~200ms, p99 is ~4.5 seconds. The default 300 seconds is too long (keeps unhealthy containers alive), but 60 seconds is too short for long-running Next.js server actions. 120 was our Goldilocks number.

Phase 3: The Container Health Check (The Real MVP)
Your Docker health check isn't optional. It's the difference between ECS leaving a broken container running for 3 minutes and killing it in 15 seconds.

dockerfile
HEALTHCHECK --interval=10s --timeout=3s --start-period=30s --retries=3 \
CMD curl -f http://localhost:3000/api/health || exit 1
Pro tip: The start-period is critical. Next.js takes ~18 seconds to boot on Fargate (node_modules, compilation). Without this, ECS will kill your container before it even starts serving traffic.

Phase 4: Next.js Graceful Shutdown (The Hard Part)
This was the hardest piece to nail. Next.js doesn't handle SIGTERM gracefully by default—it just dies.

The Solution: Custom Server with Signal Handling
We migrated from next start to a custom server:

javascript
// server.js
const { createServer } = require('http');
const next = require('next');

const app = next({ dev: false });
const handle = app.getRequestHandler();

let server;

app.prepare().then(() => {
server = createServer((req, res) => handle(req, res));

server.listen(3000, () => {
console.log('Next.js ready on 3000');
});

// --- GRACEFUL SHUTDOWN ---
const shutdown = () => {
console.log('Received SIGTERM, starting graceful shutdown...');

// Stop accepting new connections
server.close(() => {
  console.log('Closed all connections, exiting.');
  process.exit(0);
});

// Force exit after timeout
setTimeout(() => {
  console.error('Force exiting after timeout.');
  process.exit(1);
}, 30000); // ALB draining timeout - 10s buffer

};

process.on('SIGTERM', shutdown);
process.on('SIGINT', shutdown);
});
Dockerfile Entrypoint
dockerfile
ENTRYPOINT ["node", "server.js"]
NOT npm start or next start. We need process-level control.

Phase 5: ECS Task Definition (The Final Config)
json
{
"family": "nextjs-app",
"networkMode": "awsvpc",
"containerDefinitions": [{
"name": "nextjs",
"image": "...",
"essential": true,
"portMappings": [{
"containerPort": 3000,
"hostPort": 3000,
"protocol": "tcp"
}],
"healthCheck": {
"command": ["CMD-SHELL", "curl -f http://localhost:3000/api/health || exit 1"],
"interval": 10,
"timeout": 3,
"retries": 3,
"startPeriod": 30
},
"linuxParameters": {
"capabilities": {
"add": ["NET_ADMIN"] // For container-level networking
}
}
}],
"requiresCompatibilities": ["FARGATE"],
"executionRoleArn": "...",
"taskRoleArn": "..."
}
Phase 6: The Deployment Flow That Actually Works
text

Build Next.js (standalone output)
Docker build (multi-stage, ~450MB → 180MB with standalone)
Push to ECR
ECS update-service (force-new-deployment)
ALB gracefully drains old tasks (120s deregistration)
ECS starts new tasks with health check (30s start-period)
ALB begins routing after 2 successful health checks (20s)
Old tasks drain and terminate Total impact: ~3 seconds of elevated latency during the handover. No errors. No dropped connections.

Phase 7: The Monitoring That Validates It
We monitor every deployment with these CloudWatch alarms:

yaml

TargetGroup.ResponseTime.p99 > 500ms (degraded performance)
TargetGroup.UnHealthyHostCount > 0 (stuck)
ECS.Service.utilization.memory > 85% (container strain)
LoadBalancer.HTTPCode_ELB_5XX > 0 (the big red flag) Deployment dashboards: We overlay deployment timestamps on latency graphs. No spikes = success.

Phase 8: What Almost Broke Us
The Node.js Event Loop Issue: Next.js 14's server components can block the event loop during boot. Our health check timed out because the server was busy compiling. Solved by:

javascript
// next.config.js
module.exports = {
experimental: {
optimizeServerReact: true,
serverMinification: true
},
compiler: {
removeConsole: process.env.NODE_ENV === 'production'
}
}
The ALB Sticky Sessions Problem: We use session affinity. During deployments, the ALB kept routing users to dying containers. Fixed by setting:

text
StickinessEnabled: true
StickinessType: lb_cookie
CookieExpirationPeriod: 300 # 5 minutes
This gave old containers enough time to drain existing sessions while new ones handled fresh connections.

The Numbers After 2 Weeks
Metric Before After
Deployment errors (5xx) 3-8 per deploy 0
Deployment latency spike 2-8 seconds <100ms
P99 response time during rollout ~3 seconds ~250ms
Failed deployments ~10% <1%
Rollback time ~3 minutes ~90 seconds
The "I Told You So" Moment
Our VP of Engineering watched a deployment during peak traffic. No alerts. No errors. Just a Slack message: "Deploy #12,873 completed."

He looked at me and said, "So it just... works now?"

That was worth the two weeks.

Key Takeaways for Your Setup
Health checks matter more than you think. Use a lightweight endpoint, not the homepage.

Deregistration delay is a balancing act. Too short = dropped requests. Too long = zombie containers.

Custom server with SIGTERM handling is non-negotiable. Next.js's default isn't production-grade for high-frequency deploys.

Start-period in Docker health checks saves you. Next.js needs warm-up time.

Monitor everything. You can't fix what you can't measure.

What's Next for Us
CDN + Next.js ISR: Offload some traffic to reduce container load

GitOps with ArgoCD: Automated rollbacks based on error budgets

Canary deployments: 10% traffic to new version before full rollout

Warm containers: Pre-warm a container before ECS kills the old one

Final thought: Zero-downtime isn't a checkbox. It's a continuous, iterative process that requires understanding your application at the container, network, and framework level. Two weeks of tweaking ALB settings and health checks turned our deployment anxiety into deployment confidence.

And now, 15 times a day, we deploy without a single user noticing.

That's the win.

Have you wrestled with ECS + Next.js deployments? Drop your war stories in the comments—I want to hear what broke for you.

The 2-Week Journey to Next.js Zero-Downtime on ECS Fargate

Sohana Akbar — Wed, 15 Jul 2026 14:15:38 +0000

Zero-downtime isn't just a buzzword we throw around in standups. It's the result of an Application Load Balancer and a graceful shutdown config that took me two weeks to dial in perfectly. And honestly? It was worth every minute.

We deploy to ECS Fargate 15 times a day. Fifteen. That's not a flex—it's a necessity for our team's velocity. And with that many deployments, even a few seconds of downtime per deploy adds up fast. Here's how we cracked the code.

Why We Chose ECS Fargate
Let me save you the comparison analysis paralysis: we evaluated App Runner, Lambda with Web Adapter, and even EKS. App Runner looked tempting—dead simple setup—but it doesn't support our preferred region. Lambda? Cold starts and RDS connection management made it a non-starter for our Next.js SSR workloads.

ECS Fargate hit the sweet spot: container-based (so our Docker workflow stayed intact), no EC2 management, and we're only paying for what the containers actually use. Plus, it plays beautifully with RDS, which we were already using for production data.

The Architecture That Finally Worked
Here's what our stack looks like after two weeks of tweaking:

Route 53 handling DNS with ALIAS records pointing to our ALB

Application Load Balancer with TLS 1.3 (because 2026, people) and HSTS headers

ECS Fargate running our Next.js tasks in private subnets

Secrets Manager + SSM Parameter Store for all environment variables (more on this later)

CloudWatch Logs for centralized logging—non-negotiable for debugging

The critical piece? Standalone output mode in Next.js. The docs aren't great, but once you figure out what to copy over, it's magic:

text
COPY --from=build /app/.next/standalone ./
COPY --from=build /app/public ./public
COPY --from=build /app/.next/static ./.next/static
Without this, your Docker images balloon in size and your startup times tank. With it? Lean, mean containers that spin up in seconds.

The 2-Week Journey to Zero-Downtime
Week 1: The Pain

Our first attempt at zero-downtime was... optimistic. We thought the ALB's built-in health checks would handle everything. They didn't.

We learned the hard way that ECS sends a SIGTERM signal when it scales down tasks. If your app doesn't handle it properly, it dies mid-request. Users see 503 errors. You see angry Slack messages.

We started seeing unhealthy targets in the ALB—all of them timing out on health checks. The classic 503 Service Unavailable dance. After debugging networking layers, security groups, and container health check paths, we realized the problem was deeper.

Week 2: The Fix

The breakthrough came when we stopped thinking about the ALB as the solution and started thinking about it as part of the solution.

Here's what finally worked:

Proper Graceful Shutdown Configuration Your app needs to finish serving the current request before shutting down. No shortcuts. We configured:

javascript
// In your server setup
process.on('SIGTERM', () => {
server.close(() => {
process.exit(0);
});
});

ECS stopTimeout
Set this in your task definition. Give your app enough time to gracefully handle in-flight requests before ECS force-kills it. We settled on 30 seconds—enough for most SSR operations to complete.
ALB Health Check Tuning
The default health check configuration won't cut it. We had to:

Set a reasonable healthCheckGracePeriodSeconds in the ECS service (60-120 seconds to let tasks warm up)

Tune the ALB target group's health check path to something lightweight but representative of a healthy app

Ensure the health check path returns a 200 within 5 seconds—no heavy rendering for health checks

Environment Variable Strategy This one took time to get right. We eventually adopted a 5-category classification system:

Category Storage Injected At Example
Non-sensitive build-time Task def environment Container start NODE_ENV, PORT
Non-sensitive runtime SSM Standard Container start API_URLs
Semi-sensitive SSM SecureString Container start Client IDs
Highly sensitive Secrets Manager Container start DATABASE_URL, API keys
Client-bundled SSM → Docker build-arg Build time NEXT_PUBLIC_*
This separation means secrets are never in the task definition JSON. They're referenced by ARN only, which makes the ECS console not a security liability.

The Deploy Pipeline That Runs 15x Daily
Our GitHub Actions workflow uses OIDC (no long-lived AWS keys) with environment-based trust policies:

yaml
name: Deploy to ECS
on:
push:
branches: [main, develop]

jobs:
deploy:
environment: ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
runs-on: ubuntu-latest
steps:
- Build Docker image with platform linux/amd64
- Push to ECR with tag: ${{ env }}-${{ sha }}
- Render and register new task definition
- Update ECS service with forced new deployment
- Wait for stability (with 10-minute timeout)
The wait-for-service-stability step is crucial. Without it, we'd deploy, assume everything was fine, and only discover issues five minutes later when the health checks failed.

What I'd Tell My Past Self
If I could go back two weeks and give myself one piece of advice? Start with health checks, then add graceful shutdowns, then tune them together.

They're not separate concerns. The ALB needs to know when a task is healthy enough to receive traffic. The task needs to know how to become healthy and how to become not healthy gracefully. They're a system, not independent configs.

Also: use --platform linux/amd64 in your Docker builds if you're on an ARM Mac. I spent a day wondering why local builds worked but Fargate tasks crashed. Different architectures. Oops.

The Results
Fifteen deployments a day. Zero detected downtime for users.

Is it over-engineered? Maybe. But when you're shipping multiple times a day and your product is critical to your business, "over-engineered" starts to look a lot like "properly engineered."

The two weeks of configuration pain paid off in user trust. No more "we're deploying, give us 30 seconds" messages. No more 503 errors during peak traffic. Just seamless updates, every single time.

And honestly? That's what zero-downtime should mean. Not a bullet point on a marketing page, but a real, measurable property of your deployment pipeline.

Now if you'll excuse me, I have my 14th deployment of the day to trigger.

How I Tamed Flaky E2E Tests and Boosted Build Stability from 89% to 99%

Sohana Akbar — Tue, 14 Jul 2026 13:45:40 +0000

The story of how a simple retry policy saved our team's sanity and our main branch

The Breaking Point
It was the third time in a month. Our main branch had gone red again. The culprit? A flaky E2E test that decided to fail for no good reason.

Our team was frustrated. Developers were wasting hours re-running pipelines, investigating false positives, and context-switching away from feature development. Our build stability had plummeted to a dismal 89%, and trust in our CI/CD process was at an all-time low.

Something had to change.

The Diagnosis: Flaky Tests Are Not All Created Equal
Before implementing any solution, I needed to understand why our E2E tests were flaky. After analyzing our failure patterns, I identified three main categories:

Race conditions (40%): Tests expecting UI updates faster than they could render

Network timing issues (35%): API responses taking slightly longer in CI than locally

Environment-specific failures (25%): Browser quirks and CI resource contention

The key insight? 80% of our flaky failures would pass on the very next run.

The Solution: A Smart Retry Policy
Instead of overhauling our entire test suite (which would take months), I implemented a pragmatic retry strategy:

javascript
// Example: Playwright retry configuration
import { defineConfig } from '@playwright/test';

export default defineConfig({
retries: process.env.CI ? 2 : 0, // Retry twice in CI only
workers: 4,
timeout: 60000,
// ... other config
});
The rule was simple:

Local development: 0 retries (catch issues early)

CI pipeline: 2 retries (give flaky tests a second chance)

Only retry the specific failed test, not the entire suite

The Results: What 10% Improvement Looks Like
Metric Before After Change
Build stability 89% 99% +10%
Developer productivity 3-4 hours/week wasted <30 min/week ~3 hours saved/developer
Main branch breakages 3/month 0/month (so far) 100% reduction
Team confidence Low High Priceless
Lessons Learned Along the Way

Retry Policies Are a Band-Aid, Not a Cure While retries immediately improved stability, they didn't fix the underlying issues. We made a pact to:

Investigate any test that needed both retries to pass

Add logging around flaky areas to identify root causes

Gradually rewrite the flakiest tests

Communicate the Change We sent a simple Slack message explaining the new policy:

🔧 New Flaky Test Policy: We've added automatic retries (2x) for E2E tests in CI. If a test fails twice, we need to investigate. This should reduce false positives while we continue to stabilize our test suite.

Track Retry Metrics We started monitoring how often retries actually helped:

javascript
// Track retry success rate
const flakyTestMetrics = {
totalRuns: 1247,
retriesUsed: 89,
flakyPasses: 74, // 83% of retries were successful
persistentFailures: 15 // Tests that truly needed fixing
};
This data helped us prioritize which tests to permanently fix.

What We Fixed Permanently
Once the immediate crisis was over, we systematically addressed the root causes:

Category Fix Impact
Race conditions Added explicit wait strategies (waitFor, expect.poll) Reduced retries needed by 60%
Network timing Increased API timeouts in CI environment Reduced retries needed by 25%
Environment issues Fixed browser versions and resource allocation Reduced retries needed by 15%
The Bottom Line
Retries are not a silver bullet. They're a pragmatic tool to stop bleeding while you heal the wound.

For our team, implementing a flaky-test retry policy was the difference between:

Chaos: Wasting time on false positives, broken builds, and blame games

Calm: Trusting our CI, shipping faster, and knowing that when a build actually fails, it's real

From 89% to 99% stability in a week—not because our tests got better overnight, but because we stopped letting perfection be the enemy of progress.

Your Turn: How to Implement This
Step 1: Choose Your Retry Strategy
Playwright: retries: 2 in config

Cypress: retries: { runMode: 2, openMode: 0 }

Jest/Puppeteer: Custom wrapper or jest.retryTimes(2)

Step 2: Set Clear Rules
CI: 2 retries

Local: 0 retries

Report: Log when retries are used

Step 3: Fix the Worst Offenders
Use retry data to identify and permanently fix the flakiest 20% of tests—they probably cause 80% of the frustration.

Step 4: Monitor and Iterate
Track your stability score and gradually reduce retries as you fix underlying issues.

Final Thoughts
Our main branch hasn't broken in 6 weeks now. The team is shipping faster, with more confidence, and less stress. The retry policy bought us the time we needed to fix things properly.

Sometimes the best solution isn't perfect—it's the one that works right now.

Have you dealt with flaky E2E tests? How did your team handle it? Share your story in the comments below! 👇

Killing Jenkins Was the Best Career Move I Made This Year

Sohana Akbar — Mon, 13 Jul 2026 08:43:42 +0000

Senior DevOps Engineer | 15 microservices | 2,800 deployments last month

Let me tell you about the day I took down our Jenkins master during peak traffic. 3:47 PM on a Thursday. 147 builds queued. 12 angry devs in Slack. VP of Engineering asking "what's the ETA?" like I had a crystal ball.

I fixed it by disabling the master node for 14 minutes while the database recovered. Then I spent the weekend migrating everything to GitHub Actions. That was 8 months ago. We haven't had a CI outage since.

Here's what I learned, what broke, and what I'd do differently.

The Problem Wasn't Jenkins. It Was Us.
Jenkins worked fine for 4 years. We had:

15 microservices with 200+ test suites each

6 shared Jenkins agents in Kubernetes

3,400 lines of pipeline-as-code

87 plugins (most of them obsolete)

1 overworked master node with 16GB RAM

The crash happened because a developer pushed a PR that triggered 27 downstream jobs simultaneously. The master ran out of heap space. Not because of a memory leak—because we never tuned -Xmx past 8GB.

What broke: We had 4 separate Jenkinsfiles per repo (build, test, deploy, rollback). Devs kept copying them from Stack Overflow without understanding the Groovy syntax. Pipeline logs were 40MB of noise.

How I fixed it: I didn't. I killed it.

Why GitHub Actions Won (For Us)
Metric Jenkins GitHub Actions
Build time (avg) 8m 42s 3m 17s
Pipeline failures Dev: "works on my machine" Dev: sees exact error in PR
Plugins to maintain 87 13 (all official)
Cost/month $427 (EC2) $0 (free tier + 2,000 minutes)
Developer complaints Daily Zero in 8 months
The real win: devs debug their own failures now. No more "hey DevOps, my pipeline is red" tickets. They see the exact step that failed, with the exact YAML line, right in the PR checks tab.

No more SSH-ing into Jenkins agents to read workspace logs. No more "can you restart the master?" at 2 AM.

The Migration: What Actually Worked
Phase 1: The Parallel Run (2 weeks)
We kept Jenkins as primary while I ported the simplest service to Actions:

yaml
name: CI
on: [push, pull_request]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- run: npm ci
- run: npm test
- run: npm run build
That's it. 14 lines. The Jenkinsfile for the same service was 187 lines.

Phase 2: The Matrix Strategy (1 week)
Our biggest headache in Jenkins was running tests across 6 Node versions and 3 databases. Actions handled this natively:

yaml
test-matrix:
runs-on: ubuntu-latest
strategy:
matrix:
node: [16, 18, 20]
db: [postgres, mysql, sqlite]
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: ${{ matrix.node }}
- run: docker-compose up -d ${{ matrix.db }}
- run: npm test
18 parallel jobs. 4 minutes total. Jenkins took 12 minutes sequentially because we never configured parallel stages properly.

Phase 3: The Monorepo Migration (The Painful One)
We have 3 microservices sharing code in one repo. Jenkins handled this with a custom Groovy script that conditionally built based on changed paths.

Actions does this natively with path filters:

yaml
on:
push:
paths:
- 'services/auth/'
- 'shared/'
pull_request:
paths:
- 'services/auth/'
- 'shared/'
But we needed to build and deploy only the changed service. Here's the pattern that works:

yaml
changed-files:
uses: actions/checkout@v4
with:
fetch-depth: 2

id: files
uses: tj-actions/changed-files@v39
with:
files: |
services/auth/**
name: build-auth
if: steps.files.outputs.any_changed == 'true'
run: |
cd services/auth
docker build -t auth-service .
docker push ...
What I'd do differently: I'd split the monorepo sooner. We wasted 3 days fighting the path filter syntax. Use tj-actions/changed-files from day one. It's the only third-party action I recommend.

The Security Nightmare We Fixed
In Jenkins, we stored secrets as environment variables in the master's configuration. Every developer with "read" access could see them in the build logs if they added printenv.

In Actions, we use GitHub Secrets scoped to the repository:

yaml

name: Deploy to ECS env: AWS_ACCESS_KEY_ID: ${{ secrets.AWS_ACCESS_KEY_ID }} AWS_SECRET_ACCESS_KEY: ${{ secrets.AWS_SECRET_ACCESS_KEY }} ECR_REPO: ${{ secrets.ECR_REPO }} run: | aws ecs update-service ... You can also use environment-level secrets for staging vs production:

yaml
deploy-prod:
runs-on: ubuntu-latest
environment: production
steps:
- run: deploy.sh
env:
API_KEY: ${{ secrets.PROD_API_KEY }}
The one gotcha: You can't reuse the same secret name across environments. secrets.API_KEY exists once. Name them secrets.DEV_API_KEY, secrets.PROD_API_KEY.

The Things That Almost Broke Us

OIDC vs Static Credentials We started with static AWS keys rotated monthly. Then a developer committed dev.env with credentials. Never again.

Use OIDC authentication:

yaml

name: Configure AWS Credentials uses: aws-actions/configure-aws-credentials@v4 with: role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT }}:role/github-actions aws-region: us-east-1 No keys. No rotation. Each run gets a temporary session.

What broke: I forgot to update the trust relationship when we added a new repo. 5 AM Sunday, deploy failed, I'm reading AWS docs on my phone.

The fix: Use a reusable workflow with the OIDC config baked in:

yaml

.github/workflows/deploy.yml (shared)

on:
workflow_call:
inputs:
environment:
required: true
type: string

jobs:
deploy:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: read
steps:
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::${{ secrets.AWS_ACCOUNT }}:role/github-actions-${{ inputs.environment }}
Now every repo calls this. The trust relationship is in one place.

Docker Build Cache Our Jenkins agents stored Docker layers in /var/lib/docker. Builds were fast because images were cached locally.

GitHub Actions runners are ephemeral. No cache = 6-minute builds from scratch.

The fix: Use GitHub's cache action for Docker layers:

yaml

name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
name: Cache Docker layers
uses: actions/cache@v3
with:
path: /tmp/.buildx-cache
key: ${{ runner.os }}-buildx-${{ github.sha }}
restore-keys: |
${{ runner.os }}-buildx-
name: Build and push
uses: docker/build-push-action@v5
with:
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache
Build time dropped from 6m to 1m 40s. The cache key uses the commit SHA, so PRs restore from main's cache.

Self-Hosted Runners (The "We're Too Big" Problem) We peaked at 2,800 runs/month. GitHub-hosted runners were costing us $0.008/minute × ~1,200 minutes/day = $288/day.

We moved to self-hosted runners on AWS EC2 Spot instances. Cost dropped to $0.003/minute.

The YAML:

yaml
runs-on: self-hosted
The catch: Self-hosted runners don't auto-update. We're running Ubuntu 22.04 LTS with manual patching. I have an Ansible playbook that rebuilds the AMI weekly.

The setup (Terraform):

hcl
resource "aws_ec2_instance" "runner" {
ami = data.aws_ami.ubuntu.id
instance_type = "c6i.4xlarge"
spot_price = "0.15"

user_data = file("${path.module}/user-data.sh")

tags = {
Name = "github-actions-runner-${count.index}"
}
}
We have 6 runners. 3 for PRs, 3 for main branch. The PR runners terminate after 30 minutes idle. Main runners stay hot.

What I'd do differently: Start with GitHub-hosted until you actually need self-hosted. We prematurely optimized. The $288/day was worth the zero maintenance in hindsight.

What Broke That I Didn't Expect

The GITHUB_TOKEN Permissions Issue Our CD pipeline needed to comment on PRs after deployment. I used the default ${{ secrets.GITHUB_TOKEN }} and got a 403.

Solution: Set permissions at the job level:

yaml
jobs:
deploy:
runs-on: self-hosted
permissions:
contents: read
pull-requests: write
issues: write
steps:
- name: Comment on PR
uses: actions/github-script@v7
with:
script: |
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: '🚀 Deployed to production'
})
Took me 3 hours of trial and error. The GitHub docs bury this in the "Advanced" section.

Concurrency Limits Two PRs trying to deploy to the same environment simultaneously. Database migrations conflict. The first succeeds, the second fails with "table already exists."

The fix: Use concurrency groups:

yaml
concurrency:
group: ${{ github.workflow }}-${{ github.ref_name }}
cancel-in-progress: true
Now the second job waits for the first to finish. No more migration conflicts.

The 60-Minute Timeout One of our E2E test suites takes 45 minutes on a good day. GitHub-hosted runners have a 6-hour limit. Self-hosted don't (we set our own).

But we had a test that hung for 2 hours because of a race condition. The runner stayed active, blocking the queue.

Fix: Set explicit timeouts:

yaml
jobs:
e2e:
runs-on: self-hosted
timeout-minutes: 55
steps:
- run: npm run test:e2e
Now it fails at 55 minutes, not 120. The race condition got fixed because developers actually saw the failure.

The Numbers After 8 Months
Metric Before After
Average build time 8m 42s 3m 17s
Deployment frequency 42/day 93/day
CI failure rate 8% 1.2%
Pipeline code (total lines) 3,400 620
Dev tickets to DevOps 34/week 3/week
Sleep score (my Oura ring) 67 83
The last one is real. No more 3 AM Jenkins alerts. No more "can you restart the agent?" at 11 PM.

What I'd Do Differently (The Honest Retro)
Migrate one service at a time. I did the monorepo last. Should've done it first. The monorepo had the most complexity and cost us the most time.

Start with reusable workflows. I wrote the same YAML 15 times. Now I have a central .github/workflows/reusable-test.yml that all repos import:

yaml
name: Test
on: [workflow_call]

jobs:
test:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run tests
run: npm test
Don't try to replicate Jenkins 1:1. I wasted a week making Actions "look like Jenkins." The pipeline should match your workflow, not your old tool.

Document the failure points. When I migrated, I kept notes on every error. That became our internal wiki. Now new DevOps engineers onboard in 2 days instead of 2 weeks.

Set up monitoring before the migration. I had no baseline for build times. I assumed Actions was faster. It was, but I couldn't prove it. Use the GitHub API to log metrics from day one.

The Bottom Line
Jenkins is a fine tool if you have 1-2 services and a dedicated team to maintain it. We had 15 services and me.

GitHub Actions eliminated the maintenance overhead. Developers fix their own pipelines because the errors are in their PRs, not buried in a Jenkins build #4837 log.

The migration cost me 3 weeks of focused work. The ROI was 2 months.

Would I do it again? In a heartbeat. But I'd start the monorepo split 6 months earlier and I'd write reusable workflows from day one.

My advice if you're considering it: Try the migration on a Sunday. Pick your simplest service. Get it running in Actions with the PR check passing. See how many lines of YAML it takes. Compare that to your Jenkinsfile. Then decide.

Just don't kill your Jenkins master at 3:47 PM on a Thursday. Learn from me.

Senior DevOps Engineer. 15 microservices. 2,800 deployments/month. 0 CI outages in 8 months.

Connect if you want to compare war stories. I've got more.

DEV Community: Sohana Akbar

Why We Ditched npm install in Production (And You Should Too)

Before (Dangerous)

After (Production-Ready)

The magic line

GitHub Actions example

From 1.2GB to 118MB: How We Slashed Our Docker Image Size and EKS Pull Time to Under 8 Seconds

Copy only what we need

Stage 1: Builder

Stage 2: Runner

Docker #Kubernetes #EKS #DevOps #pnpm #ContainerOptimization #AWS #CloudNative

From 5 PHP Nightmares to 1 Node Pipeline: Why Boring Standardization Is My Greatest Achievement

From 1 Week to 1 Day: The Unsexy Power of Standardization

From Deployment Chaos to Clarity: Automating Jira Ticket Linking in Our Production Pipeline

.github/workflows/deploy.yml snippet

Fetch details for each ticket

!/bin/bash

fetch-jira.sh - Fetch ticket details from Jira

When Your Code Quality Gate Makes Developers Cry (But Your Bugs Disappear)

How We Banished "Latest" and Made Rollbacks Instant

In our CI pipeline

deployment.yaml

Terraform validation

How We Caught 12 Breaking API Changes Before They Hit Main: Our Journey to Ephemeral Staging Environments

From 15 Daily Deployments to Zero Downtime: Our Next.js ECS Fargate Journey

The 2-Week Journey to Next.js Zero-Downtime on ECS Fargate

How I Tamed Flaky E2E Tests and Boosted Build Stability from 89% to 99%

Killing Jenkins Was the Best Career Move I Made This Year

.github/workflows/deploy.yml (shared)