Blue-green deployment that left the old environment running for weeks, doubling infrastructure cost

Usman Tahir Qureshi — Wed, 26 Aug 2026 21:47:03 +0000

The deploy worked. The bill doubled.

The blue-green cutover went perfectly. Traffic shifted to green, health checks passed, the team signed off, and moved on. It was one of those rare deployments that goes exactly as planned.

Six weeks later, a cost anomaly surfaced in the monthly AWS review. Infrastructure spend had been running at roughly double what it should have been since the deployment date. Every EC2 instance, every RDS node, every load balancer from the blue environment was still running. Serving zero traffic. Billed at full price. For six weeks.

Nobody had decommissioned it because nobody owned it after cutover. The team that ran the deployment assumed operations would clean it up. Operations assumed the team that deployed it would tear it down. The blue environment sat in a perfect ownership gap, healthy and idle and expensive, while both teams closed their tickets and moved on.

This is the part blue-green deployment guides don't emphasize enough. The strategy is excellent for zero downtime releases and instant rollback capability. The rollback window is the dangerous part. It's open-ended by default, which means the old environment stays alive until someone makes a deliberate decision to shut it down. That decision requires ownership, and ownership requires someone to be responsible for it after the deployment is considered done.

The fix is treating decommissioning as part of the deployment itself, not cleanup that happens afterward.

Tag every blue environment resource at launch with a TTL:

aws ec2 create-tags \
  --resources i-1234567890abcdef0 \
  --tags Key=DeploymentColor,Value=blue \
           Key=CutoverDate,Value=2026-01-14 \
           Key=TTL,Value=2026-01-21

Then wire Cost Anomaly Detection to alert when a specific environment tag is still generating spend past its TTL. The old environment doesn't get to become invisible just because traffic moved away from it.

The deeper issue is that blue-green deployments create a window of parallel infrastructure that most cost monitoring ignores because both environments look healthy. Neither throws an error. Neither generates an alert. The only signal is the bill, and by the time the bill arrives, the window has been open for thirty days.

Rollback capability has a cost, and that cost has a deadline. If you haven't defined when blue gets torn down before you cut over to green, you've defined it as never. And never is expensive.

AWS #DevOps #SRE #BlueGreen #FinOps #CloudEngineering #PlatformEngineering

Building DrScore: A Practical Way to Measure Disaster Recovery Readiness

Usman Tahir Qureshi — Mon, 24 Aug 2026 17:07:22 +0000

GitHub Repo

The checkbox that wasn't real

A few months ago, I sat in a meeting where someone confidently said, "Yes, Multi AZ is on for everything in production." Nobody pushed back. It sounded right. It had probably been true at some point, for some database, in some region.

Nobody actually checked.

That is the moment DrScore started for me. Not as a grand idea, but as a small, uncomfortable realization: most disaster recovery plans are not plans at all. They are memories of a plan, slowly going stale while nobody notices, because nothing forces anyone to look.

What DrScore actually is

DrScore is a tool for measuring disaster recovery posture without the ceremony. No forty-page whitepaper, no consultant, no compliance binder that gets updated once a year right before an audit.

Today it works as a guided self-assessment. You work through the controls that actually matter: backups, failover, monitoring, the usual suspects, and you get a score along with a breakdown by category. Every control can also carry tips from other users, the kind of "here is how we actually did this" detail that never makes it into official docs. The score tells you what is weak. The tips tell you how other engineers fixed the same weakness.

Under the hood, it is Next.js and Supabase, with row-level security keeping everyone's data locked to their own account. Nothing exotic. That is deliberate. The interesting engineering is happening in what comes next.

Trust, but verify

Here is the uncomfortable truth about self-assessments: they only work if people answer honestly, and honesty requires actually knowing the answer. Someone checks a box because it was true six months ago, or true in one region, or true before an intern changed a setting nobody documented.

So the next phase of DrScore, called AWS Live Scan, skips the honor system entirely. It connects to a real AWS account, read-only, and checks what is actually configured instead of asking someone to remember.

The flow looks like this:

User clicks Connect AWS Account | DrScore generates a unique external ID | DrScore generates a least-privilege IAM policy scoped to read-only checks | User deploys a one-click CloudFormation template in their own account | DrScore's scanner identity assumes the role via sts: AssumeRole | Findings flow back into DrScore. Nothing is ever written to the customer account.

No stored access keys, ever. If someone is going to trust a tool with visibility into their production AWS account, the least I can do is make sure that trust cannot be abused, even by accident. I would rather ship five verified checks than one convenient shortcut involving a long-lived credential.

Where things stand

I am building this in public, unfinished parts included.

The connection flow and data model are live: the tables for connections, scans, and findings, protected end-to-end with row-level security, plus the actual "Connect AWS Account" experience. The scanning backend is next, starting with a real sts AssumeRole verification and the first checks: RDS Multi-AZ, AWS Backup coverage, and EBS snapshot freshness. Right now, clicking Verify honestly tells you it is not ready yet, because I would rather show you a true "not yet" than a fake "done."

After that comes a results view with findings grouped by severity, then broader multi-region coverage: S3 replication, Route 53 failover, Aurora Global Database, and more.

If you have solved this before

If you have ever built anything that reads AWS accounts read-only, whether it was Steampipe, Prowler, or something you duct-taped together at 2 am, I want to hear about it. Specifically, how did you handle multi-region checks without getting throttled into oblivion? That is the wall I expect to hit next.

This is the first post in a series I plan to keep writing as DrScore gets built, including the parts that break. Follow along, and tell me what I am getting wrong.

DEV Community: Usman Tahir Qureshi

Blue-green deployment that left the old environment running for weeks, doubling infrastructure cost

AWS #DevOps #SRE #BlueGreen #FinOps #CloudEngineering #PlatformEngineering

Building DrScore: A Practical Way to Measure Disaster Recovery Readiness