How We Banished "Latest" and Made Rollbacks Instant

Stop using :latest in production. Here’s why—and how we did it.

The Problem We All Know
It starts innocently enough:

bash
docker build -t myapp:latest .
docker push myapp:latest
You deploy. It works. You move on.

Then Monday morning hits. Your monitoring dashboard turns red. Users are complaining. Something broke in the last deploy—but what exactly changed? The :latest tag moved yesterday, then again this morning, and maybe once more during that hotfix at 2 AM.

Good luck figuring out which version is actually running.

Our Breaking Point
We had a "stable" production environment. We had CI pipelines. We had rollback scripts. And yet, every incident turned into a forensic investigation:

"Which commit is this container running?"

"Did that hotfix make it in?"

"Is staging on the same version as production?"

"Can I even roll back to yesterday's image?"

The answer was almost always: ¯_(ツ)_/¯

Worst of all—rollbacks weren't truly instant. We'd revert code, rebuild, repush, and redeploy. That's 8-12 minutes of downtime during an active incident.

The Decision: Ban :latest in Production
We made a drastic but simple rule:

No :latest tag is ever pushed to our production ECR repository.

Not "discouraged." Not "we'll phase it out."

Banned. Blocked. Period.

What We Did Instead
Every image gets tagged with its commit hash (shortened to 7 characters):

bash

In our CI pipeline

COMMIT_HASH=$(git rev-parse --short HEAD)
docker build -t myapp:$COMMIT_HASH .
docker push myapp:$COMMIT_HASH
That's it. No :latest. No environment-specific tags. Just the immutable commit hash.

The Architecture
Here's what our ECR repo looks like now:

text
123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp
├── abc1234 # commit hash
├── def5678
├── ghi9012
├── jkl3456
└── mno7890
Every hash is immutable. Once pushed, it never changes. No overwrites. No ambiguity.

Deployments Become Explicit
Our deployment manifest (Kubernetes, ECS, etc.) now references the exact commit hash:

yaml

deployment.yaml

image: 123456789012.dkr.ecr.us-east-1.amazonaws.com/myapp:abc1234
No more abstract "latest" that means different things at different times. The deployment manifest tells you exactly what's running.

The Magic: Instant Rollbacks
This is where everything changed.

Before:

text

1. Detect incident (2 min)
2. Identify bad commit (5-15 min)
3. Revert code in Git (3 min)
4. CI builds new image (5-8 min)
5. Push to ECR (2 min)
6. Deploy (2 min) Total: 19-32 minutes After:

text

1. Detect incident (2 min)
2. Find previous known-good commit hash
3. Update manifest and redeploy (2 min) Total: 4 minutes Because every commit hash is an immutable, ready-to-run image in ECR, rollback is just a manifest change. No rebuild. No repush. No waiting.

We went from 20+ minute rollbacks to under 5 minutes.

How We Enforced It
We implemented three layers of protection:

1. CI Pipeline Enforcement bash # Block :latest pushes to production ECR if [[ "$ECR_REPO" == "production" && "$IMAGE_TAG" == "latest" ]]; then echo "❌ Cannot push :latest to production ECR" exit 1 fi

2. ECR Lifecycle Policy
   json
   {
   "rules": [
   {
   "rulePriority": 1,
   "description": "Expire all untagged images",
   "selection": {
   "tagStatus": "untagged",
   "countType": "imageCountMoreThan",
   "countNumber": 100
   },
   "action": {
   "type": "expire"
   }
   }
   ]
   }
   We also expire old images after 90 days to keep costs manageable.

3. Terraform/CloudFormation Guardrails
   Our infrastructure-as-code templates block any deployment that doesn't specify an explicit, non-latest tag:

hcl

Terraform validation

variable "image_tag" {
type = string
validation {
condition = var.image_tag != "latest"
error_message = "Cannot use 'latest' tag in production."
}
}
The Unexpected Benefits
Once we banned :latest, we discovered perks we hadn't anticipated:

1. Instant Audit Trail
   Each deployment is tied to a Git commit SHA. You can trace every container back to its source code, CI build logs, and the exact moment it was created.

2. Zero Ambiguity
   "Which version is running?" Look at the tag. No guessing. No "well, it's latest from yesterday."

3. Simplified Debugging
   When developers SSH into a container, docker inspect shows the commit hash. They know exactly which code they're debugging.

4. Safe Canary Deployments
   We can deploy abc1234 to 10% of traffic and def5678 to 90%, comparing metrics without any tagging confusion.

5. Cleaner Rollforward
   If the rollback fixes the issue, we can later redeploy the same hash or a new one—all without tagging collisions.

What About Staging and Dev?
We kept :latest for non-production environments:

Dev: :latest is fine for rapid iteration.

Staging: We use commit hashes too, but allow :latest as an alias for convenience.

The production repo is the only one with the strict ban.

The One Thing to Watch Out For
Image bloat.

If you push every commit to ECR and never clean up, your storage costs will creep up. Our solution:

Lifecycle rule: Expire images older than 90 days.

Selective retention: Keep the last 50 images regardless of age (for emergency rollbacks).

Manual archives: For major releases, we pin specific hashes as "golden images" with infinite retention.

The Results (By The Numbers)
After 6 months with this system:

Metric Before After
Avg rollback time 22 min 4 min
Rollback success rate 72% 98%
Incident resolution time 45 min 18 min
Developer confusion High Zero
"What's deployed?" questions Daily Never
How to Implement This Tomorrow
Stop pushing :latest to prod today. Seriously. Just stop.

Use git rev-parse --short HEAD as your primary tag.

Update your CD pipeline to reference explicit hashes in manifests.

Store the current production hash somewhere (we use a simple S3 file with the hash and timestamp).

Test a rollback right now—manually update your manifest to a known-good hash and redeploy. Time it.

Celebrate when it takes < 5 minutes.

The Bottom Line
docker push myapp:latest is a developer convenience, not a production strategy.

Banning :latest from our ECR repo was one of the simplest, highest-impact changes we made to our deployment pipeline. It didn't require rewriting architecture or buying new tools. It just required a rule—and the discipline to enforce it.

Rollbacks went from a stressful, multi-step ordeal to a boring, scripted operation.

And boring operations are the best kind.

Your turn: Are you still using :latest in production? What's stopping you from switching?

Follow me for more production engineering lessons learned the hard way.