DEV Community

Honey Kiran
Honey Kiran

Posted on Originally published at honeykiran.substack.com on

Your Image Tag Says ‘latest’. Your Rollback Says Good Luck.

Illustrative — but you have lived a version of it. A night deploy goes wrong; the team rolls back: re-apply the old manifest, image: api:latest. The pods come back green.

Nothing rolled back. Between deploys, another build pushed to latest. The tag moved.

Not a people problem. A naming problem.

Animated illustration of a night registry dock: a ROLLBACK lever re-applies api:latest while a new push moves the api:latest sticker from good build #41 to broken build #42, and the deploy lands on #42. End frame: You rolled back. The name stayed. The code didn't.

A nickname, not a name

A tag is a pointer the registry lets you move. Push the same tag again, and it quietly points at the new image.

The digest is the real name: a SHA-256 hash of the content itself. Change one byte, different fingerprint. api@sha256:... pins that exact content.

No tag means everyone assumes latest, which promises neither newest nor tested. Even a tidy 1.4.2 can move, unless the registry locks it.

A tag tells you what someone called it. A digest tells you what it actually is.

Animated close-up of two crates: the orange latest sticker (TAG) peels off crate #41 and sticks to newly pushed #42, while #41's bolted sha256:1a2b3c plate (DIGEST) never moves. End frame: A tag tells you what someone called it. A digest tells you what it actually is.

Why rollbacks break first

A rollback has one job: run the exact code that ran before. A moving tag cannot name that code.

Leave out imagePullPolicy, and :latest or no tag defaults to Always: every new pod re-asks the registry now. A specific tag or digest defaults to IfNotPresent.

Mid-incident, old pods run the old build, new pods pull the new one: a mixture in no manifest.

Kubernetes' own guidance: avoid :latest in production. Harder to track, harder to roll back.

What ran at 2 PM? With digests, a lookup. With latest, archaeology.

Animated diagram: one manifest reading image: api:latest feeds four pods on build #41; after NEW BUILD PUSHED #42 to latest, pods 2 and 4 restart onto #42 while pods 1 and 3 stay on #41. Legend: :latest defaults to Always, a digest to IfNotPresent. End frame: One manifest. A mixture that exists in no manifest anywhere.

Tag like you mean it

Two tags, two jobs.

Every build gets the full Git SHA as a tag: unique to its source, walkable from a running image back to the commit.

Releases also get a semver tag for humans: MAJOR breaks, MINOR adds, PATCH fixes. Semver's rule fits this post: once released, a version's contents must not be modified.

The pipeline then records the pushed image's digest with the build. That digest is the deployable fact.

IMAGE=registry.example.com/shop/api
SHA=$(git rev-parse HEAD)
VERSION=1.4.2

docker build -t $IMAGE:$SHA -t $IMAGE:v$VERSION .
docker push $IMAGE:$SHA
docker push $IMAGE:v$VERSION

# Capture the digest - this is the identity you deploy
docker inspect --format='{{index .RepoDigests 0}}' $IMAGE:$SHA
# Illustrative output:
# registry.example.com/shop/api@sha256:1a2b3c...(full 64-character hash in real output)
Enter fullscreen mode Exit fullscreen mode

Deploy the fingerprint

The manifest carries name:tag@sha256:digest. Humans read the tag; given both, Kubernetes pulls only the digest.

IfNotPresent sits happily beside a digest: it never changes, so there is no newer to fetch.

Rollback gets boring on purpose: put the previous recorded digest back and apply. No re-pushing, no re-tagging. Boring is the feature.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
  namespace: shop
spec:
  replicas: 3
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      containers:
        - name: api
          image: registry.example.com/shop/api:1.4.2@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 8080
Enter fullscreen mode Exit fullscreen mode

Animated clip: a manifest reading image: api:latest feeds four pods on #41; after a new push, pods 2 and 4 restart and come back on #42 while pods 1 and 3 stay on #41, beside a pull-policy defaults legend. End frame: One manifest. A mixture that exists in no manifest anywhere.

Lock it, then sign it

Immutable tags make the registry refuse to move a pushed tag.

Amazon ECR defaults to MUTABLE. Set ImageTagMutability to IMMUTABLE, and re-pushing an existing tag fails with ImageTagAlreadyExistsException.

On Google Artifact Registry, an immutable tag can't be removed, moved, or reused, and a tagged image can't be deleted. Docker Hub offers the same switch.

Flip it per repository, tell the team first, and test in staging: push one tag twice; watch the second fail.

If you can, sign with Sigstore's cosign. Sign the digest, never just the tag: a tag can move after you sign. Verification checks the signature's digest matches the image.

# Sign the digest, never just the tag
cosign sign --key cosign.key registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD

# Verify before anything deploys it
cosign verify --key cosign.pub registry.example.com/shop/api@sha256:REPLACE_WITH_THE_DIGEST_FROM_YOUR_BUILD
Enter fullscreen mode Exit fullscreen mode

The Image Identity Checklist

  1. Never deploy latest: in production or staging; no tag means latest.
  2. Tag every CI build with the full Git SHA: traceable to its source.
  3. Add a semver tag on releases: for humans.
  4. Record every build's digest: no record, no rollback.
  5. Deploy by digest: name:tag@sha256:digest.
  6. Roll back by restoring the previous digest: never re-push or re-tag.
  7. Turn on immutable tags: ECR, Artifact Registry, Docker Hub, repository by repository.
  8. Test the lock in staging, then sign the digest if you can: cosign sign, then verify.

Saveable checklist card titled The Image Identity Checklist, listing the eight items from this post: never deploy latest, tag every CI build with the full Git SHA, add a semver tag on releases, record every build's digest, deploy by digest, roll back by restoring the previous digest, turn on immutable tags, test the lock in staging then sign the digest. Footer: Nicknames are for friends. Rollbacks need names.

Let latest live in development, where surprises are cheap. In production, let the manifest say the fingerprint.

Nicknames are for friends. Rollbacks need names.

Quick check before you close this tab: what do your production manifests reference right now — latest, a tag, or a digest? Tell me in the comments — and if it is latest, tell me what your rollback plan calls it.


Disclaimer

  • This post describes my personal experience with my own systems and accounts, in my own words. It is a first-person account of what I observed and what I do in my own work — not a statement of any company’s intentions, policies, or internal decisions. Where I do not know something, the post says so.

  • I am not affiliated with, endorsed by, or speaking for any company, product, or vendor named in this post. All product and company names are trademarks of their respective owners and are mentioned only to describe the experience accurately.

  • Any error messages, documentation excerpts, or figures quoted here are reproduced from my own records or from the public sources linked in the post, believed accurate at the time of writing. Product behavior changes over time; verify against current official documentation before relying on any detail.

  • Nothing in this post is professional, financial, legal, or security advice for your specific situation. Examples are illustrative — validate in your own environment before running anything in production.

  • Corrections are welcome. If any company named here publishes an official statement, or if any detail is wrong, I will update this post and note the correction. The goal is an accurate public record, not a verdict.


© 2026 Honey Kiran. All rights reserved.

Honey Kiran · DevOps & Platform Engineering

AI agents transforming DevOps, infra, platform, security & SRE. The shift nobody works anymore: deep dives, 3AM Files, and OpsBites.

Examples are illustrative — validate in your own environment before running in production.

Top comments (0)