DEV Community

Anandh
Anandh

Posted on

The Case of the Zombie Code: Why my EC2 refused to update 🕵️‍♂️

You know that feeling when you push a brilliant bug fix, the CI/CD pipeline gives you a "thumbs up," and you go to grab a celebratory coffee?

Then you open your app and realize... nothing changed. The code in your browser is the "Zombie version"—the one you thought you killed 10 minutes ago. It’s back from the dead, and it’s haunting your EC2 instance. 🧟‍♂️

The Crime Scene

GitHub Actions: Success.
AWS ECR: The latest image is sitting there with a fresh timestamp.
EC2 Instance: Still running code from last Tuesday.

The Culprit? Docker’s local image cache.

Docker is like that friend who "already has a guy for that." When you tell it to run the latest image, it looks at its own internal shelf first. If it sees an image tagged latest, it says, "I’ve already got it! No need to download anything new." Even if the "new" latest is totally different.

The Investigation: How to Force a "Search and Seizure"
To fix this, we have to perform an "Eviction Notice" on the old image. Here is the play-by-play on how to force your EC2 to stop living in the past.

Step 1: Prove who you are (ECR Login)
You can't take anything from ECR if you don't have the keys.

Bash

aws ecr get-login-password --region YOUR_REGION | docker login --username AWS --password-stdin YOUR_ECR_REGISTRY
Enter fullscreen mode Exit fullscreen mode

Step 2: Stop the "Imposter" Container
We can't update the engine while the car is still driving. Shut it down.

Bash

docker stop backend-app
docker rm backend-app
Enter fullscreen mode Exit fullscreen mode

Step 3: Delete the Evidence (The Cache Buster)
This is the most important step. If you don't delete the local image, Docker will just reuse it. Delete the local copy so Docker is forced to go back to AWS ECR.

Bash

docker rmi YOUR_ECR_REGISTRY/YOUR_REPO:latest
Enter fullscreen mode Exit fullscreen mode

Step 4: The Fresh Start
Now, we bring in the actual "Latest" version from the cloud.

Bash

Pull the actual new image

docker pull YOUR_ECR_REGISTRY/YOUR_REPO:latest
Enter fullscreen mode Exit fullscreen mode

Run the app

docker run -d --name backend-app -p 8081:8081 --restart always YOUR_ECR_REGISTRY/YOUR_REPO:latest
Enter fullscreen mode Exit fullscreen mode

💡 The "Senior Dev" Advice
If you find yourself doing this manual dance every day, it might be time to move away from the :latest tag in your deployment scripts.

The Better Way: Use the Commit SHA as your tag (e.g., my-app:8f2a4b1).

Your GitHub Action builds and tags the image with the SHA.

Your EC2 deployment script asks for that specific SHA.

Docker looks at its cache, sees it doesn't have that specific version, and pulls it automatically. No more zombies.

Closing Thoughts
Docker cache is a double-edged sword. It makes things fast, but it can also make you feel like you're losing your mind. When in doubt: Delete the local image and pull again.

Have you ever spent hours debugging something that turned out to be a cache issue? Share your "war stories" below! 👇

AWS #Docker #DevOps #EC2 #ECR

Top comments (0)