ErrImagePull and ImagePullBackOff in Kubernetes

When you create a pod in Kubernetes, it goes through a lot of steps. One of the important tasks is to pull the container image from the registry. While pulling those images from the registry, pods sometimes experience issues. If an error occurs, pods go into ErrImagePull, ImagePullBackOff errors.

• ErrImagePull happens when the image that is specified in the pod spec can’t be fetched.

• ImagePullBackOff is the grace period while the image pull is fixed. After each unsuccessful pull, the delay (grace period) increases.

Container Image and Image Pull Policy

A container image includes the binary data of the application and its dependencies. So you have to create the image for your application and push it to the registry before you refer that image in the Pod. The image registry by default is Docker unless you specified the registry.

Tags are also an important part of the image name, tags are used to identify different versions of the image.

Image pull policy defines when the kubelet tries to pull the image. the IfNotPresent value tells the kubelet to pull the image only if it is not present locally. Never tells the kubelet not to fetch the image. And Always tells kubelet to check the registry and pull the image for this pod. By default, the Image pull policy is set to Always.

ErrImagePull

This occurs when the kubelet tries to pull the image from the registry (based on its image pull policy). Sometimes things can go wrong, we might have messed up the image spec in the pod definition.

For example, I have specified my image as nginxxx . Let’s check the pod status by running kubectl get pods .

controlplane $ kubectl get pods
NAME     READY   STATUS         RESTARTS   AGE
my-pod   0/1     ErrImagePull   0          13s

ImagePullBackOff

ImagePullBackOff is the waiting status, a grace period with increasing time for every retry. Once this grace period completes, Kubernetes pulls the image again. Maximum backoff time is of five minutes.

So let’s try seeing the pod status again.

controlplane $ kubectl get pods
NAME     READY   STATUS             RESTARTS   AGE
my-pod   0/1     ImagePullBackOff   0          2m43s

We can get more information about the issues and retries by kubectl decsribe pod/<your-podname> .

controlplane $ kubectl describe pod/my-pod 
Name:             my-pod
Namespace:        default
--------------------------
Status:           Pending
IP:               192.168.1.3
IPs:
  IP:  192.168.1.3
Containers:
  my-pod:
    Container ID:   
    Image:          nginxxx
    Image ID:       
    Port:           <none>
    Host Port:      <none>
    State:          Waiting
      Reason:       ImagePullBackOff
    Ready:          False
---------------------------
Events:
  Type     Reason     Age                    From               Message
  ----     ------     ----                   ----               -------
  Normal   Scheduled  5m18s                  default-scheduler  Successfully assigned default/my-pod to node01
  Normal   Pulling    3m26s (x4 over 5m18s)  kubelet            Pulling image "nginxxx"
  Warning  Failed     3m20s (x4 over 5m12s)  kubelet            Failed to pull image "nginxxx": rpc error: code = Unknown desc = failed to pull and unpack image "docker.io/library/nginxxx:latest": failed to resolve reference "docker.io/library/nginxxx:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
  Warning  Failed     3m20s (x4 over 5m12s)  kubelet            Error: ErrImagePull
  Warning  Failed     3m5s (x6 over 5m11s)   kubelet            Error: ImagePullBackOff
  Normal   BackOff    4s (x19 over 5m11s)    kubelet            Back-off pulling image "nginxxx"

The events section shows details about the issue that happened.

Common causes for ErrImagePull and ImagePullBackOff

• Pod Spec provides the wrong registry name- resolve it by editing the pod specification and providing the correct registry

• Pod Spec mentioned the wrong image name — we can resolve it by providing the correct image in the pod spec.

• Invalid image tag in Pod Spec — we can resolve it by giving the correct image tag. If not provided, by default it will be labeled as the latest. If the image doesn’t contain the latest tag, you must provide a valid tag.

• Authentication issue while accessing registry — Resolve it by providing correct secrets to the image registry. Add the secret reference in the Pod spec.

• The container registry is not accessible- we can fix the issue by restoring the network connection and allowing the pod to retry pulling the image.

• And those are just some of there might be more reasons why kubelet is not able to pull the image.

For troubleshooting these errors, your best friend is kubectl describe pod/<yourpodname> .

---

That’s a wrap!!!

Hope you learned something. Let me know your feedback and if you have any suggestions or questions, you can connect with me on Twitter.

Till next,

Happy kubectl’ing

Comments

[Kishore S]

Events:
Type Reason Age From Message

---

Normal Scheduled 2m54s default-scheduler Successfully assigned default/facimateimg-647d4d5cdd-h7c46 to gke-perfectio-default-pool-13914582-fj5k
Normal Pulling 91s (x4 over 2m54s) kubelet Pulling image "facitest/facimateimg:v1"
Warning Failed 90s (x4 over 2m53s) kubelet Failed to pull image "facitest/facimateimg:v1": failed to pull and unpack image "docker.io/facitest/facimateimg:v1": failed to resolve reference "docker.io/facitest/facimateimg:v1": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 90s (x4 over 2m53s) kubelet Error: ErrImagePull
Warning Failed 68s (x6 over 2m53s) kubelet Error: ImagePullBackOff
Normal BackOff 54s (x7 over 2m53s) kubelet Back-off pulling image "facitest/facimateimg:v1"