Pod statuses like ImagePullBackOff or ErrImagePull are common when working with containers.
ErrImagePull is an error happening when the image specified for a container can’t be retrieved or pulled.
ImagePullBackOff is the waiting grace period while the image pull is fixed.
In this article, we will take a look at:
- Container Images
- Pulling Images
- Image Pull Policy
- ErrImagePull
- Debugging ErrImagePull
- Monitoring Image Pull Errors
- Other Image Errors
Container Images
One of the greatest strengths of containerization is the ability to run any particular image in seconds. A container is a group of processes executing in isolation from the underlying system. A container image contains all the resources needed to run those processes: the binaries, libraries, and any necessary configuration.
A container registry is a repository for container images, where there are two basic actions:
- Push: upload an image so it’s available in the repo
- Pull: download an image to use it in a container
The docker CLI will be used in the examples for this article, but you can use any tool that implements the Open Container Initiative Distribution specs for all the container registry interactions.
Pulling images
Images can be defined by name. In addition, a particular version of an image can be labeled with a specific name or tag. It can also be identified by its digest, a hash of the content.
The tag latest refers to the most recent version of a given image.
Pull images by name
By only providing the name for the image, the image with tag latest will be pulled
docker pull nginx kubectl run mypod nginx
Pull images by name + tag
If you don’t want to pull the latest image, you can provide a specific release tag:
docker pull nginx:1.23.1-alpine kubectl run mypod nginx:1.23.1-alpine
For more information, you can check this article about tag mutability.
Pull images by digest
A digest is sha256 hash of the actual image. You can pull images using this digest, then verify its authenticity and integrity with the downloaded file.
docker pull sha256:d164f755e525e8baee113987bdc70298da4c6f48fdc0bbd395817edf17cf7c2b kubectl run mypod --image=nginx:sha25645b23dee08af5e43a7fea6c4cf9c25ccf269ee113168c19722f87876677c5
Image Pull Policy
Kubernetes features the ability to set an Image Pull Policy (imagePullPolicy field) for each container. Based on this, the way the kubelet retrieves the container image will differ.
There are three different values for imagePullPolicy:
- Always
- IfNotPresent
- Never
Always
With imagePullPolicy set to Always, kubelet will check the repository every time when pulling images for this container.
IfNotPresent
With imagePullPolicy set to IfNotPresent, kubelet will only pull images from the repository if it doesn’t exist in the node locally.
Never
With imagePullPolicy set to Never, kubelet will never try to pull images from the image registry. If there’s an image cached locally (pre-pulled), it will be used to start the container.
If the image is not present locally, Pod creation will fail with ErrImageNeverPull error.
Note that you can modify the entire image pull policy of your cluster by using the AlwaysPullImages admission controller.
Default Image Pull Policy
- If you omit the imagePullPolicy and the tag is
latest, imagePullPolicy is set toAlways. - If you omit the imagePullPolicy and the tag for the image, imagePullPolicy is set to
Always. - If you omit the imagePullPolicy and the tag is set to a value different than
latest, imagePullPolicy is set toIfNotPresent.
ErrImagePull
When Kubernetes tries to pull an image for a container in a Pod, things might go wrong. The status ErrImagePull is displayed when kubelet tried to start a container in the Pod, but something was wrong with the image specified in your Pod, Deployment, or ReplicaSet manifest.
Imagine that you are using kubectl to retrieve information about the Pods in your cluster:
$ kubectl get pods NAME READY STATUS RESTARTS AGE goodpod 1/1 Running 0 21h mypod 0/1 ErrImagePull 0 4s
Which means:
- Pod is not in
READYstatus - Status is
ErrImagePull
Additionally, you can check the logs for containers in your Pod:
$ kubectl logs mypod --all-containers Error from server (BadRequest): container "mycontainer" in pod "mypod" is waiting to start: trying and failing to pull image
In this case, this is pointing to a 400 Error (BadRequest), since probably the image indicated is not available or doesn’t exist.
ImagePullBackOff
ImagePullBackOff is a Kubernetes waiting status, a grace period with an increased back-off between retries. After the back-off period expires, kubelet will try to pull the image again.
This is similar to the CrashLoopBackOff status, which is also a grace period between retries after an error in a container. Back-off time is increased each retry, up to a maximum of five minutes.
Note that ImagePullBackOff is not an error. As mentioned, it’s just a status reason that is caused by a problem when pulling the image.
$ kubectl get pods NAME READY STATUS RESTARTS AGE goodpod 1/1 Running 0 21h mypod 0/1 ImagePullBackOff 0 84s
Which means:
- Pod is not in
READYstatus - Status is
ImagePullBackOff - Unlike CrashLoopBackOff, there are no restarts (technically Pod hasn’t even started)
$ kubectl describe pod mypod State: Waiting Reason: ImagePullBackOff ... Warning Failed 3m57s (x4 over 5m28s) kubelet Error: ErrImagePull Warning Failed 3m42s (x6 over 5m28s) kubelet Error: ImagePullBackOff Normal BackOff 18s (x20 over 5m28s) kubelet Back-off pulling image "failed-image"
Debugging ErrImagePull and ImagePullBackOff
There are several potential causes of why you might encounter an Image Pull Error. Here are some examples:
- Wrong image name
- Wrong image tag
- Wrong image digest
- Network problem or image repo not available
- Pulling from a private registry but not imagePullSecret was provided
This is just a list of possible causes, but it’s important to note that there might be many others based on your solution. The best course of action would be to check:
$ kubectl describe pod podname $ kubect logs podname –all-containers $ kubectl get events --field-selector involvedObject.name=podname
In the following example you can see how to dig into the logs, where an image error is found.
Other image errors
ErrImageNeverPull
This error appears when kubelet fails to pull an image in the node and the imagePullPolicy is set to Never. In order to fix it, either change the Pull Policy to allow images to be pulled externally or add the correct image locally.
Pending
Remember that an ErrImagePull and the associated ImagePullBackOff may be different from a Pending status on your Pod.
Pending status, most likely, is the result of kube-scheduler not being able to assign your Pod to a working or eligible Node.
Monitoring Image Pull Errors in Prometheus
Using Prometheus and Kube State Metrics (KSM), we can easily track our Pods with containers in ImagePullBackOff or ErrImagePull statuses.
kube_pod_container_status_waiting_reason{reason="ErrImagePull"}
kube_pod_container_status_waiting_reason{reason="ImagePullBackOff"}
In fact, these two metrics are complementary, as we can see in the following Prometheus queries:
The Pod is shifting between the waiting period in ImagePullBackOff and the image pull retrial returning an ErrImagePull.
Also, if you’re using containers with ImagePullPolicy set to Never, remember that you need to track the error as ErrImageNeverPull.
kube_pod_container_status_waiting_reason{reason="ErrImageNeverPull"}
Conclusion
Container images are a great way to kickstart your cloud application needs. Thanks to them, you have access to thousands of curated applications that are ready to be started and scaled.
However, due to misconfiguration, misalignments, or repository problems, image errors might start appearing. A container can’t start properly if the image definition is malformed or there are errors on the setup.
Kubernetes provides a graceful period in case of an image pull error. This Image Pull Backoff is quite useful, as it gives you time to fix the problem in the image definition. But you need to be aware when this happens in your cluster and what does it mean each time.
Troubleshoot Image Pull Errors with Sysdig Monitor
With the Advisor feature of Sysdig Monitor, you can easily review Image Pull Errors happening in your Kubernetes cluster.
Sysdig Advisor accelerates mean time to resolution (MTTR) with live logs, performance data, and suggested remediation steps. It’s the easy button for Kubernetes troubleshooting!
Top comments (0)