Authenticated Docker Hub image pulls in Kubernetes

I recently stumbled over the Docker Hub image pull rate limit in one of my Kubernetes clusters. A pod failed to start due to being unable to pull an image, with a 429 Too Many Requests error response. The Docker Hub documentation says:

For anonymous users, the rate limit is set to 100 pulls per 6 hours per IP address. For authenticated users, it’s 200 pulls per 6 hour period. Users with a paid Docker subscription get up to 5000 pulls per day.

With a small cluster that doesn't change frequently, the rate limit is typically not an issue. However, as your cluster grows and pods are started or replaced more frequently, the likelihood of image pulls failing due to hitting the rate limit increases.

So, what can be done to avoid this? There are a few options:

1. Authenticate your Docker Hub image pulls. This seems like the obvious answer, but as we will discuss, this can be more complex than you might expect.
2. Operate a pull-through cache registry, like Artifactory or the open source reference Docker registry. This will allow you to pull images from Docker Hub less frequently, improving your chances of staying under the anonymous usage limit.
3. Use images from repositories directly controlled by your organization. For example, you could exclusively use images stored in a registry provided by your cloud provider (e.g. AWS Elastic Container Registry).

Options 2 and 3 are worth considering for reasons beyond the scope of this article - reduced data transfer fees, more visibility into the images you deploy, and better options for mitigating supply chain attacks. However, they are not always practical options - the overheads of configuring, operating, and monitoring a private registry can be substantial. Additionally, you will likely need to change all of your image references - a default image reference like busybox:1.36 is implicitly referencing Docker Hub, and would need to be changed to something else like my-image-registry.example/busybox:1.36. If you are using Helm charts to manage the install of common services in your cluster, such overrides are not always possible - the chart may hard-code image references.

So, how can we authenticate our Docker Hub image pulls? If you have control over the underlying operating system of your Kubernetes nodes (e.g. through a custom virtual machine image, or cloud-init configuration), you can provide Docker Hub credentials directly in the containerd registry configuration. It may also be possible to use a kubelet credential provider, though this interface is primarily designed for dynamic credential generation or retrieval, whereas Docker Hub credentials are currently static. I could not find a credential provider for this interface that could supply either static credentials or those sourced from a credential vault.

Staying within the bounds of what Kubernetes offers at the conceptual layer, we can declaratively configure authenticated image pulls using image pull secrets. We will go through the details of this approach, then discuss some of the complexities that arise in larger clusters.

Creating and using image pull secrets

Creating a Docker Hub credential

First, we need a Docker Hub user and password for pulling images. I recommend creating a Docker Hub account specifically for this purpose, separate from that of any specific person in your organization. This will allow you independently manage the lifecycle of this account and its security. If you have a Docker organization, it is best to create this "service account" under that organization, which has the added benefit of giving the account a significantly higher image pull rate limit (16x what you get with unauthenticated pulls). If you need even more, and you still don't want to operate a pull-through cache or private registry, you can pay Docker for even higher limits if needed.

Within your chosen account, you can create a personal access token that can be used as the "password" for authenticated image pulls. I recommend configuring this token to be "Read-only" or "Public Repo Read-only" to limit exposure. A "Read-only" token will allow pulls of images from private Docker Hub repositories that the account has access to, which may be desirable if you are also using Docker Hub as your primary store for private images. If you only intend to use public images from Docker Hub, "Public Repo Read-only" is sufficient.

Creating an image pull secret

To use the personal access token from your Docker Hub account for image pulls in a Kubernetes cluster, we must create a secret object with type kubernetes.io/dockerconfigjson to hold the credentials. The credentials are embedded in a JSON object with the following structure:

{
  "auths": {
    "https://index.docker.io/v1/": {
      "username": "my-robot-account-1234",
      "password": "dckr_pat_asd-fghjklqwertyuiopZXCVBNM"
    }
  }
}

The secret object embeds the Base64 encoded form of that JSON object. It will look something like:

apiVersion: v1
kind: Secret
metadata:
  name: dockerhub-image-pull-secret
  namespace: default
type:  kubernetes.io/dockerconfigjson
data:
  .dockerconfigjson: |
    ewogICJhdXRocyI6IHsKICAgICJodHRwczovL2luZGV4LmRvY2tlci5pby92  
    MS8iOiB7CiAgICAgICJ1c2VybmFtZSI6ICJteS1yb2JvdC1hY2NvdW50LTEy  
    MzQiLAogICAgICAicGFzc3dvcmQiOiAiZGNrcl9wYXRfYXNkLWZnaGprbHF3  
    ZXJ0eXVpb3BaWENWQk5NIgogICAgfQogIH0KfQo=

It is important to note that secrets are namespaced. This means they can only be referenced by other Kubernetes resources in the same namespace, unless you set up specific role-based access control rules to allow cross-namespace access to the secret.

Using an image pull secret

With an image pull secret defined, we have two main options for using it. First, we can reference the secret in our pod specifications under the imagePullSecrets field:

apiVersion: v1
kind: Pod
metadata:
  name: nginx
  namespace: default
spec:
  containers:
    - name: nginx
      image: nginx:1.23.4
  imagePullSecrets:
    - name: dockerhub-image-pull-secret

However, it can be tedious and error-prone to specify this reference across many different pod specifications. The second option is to reference the secret as part of a service account object:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: default
  namespace: default
imagePullSecrets:
  - name: dockerhub-image-pull-secret

Unless explicitly changed, pods reference the default service account of their namespace. So, this service account acts as a shared location to define the image pull secrets.

Problems at scale

Configuring and using a Docker Hub image pull secret for a single namespace is relatively straightforward. However, repeating this work for tens or hundreds of namespaces is tedious and error prone:

• You must clone the secret into every namespace, or ensure that one secret is accessible in all namespaces. Both require per-namespace configuration, and if you are dynamically creating namespaces, you will need associated automation.
• You must reference the secret in all relevant places, whether those are pod specifications or service accounts.

Fortunately, tools exist that can help automate these tasks.

imagepullsecret-patcher

The problem of using image pull secrets in larger clusters has been known for quite some time. TitanSoft decided to do something about it in 2019, releasing the imagepullsecret-patcher tool. This executes within your cluster and does two things every 10 seconds for every namespace:

• Checks to see if an image pull secret exists; if it does not exist or has stale contents, it is cloned from a primary secret.
• Checks to ensure the default service account has an imagePullSecrets reference to the cloned secret in that namespace. If it does not, the service account is patched to include the reference. This can also be optionally applied to all service accounts, not just the default service account.

This does exactly what we would want, in the absence of a more official mechanism provided by Kubernetes itself. It appears to have worked well for many people, at least based on the popularity of the GitHub repository. However, there are some risks:

• The tool requires cluster-wide read-write access to all secrets and service accounts, via a ClusterRole and ClusterRoleBinding. Any potential vulnerabilities in the tool could be leveraged to gain access to all of your secrets, including service account secrets, which may provide access into other parts of your infrastructure (e.g. via tokens issued by AWS IAM Roles for Service Accounts or equivalents in other cloud providers).
• The tool has not been updated since October 2020 - this isn't necessarily an issue, as what the tool does is relatively simple. However, it does mean that the last release is compiled against relatively old versions of the Go standard library and other dependencies. This increases the risk that there are some known vulnerabilities via those dependencies that could be exploited.

The tool is simple and effective, and at only ~1k lines of Go code, it is entirely feasible for a small devops team to maintain a fork for updates or tweaks if desired. For my purposes, I was interested to see if other tools existed that are more actively maintained and could solve the same problems.

Cluster-wide secrets with External Secrets Operator

When you are dealing with larger clusters, it is also likely that your organization is using a centralized secret store like Hashicorp Vault, or cloud-specific solutions like AWS Secrets Manager. If you are doing this, you may also be using the external secrets operator to import secrets from those environments into Kubernetes. This operator supports defining a ClusterExternalSecret, which allows an external secret to be imported into multiple namespaces. The definition will look something like:

apiVersion: external-secrets.io/v1beta1
kind: ClusterExternalSecret
metadata:
  name: dockerhub-image-pull-secret
spec:
  # instantiate the secrets in _every_ namespace
  namespaceSelector: {}
  externalSecretSpec:
    secretStoreRef:
      name: cluster-secret-store
      kind: ClusterSecretStore
    target:
      template:
        type: kubernetes.io/dockerconfigjson
        data:
          .dockerconfigjson: |
            {
              "auths": {
                "https://index.docker.io/v1": {
                  "username": "{{ .username }}"
                  "password": "{{ .password }}"
                }
              }
            }
        dataFrom:
          - extract:
              key: dockerhub-account

This definition imports a secret with external name "dockerhub-account" from a ClusterSecretStore. We extract from this secret the fields "username" and "password", and inject those values into the expected image pull secret structure using a template. The external secrets operator will create an ExternalSecret with the same name in every namespace (as they all implicitly match the namespaceSelector). Those ExternalSecrets will produce Secrets with the same name and the rendered template. The end result is that a Secret named dockerhub-image-pull-secret will exist in every namespace, ready to be referenced as needed.

Additional configuration may be required for your specific environment and needs - see the documentation, and in particular consider changing the default refreshInterval - the default interval is one hour. While your Docker Hub credentials are not likely to change frequently, you may wish to ensure that when you change it in your central system that it propagates to your clusters quickly and without manual intervention.

Patching service accounts with RedHat's patch-operator

The general problem of patching resource definitions that are not fully under your control has also been recognized for some time. This is true of default resources created and updated by cluster maintenance tools (e.g. kOps), or by public helm charts that you use to install common services and operators (e.g. nginx-ingress, cert-manager, and so on). High quality charts will allow you to override the configuration of important components such as service account references, but some simpler charts offer much less configuration.

Red Hat's patch-operator is designed to allow you to declare patches to target resources in your cluster. We can use this to patch service accounts to include references to our image pull secrets:

apiVersion: redhatcop.redhat.io/v1alpha1
kind: Patch
metadata:
  name: dockerhub-image-pull-secret-patch
  namespace: patches
spec:
  serviceAccountRef:
    name: patching-service-account
  patches:
    service-account-patch:
      targetObjectRef: 
        apiVersion: v1
        kind: ServiceAccount
      patchType: application/strategic-merge-patch+json
      patchTemplate: |
        imagePullSecrets:
          - name: dockerhub-image-pull-secret

This Patch custom resource definition declares that we would like to add the dockerhub-image-pull-secret reference to the imagePullSecrets list of all service accounts. The targetObjectRef is not restricted to a particular namespace or name, so it will match all service accounts (as described in the documentation for the operator).

This, combined with external secrets operator to produce the secrets we need in each namespace, allows us to configure image pull secrets across all service accounts. The permissions required to apply this patch are attached to the referenced service account, patching-service-account. This allows us to isolate different permission sets for different patches. For this patch, we need:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: patching-service-account
  namespace: patches
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: service-account-modifier
rules:
  - apiGroups: [""]
    resources: ["serviceaccounts"]
    verbs: ["get", "watch", "list", "update", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: service-account-modifier-binding
subjects:
  - kind: ServiceAccount
    name: patching-service-account
    namespace: patches
roleRef:
  kind: ClusterRole
  name: service-account-modifier
  apiGroup: rbac.authorization.k8s.io

There is one significant issue with this approach, however: there is no declared patch strategy for imagePullSecrets on service accounts. Without this, the default behavior is to replace the list - so if you had any existing image pull secret references in your service account, these would be removed. See this kubernetes GitHub issue from 2019 that describes the problem in more detail, and why it has not been fixed (tl;dr: specifying a patch strategy will break backwards compatibility, and there has not yet been any desire to introduce a v2 of the ServiceAccount object kind, so we're stuck with the behavior).

In my situation, I did not have any service accounts with specifically configured lists of image pull secrets, so the patch is replacing an empty list in every service account with the single reference in the patch. However, the situation in your cluster may differ, and you may want to change the patch targetObjectRef to target a more specific set of service accounts, such as just the default service accounts. It is also possible to use a labelSelector or annotationSelector with a matchExpressions list to avoid modifying service accounts with specific labels or annotations, e.g.

labelSelector:
  matchExpressions:
    key: do-not-patch-image-pull-secrets
    operator: DoesNotExist

This selector will exclude any service account with a label do-no-patch-image-pull-secrets, which you could specifically add to the service accounts that the patch would break. This would have to be communicated to all engineers that define service accounts in your cluster.

Conclusion

Managing authenticated image pulls to Docker Hub in a large cluster is surprisingly difficult. It was likely not anticipated that Docker Hub would introduce such strict rate limits for unauthenticated requests in November 2020 - this now essentially requires that all cluster operators know how to configure authenticated image pulls, as you will need these whether you continue to use Docker Hub or migrate to a pull-through cache registry or privately managed registry with clones of your essential images.

With control over the virtual machines that your kubelets run on, it is possible to configure the necessary credentials for authenticated image pulls to Docker Hub via the containerd config, or by writing a custom credential provider for your kubelets. This can avoid a lot of additional configuration in your Kubernetes resources, but is not necessarily a desirable or available option for all cluster operators. The alternative is to create and use image pull secrets via service accounts.

TitanSoft's imagepullsecret-patcher is a single-binary solution to replicating and using an image pull secret across all namespaces. It is not actively maintained, but the tool is simple enough that a small team should be able to patch and maintain a fork if needed. If you want to stick to other maintained open source tools, a reasonable solution can also be put together using external secrets operator. If you are operating a cluster at scale, you may already be using this. Red Hat's patch-operator can be used to attach the imported secrets to your service accounts across all namespaces, though there are some quirks to be wary of, due to the lack of a defined patch strategy for imagePullSecrets on service accounts.

Comments

[Eric Horwath]

External secrets operator sounds like a good solution for the ever lasting image pull secrets problem! Often, I see people abusing OPA/Kyverno for this, which somehow does not feel right...
As many other teams, we ended up implementing our own solution: github.com/sap/clustersecret-operator