Kubernetes `cronjob` to refresh .dockerconfig secret

A lot is taken from https://linuxhandbook.com/auto-update-aws-ecr-token-kubernetes/, with few modifications here and there.

Overview

While AWS ECR offers a secure way to store container images, its authentication system relies on tokens that expire every 12 hours. This frequent renewal can be cumbersome if we want to use images stored in AWS ECR for our deployment outside the AWS network where the AWS IAM Role is not accessible.

This post proposes a solution to automate the AWS ECR authentication token generation. The solution will be divided into areas:

• Preparations: collect all the necessary information, and store it as environment variables.
• Create resource: Create all the resources needed (including the Cronjob), by using the information collected earlier.
• Test and verify: if the CronJob working as expected, and the resulting token can be used to access the AWS ECR.

Once CronJob installed, triggered and completed successfully, it will (re)create a (new) secret regcred-aws (type kubernetes.io/dockerconfigjson) within the targeted namespace. Once the secret available, it can be referred from kind: deployment as spec.template.spec.imagePullSecrets, or from kind: pod as spec.imagePullSecrets

Preparations

1. The ECRPULL user referred in the following step, are configured with the following permissions:
   

   {
       "Version": "2012-10-17",
       "Statement": [
           {
               "Sid": "AllowPull",
               "Effect": "Allow",
               "Action": [
                   "ecr:BatchGetImage",
                   "ecr:GetDownloadUrlForLayer",
                   "ecr:GetAuthorizationToken"
               ],
               "Resource": [
                   "*"
               ]
           }
       ]
   }
   

2. Acquire the AWS credential by capturing the output of command aws iam create-access-key --user-name ECRPULL. The output may look similar to this:
   

   {
      "AccessKey": {
          "UserName": "ECRPull",
          "AccessKeyId": "AAAAAAAAAAAAAAAAAAAA",
          "Status": "Active",
          "SecretAccessKey": "BBBBBBBBBBBBBBBBBBBBBBBBBBBB",
          "CreateDate": "2024-01-01T00:00:00+00:00"
      }
   }
   

3. Create enviroment variables as follow based on the output above.
   

   export AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAAAA
   export AWS_SECRET_ACCESS_KEY=BBBBBBBBBBBBBBBBBBBBBBBBBBBB
   export AWS_ACCOUNT=`aws sts get-caller-identity --query='Account' --output text`
   export AWS_REGION="ap-southeast-3"
   export APP_NAMESPACE="TargetNameSpace" # Namespace to keep the generated secret
   

Create Resources

1. Create secret ecr-registry-helper-secrets to store the AWS credentials.
   

   kubectl create secret generic ecr-registry-helper-secrets \
      --namespace=${APP_NAMESPACE} \
      --from-literal=AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
      --from-literal=AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
      --from-literal=AWS_ACCOUNT=${AWS_ACCOUNT}
   

2. Create configmap & service-account:
   

   cat <<EOL | kubectl apply -f -
   apiVersion: v1
   kind: ConfigMap
   metadata:
     name: ecr-registry-helper-cm
     namespace: ${APP_NAMESPACE}
   data:
     AWS_REGION: ${AWS_REGION}
     DOCKER_SECRET_NAME: regcred-aws
   ---
   apiVersion: v1
   kind: ServiceAccount
   metadata:
     name: ecr-registry-helper-sa-${APP_NAMESPACE}
     namespace: ${APP_NAMESPACE}
   EOL
   

3. Create the role and its binding in the cluster
   

   cat <<EOL | kubectl apply -f -
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     namespace: ${APP_NAMESPACE}
     name: role-manage_regcred-aws_secrets
   rules:
   - apiGroups: [""]
     resources: ["secrets"]
     resourceNames: ["regcred-aws"] # Replace with your desired ECR token secret name
     verbs: ["delete"]
   - apiGroups: [""]
     resources: ["secrets"]
     verbs: ["create"]
   ---
   kind: RoleBinding
   apiVersion: rbac.authorization.k8s.io/v1
   metadata:
     name: ${APP_NAMESPACE}-role-binding-ecr-registry-helper
     namespace: ${APP_NAMESPACE}
   subjects:
   - kind: ServiceAccount
     name: ecr-registry-helper-sa-${APP_NAMESPACE} # Replace with your service account name if different
     namespace: ${APP_NAMESPACE}
     apiGroup: ""
   roleRef:
     kind: Role
     name: role-manage_regcred-aws_secrets # Replace with your role name if different 
     apiGroup: ""
   EOL
   

4. Deploy the cronjob, that will execute itself every 8th hours since the script installed (schedule: "0 */8 * * *").
   

   cat <<EOL | kubectl apply -f -
   apiVersion: batch/v1
   kind: CronJob
   metadata:
     name: regcred-aws
     namespace: ${APP_NAMESPACE}
   spec:
     schedule: "0 */8 * * *"
     jobTemplate:
       spec:
         template:
           spec:
             serviceAccountName: ecr-registry-helper-sa-${APP_NAMESPACE}
             volumes:
             - name: data
               emptyDir:
                 medium: Memory
             initContainers:
             - name: getecr-registry-helper-get-token
               image: amazon/aws-cli
               imagePullPolicy: IfNotPresent
               volumeMounts:
               - mountPath: /data
                 name: data
               envFrom:
               - secretRef:
                   name: ecr-registry-helper-secrets
               - configMapRef:
                   name: ecr-registry-helper-cm
               command:
                 - /bin/bash
                 - -c
                 - |-
                   aws ecr get-login-password --region \${AWS_REGION} > /data/token
             containers:
             - name: ecr-registry-helper-get-docker-registry-credentials
               image: bitnami/kubectl
               imagePullPolicy: IfNotPresent
               volumeMounts:
               - mountPath: /data
                 name: data
               envFrom:
                 - secretRef:
                     name: ecr-registry-helper-secrets
                 - configMapRef:
                     name: ecr-registry-helper-cm
               command:
                 - /bin/bash
                 - -c
                 - |-
                   ECR_TOKEN=\$(cat /data/token)
                   NAMESPACE_NAME=${APP_NAMESPACE}
                   kubectl delete secret --ignore-not-found \${DOCKER_SECRET_NAME} -n \${NAMESPACE_NAME}
                   kubectl create secret docker-registry \${DOCKER_SECRET_NAME} --docker-server=https://\${AWS_ACCOUNT}.dkr.ecr.\${AWS_REGION}.amazonaws.com --docker-username=AWS --docker-password=\${ECR_TOKEN} --namespace=\${NAMESPACE_NAME}
                   echo -n "Secret was successfully updated at "; date
             restartPolicy: Never
   EOL
   

Test and verify

Run the first job interactively for testing. The last command output should show string resemblence the ~/.docker/config file, which confirm our cronjob is correctly configured.

kubectl create job --from=cronjob/regcred-aws regcred-aws-manual --namespace=${APP_NAMESPACE}
kubectl describe jobs.batch regcred-aws-manual --namespace=${APP_NAMESPACE}
kubectl get secret regcred-aws --namespace=${APP_NAMESPACE} -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d | jq