How to overcome Docker Hub rate limiting using AWS ECR and AWS CloudFormation

In this blog post, you will see how AWS ECR and AWS CloudFormation overcome the rate limiting imposed by Docker Hub and provide full control over your base images.

The popular registry Docker Hub is home to thousands of useful container images, used by many software delivery processes. Unfortunately, the registry enforces a rate limit for anonymous and free-tier users. Whenever you try to pull an image from an AWS CodeBuild project, you immediately run it this problem. AWS offers many Docker Hub images directly from their public AWS ECR registry https://public.ecr.aws, but not all of them. So, when you want to use a public image not on offer, you are stuck.

Overcome Docker Hub rate limiting

Our Custom CloudFormation Container Image Provider offers an effective workaround by allowing you to clone public images into a private Amazon Elastic Container Registry repository. By leveraging the custom provider you use CloudFormation to avoid the rate limit imposed by Docker Hub. Once the image is copied, you can pull the image as often as you want from your own ECR repository. This ensures a smooth and uninterrupted development process.

Full control over updating base images

Another advantage of using the Custom CloudFormation Container Image Provider is that you gain complete control over the base images. You can enable container image scanning and see which vulnerabilities live inside the public image. By using a CloudFormation template you specify the exact image version you want.

The container reference update utility – cru can be used to updates image references in the CloudFormation template and trigger the provisioning of the latest version to your ECR repository.

This effectively gives you a well defined process for provisioning container images.

Example usage

To demonstrate the usage of the Custom CloudFormation Container Image Provider, let’s consider the following CloudFormation template:

Resources:
  Repository:
    Type: AWS::ECR::Repository
    Properties:
      RepositoryName: python

  Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: docker.io/library/python:3.7
      RepositoryArn: !GetAtt Repository.Arn
      ServiceToken: !Sub 'arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:cfn-container-image-provider'

In this example, we clone the current repository from the public image ‘python:3.7’ into our ‘python’ repository in ECR. The ‘Repository’ resource creates the ECR repository, and the ‘Python37’ resource uses the custom resource ‘Custom::ContainerImage’ to clone the image.

Updating the image reference

To pin the image to a specific version, you can use the container reference update utility – cru as follows:

$ cru update \
        --resolve-digest --all \
        --matching-tag \
        demo.yaml

023/10/07 16:20:56 INFO: 1 image references found
2023/10/07 16:20:57 resolving repository docker.io/library/python Tag 3.7 to Digest sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0
2023/10/07 16:20:57 INFO: updated a total of 1 files
2023/10/07 16:20:57 INFO: no commit message, skipping commit and push

Now the container image reference will have the associated digest of the image, so you know exactly which image is used.

Python37:
    Type: 'Custom::ContainerImage'
    Properties:
      ImageReference: 'docker.io/library/python:3.7@sha256:eedf63967cdb57d8214db38ce21f105003ed4e4d0358f02bedc057341bcf92a0'

Installing the provider

To install this custom resource provider, type:

read -p 'VPC id:' VPC_ID
read -p 'private subnet ids (comma separated):' PRIVATE_SUBNET_IDS
read -p 'security group ids (comma separated):' SECURITY_GROUP_IDS

aws cloudformation create-stack \
       --capabilities CAPABILITY_IAM \
       --stack-name cfn-container-image-provider \
       --template-url s3://binxio-public-eu-central-1/lambdas/cfn-container-image-provider-0.2.4.yaml \
       --parameter-overrides \
          ParameterKey=AppVPC,ParameterValue=$VPC_ID \
          ParameterKey=Subnets,ParameterValue=$PRIVATE_SUBNET_IDS \
          ParameterKey=SecurityGroupIds,ParameterValue=$SECURITY_GROUP_IDS

aws cloudformation wait stack-create-complete \
       --stack-name cfn-container-image-provider

or use launch stack.

The provider is installed on the private subnets in your VPC, to ensure that your NAT gateway IP addresses are used to pull images from docker hub.

Conclusion

The Custom CloudFormation Container Image Provider addresses two important challenges that developers and organisations face when working with container images. By cloning public images into your ECR repository, you can overcome the rate limit imposed by Docker Hub, and ensure uninterrupted access to the images you need. Additionally, you gain full control over which images are used in your organisation.

---

Photo by Zé Maria on Unsplash

The post How to overcome Docker Hub rate limiting using AWS ECR and AWS CloudFormation appeared first on Xebia.