DEV Community

Cover image for How to Speed Up AWS CodeBuild Docker Builds by 25% or more Using ECR as a Remote Cache
AWS Community Builders profile image Yusuf Adeyemo
Yusuf Adeyemo for AWS Community Builders

Posted on • Originally published at blog.yusadolat.me on

How to Speed Up AWS CodeBuild Docker Builds by 25% or more Using ECR as a Remote Cache

#aws #docker #caching #ecr

Have you ever sat there waiting for your CodeBuild project to rebuild your entire Docker image... again? Even though you only changed a single line of code?

Yeah, me too. And it's frustrating.

Today, I'm going to show you how I reduced our Docker build times from ~7 minutes down to ~2 minutes by implementing Amazon ECR as a persistent cache backend. This is based on an official AWS blog post, but I'll walk you through the practical implementation.

The Problem: Why Your Builds Are Slow

Here's the thing about AWS CodeBuild: every build runs in a completely fresh, isolated environment. That means:

  • No build artifacts carry over between builds

  • Every build starts from scratch

  • CodeBuild's "local cache" is temporary and unreliable (works on a "best-effort" basis)

  • If your builds happen at different times throughout the day, the local cache probably isn't helping you

So even if you only changed one line in your code, CodeBuild rebuilds every single Docker layer. Every. Single. Time.

The Solution: ECR Registry Cache Backend

The solution is surprisingly elegant: store your Docker layer cache persistently in Amazon ECR (Elastic Container Registry). Think of it as a separate "cache image" that lives alongside your actual application image.

Here's how it works:

  1. First Build : Build from scratch, then export the cache to ECR as a separate image

  2. Subsequent Builds : Import the cache from ECR, rebuild only what changed, export the updated cache back

The beauty? Your cache is always available, no matter when you trigger a build.

What You'll Need

Before we start, make sure you have:

  • An existing AWS CodeBuild project that builds Docker images

  • An ECR repository where your images are stored

  • IAM permissions for your CodeBuild role to push/pull from ECR (if you can already push images, you're good!)

  • About 10 minutes to implement this

Step-by-Step Implementation

Step 1: Understanding Your Current Buildspec

Your current buildspec probably looks something like this:

version: 0.2phases: install: commands: - aws ecr get-login-password | docker login ... build: commands: - docker build -t myapp:latest . - docker tag myapp:latest $ECR_REPO:latest post_build: commands: - docker push $ECR_REPO:latest
Enter fullscreen mode Exit fullscreen mode

This is the "basic" approach. Every build starts from zero.

[📸 IMAGE SUGGESTION: Split screen showing "Basic Build" vs "Cached Build" with layer rebuilding visualization]

Step 2: Add Cache Tag Variable

First, let's define a separate tag for our cache image. In your install phase, add:

install: commands: - CACHE_TAG=dev-cache # or prod-cache, staging-cache, etc. - IMAGE_TAG=latest # your actual app image tag
Enter fullscreen mode Exit fullscreen mode

This creates a separate cache image (e.g., myapp:dev-cache) that's distinct from your application image (myapp:latest).

Step 3: Create the Buildx Builder

Here's the key part: Docker's default builder doesn't support registry cache backends. We need to create a new builder using buildx with the containerd driver.

Add this to your install phase:

install: commands: # ... your existing commands ... - docker buildx create --name containerd --driver=docker-container --driver-opt default-load=true --use || docker buildx use containerd
Enter fullscreen mode Exit fullscreen mode

What's happening here?

  • docker buildx create: Creates a new builder instance

  • --driver=docker-container: Uses containerd (required for registry cache)

  • --driver-opt default-load=true: Loads built images into local Docker (important!)

  • || docker buildx use containerd: If the builder already exists, just switch to it

Step 4: Replace Your Docker Build Command

Now replace your regular docker build command with the new buildx version:

build: commands: - | docker buildx build \ --builder=containerd \ --cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG \ --cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true \ -t $ECR_REPO:$IMAGE_TAG \ --load \ .
Enter fullscreen mode Exit fullscreen mode

Let me break down what each flag does:

  • --builder=containerd: Use the builder we just created

  • --cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG: Import cache from ECR

  • --cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true: Export cache back to ECR

  • -t $ECR_REPO:$IMAGE_TAG: Tag your final image as usual

  • --load: Load the built image into local Docker (so you can run it in post_build)

  • .: Your Dockerfile location

Step 5: Complete Example Buildspec

Here's what a complete, production-ready buildspec looks like:

version: 0.2phases: install: commands: - echo Logging in to Amazon ECR - aws ecr get-login-password --region $AWS_REGION | docker login --username AWS --password-stdin $AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com - CACHE_TAG=dev-cache - IMAGE_TAG=latest - ECR_REPO=$AWS_ACCOUNT_ID.dkr.ecr.$AWS_REGION.amazonaws.com/myapp - docker buildx create --name containerd --driver=docker-container --driver-opt default-load=true --use || docker buildx use containerd build: commands: - echo Build started on `date` - | docker buildx build \ --builder=containerd \ --cache-from type=registry,ref=$ECR_REPO:$CACHE_TAG \ --cache-to type=registry,ref=$ECR_REPO:$CACHE_TAG,mode=max,image-manifest=true \ -t $ECR_REPO:$IMAGE_TAG \ --load \ . - docker tag $ECR_REPO:$IMAGE_TAG $ECR_REPO:latest post_build: commands: - echo Build completed on `date` - docker push $ECR_REPO:$IMAGE_TAG - docker push $ECR_REPO:latestartifacts: files: - imageDefinitions.json
Enter fullscreen mode Exit fullscreen mode

Step 6: Update Your CodeBuild Project

You can update your buildspec in two ways:

Option 1: If your buildspec is in your repo Just commit the changes and push. CodeBuild will pick up the new buildspec automatically.

Option 2: If your buildspec is defined in CodeBuild Use the AWS CLI:

aws codebuild update-project --name your-project-name --cli-input-json file://buildspec.json
Enter fullscreen mode Exit fullscreen mode

Or update it through the AWS Console: CodeBuild Your Project Edit Buildspec

What to Expect: First Build vs Subsequent Builds

First Build (The Investment)

Your first build after implementing this will actually take slightly longer (maybe 30-60 seconds more). Don't panic! This is normal.

Here's what's happening:

  1. Creating the buildx builder (~5-10 seconds)

  2. Attempting to import cache (fails - no cache exists yet)

  3. Building all layers from scratch

  4. Exporting the cache to ECR (new step, adds ~20-40 seconds)

You'll see messages like:

=> importing cache manifest from $ECR_REPO:dev-cache=> error: not found
Enter fullscreen mode Exit fullscreen mode

This is expected! The cache doesn't exist yet.

Subsequent Builds (The Payoff)

This is where the magic happens. Your next builds will:

  1. Successfully import the cache from ECR

  2. Identify which layers haven't changed

  3. Reuse cached layers (fast!)

  4. Rebuild only the changed layers

  5. Export the updated cache

Expected time savings:

  • Before : 6-7 minutes (full rebuild every time)

  • After : 5-5.5 minutes (25-30% faster!)

  • Savings : 1-2 minutes per build

If you're doing 10 builds a day, that's 10-20 minutes saved daily. Over a month? That's 5-10 hours of compute time and costs saved.

Verifying It's Working

After your first build completes, check your ECR repository. You should now see two image tags :

  1. Your application image (e.g., latest)

  2. Your cache image (e.g., dev-cache)

The cache image will be roughly the same size as your application image - this is normal! It's storing all the layer information.

Troubleshooting Common Issues

Issue 1: "buildx: command not found"

Solution : Update your CodeBuild image to a newer version. Use aws/codebuild/standard:7.0 or later (or the ARM equivalent).

Issue 2: Cache Import Keeps Failing

Solution : Check your IAM permissions. Your CodeBuild role needs:

  • ecr:BatchGetImage

  • ecr:GetDownloadUrlForLayer

  • ecr:BatchCheckLayerAvailability

  • ecr:PutImage

  • ecr:InitiateLayerUpload

  • ecr:UploadLayerPart

  • ecr:CompleteLayerUpload

Issue 3: Build Hangs at "exporting cache"

Solution : Make sure privilegedMode: true is enabled in your CodeBuild environment settings. This is required for Docker-in-Docker operations.

Advanced: Multi-Environment Setup

If you have multiple environments (dev, staging, prod), use different cache tags for each:

- CACHE_TAG=${ENVIRONMENT}-cache # Results in: dev-cache, staging-cache, prod-cache
Enter fullscreen mode Exit fullscreen mode

This way:

  • Dev builds don't invalidate staging cache

  • Each environment maintains its own optimized cache

  • You can still share a base cache if needed

Cost Considerations

Storage Cost : You're now storing an additional cache image in ECR. At roughly the same size as your app image, this might add $0.10-0.50/month per repository depending on image size.

Compute Savings : Faster builds = less compute time. If you're saving 1-2 minutes per build and doing 10 builds/day, that's roughly 10-20 fewer compute hours per month. At ~$0.005/minute for BUILD_GENERAL1_SMALL, you could save $3-6/month.

Net Result : Typically a small net savings, plus the huge developer experience win of faster feedback loops.

Conclusion

By implementing ECR as a remote cache backend for your CodeBuild Docker builds, you get:

Faster build times

Persistent, reliable caching across all builds

Better layer reuse with intelligent cache management

Minimal code changes (just updating your buildspec)

Cost savings from reduced compute time

The implementation is straightforward, and the benefits are immediate (after the first build). Give it a try on your next project!

References

Got questions or run into issues? Drop a comment below - I'd love to hear about your experience implementing this!

]]>

Top comments (0)