DEV Community

Mustafa ERBAY
Mustafa ERBAY

Posted on • Originally published at mustafaerbay.com.tr

Build Cache Optimization in CI/CD Pipelines: 3 Practical Ways

#life #devops #cicd #productivity

Slow CI/CD processes are a silent enemy of developer quality of life, directly sabotaging a developer's ability to focus during the day. Waiting 15 minutes for tests and build processes to finish every time I push code used to pull me away from my computer and drag me into an unproductive cycle. I spent a significant amount of time trying to shorten this duration in my own side projects and in the production ERP systems I worked on.

In this article, I will explain Build Cache Optimization in CI/CD Pipelines—methods I have directly implemented that cut pipeline times by nearly 70%—complete with concrete configurations and real-world experiences. My goal is to show how we can protect our own quality of life and mental health while doing a technical job.

What is Build Cache Optimization in CI/CD Pipelines and How Does It Affect Our Lives?

In software development, "waiting" is the ultimate motivation killer. Watching the pipeline download packages from scratch for minutes after pushing a hotfix to the repository doesn't just inflate the cloud bill; it also disrupts the developer's flow state. Build Cache Optimization in CI/CD Pipelines aims to process only the changed code segments by keeping the results of previously completed intermediate steps in memory.

In the backend services of a mobile app I developed that has thousands of users, each deployment initially took exactly 12 minutes. Waiting 12 minutes just to add a single-line log message almost made me lose interest in my work. In the table below, I have compiled the time spent before and after a simple optimization from my own records:

Pipeline Step Time Before Optimization Time After Optimization Time Saved (Percentage)
Downloading Dependencies 4 min 12 sec 22 sec 91%
Docker Image Build 5 min 45 sec 1 min 10 sec 79%
Running Tests 2 min 03 sec 45 sec (Cached) 63%
Total Time 12 min 00 sec 2 min 17 sec 81%

This table didn't just save me server costs; it also gave me the freedom to deploy my code before my coffee went cold. If you find yourself heading to the kitchen to brew tea after every git push, there is an engineering problem there. As I mentioned in my previous article [related: VPS disk fullness issues], every unoptimized temporary file and cache layer eventually turns into an avalanche that clogs the entire system.

Method 1: Saving Wasted Minutes with Docker Layer Caching (DLC)

The biggest mistake made when creating a Docker image is not processing package management files (package.json, go.mod, requirements.txt) in the correct order before copying the source code. Docker saves each instruction as a layer. If you copy your source code into the image before the package list, even a single space character you write will cause all packages to be downloaded from scratch.

Below, I share an optimized Dockerfile structure that I use in my own projects, which dramatically reduces build times. This structure exploits Docker's layer caching mechanism to the fullest:

# Dockerfile
FROM node:20-alpine AS builder
WORKDIR /app

# Copy only package definitions
COPY package.json package-lock.json ./

# Install dependencies (This layer is read from cache unless packages change)
RUN npm ci --prefer-offline --no-audit

# Now we can copy the source code
COPY . .

# Build step
RUN npm run build

FROM node:20-alpine AS runner
WORKDIR /app
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/package.json ./package.json

EXPOSE 3000
CMD ["node", "dist/main.js"]
Enter fullscreen mode Exit fullscreen mode

The critical trick here is the COPY package.json package-lock.json ./ line. If you directly do COPY . . and then run npm ci, Docker will start pulling the entire node_modules folder from the internet from scratch whenever any .ts or .js file in the project changes.

ℹ️ Docker BuildKit Advantage

You can further increase caching performance by enabling the BuildKit architecture, which comes with Docker 18.09+ and above. Enabling this feature on the pipeline runner with the export DOCKER_BUILDKIT=1 command activates parallel layer compilation.

Method 2: Persisting Package Manager Caches (Node_Modules, Cargo, Go)

Docker layers work very well locally, but if you are using GitHub Actions, GitLab CI, or self-hosted runners, a clean virtual machine (clean slate) boots up for every pipeline run. In this case, the local Docker cache is lost. The solution is to use the global cache mechanism provided by the CI/CD provider to carry the files downloaded by package managers between pipelines.

Especially in Rust (cargo) or Go projects, compilation times can reach frustrating levels due to CPU limits. Below, you can examine the YAML configuration showing how I lock the package cache in a workflow running on GitHub Actions:

# .github/workflows/ci.yml
name: Continuous Integration

on:
  push:
    branches: [ main ]

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout Repository
        uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: 20

      # Generate cache key based on the hash of lock files
      - name: Cache Node Modules
        uses: actions/cache@v4
        id: npm-cache
        with:
          path: ~/.npm
          key: ${{ runner.os }}-node-${{ hashFiles('**/package-lock.json') }}
          restore-keys: |
            ${{ runner.os }}-node-

      - name: Install Dependencies
        run: npm ci --prefer-offline --no-audit
Enter fullscreen mode Exit fullscreen mode

In this configuration, the hashFiles('**/package-lock.json') expression is of vital importance. If there are no changes in the dependencies, GitHub Actions goes ahead and restores the ~/.npm folder in seconds. The npm ci --prefer-offline parameter forces npm to check the local cache before going to the internet. This reduces the package installation phase, which normally takes 3-4 minutes, down to 15-20 seconds.

Method 3: Multi-Stage Builds and Runner-Local Cache Integration

If you are running GitLab Runner or Jenkins on your own server (bare-metal or VPS), your internet bandwidth and disk write speeds directly affect pipeline performance. In this scenario, instead of using cloud-based cache services, it makes much more sense to "mount" and use the local disk space residing on the runner.

Especially while working on a production ERP, migrating PostgreSQL database schemas and spinning up backend tests required setting up the database from scratch every single time. We solved this problem by sharing the docker socket (/var/run/docker.sock) on the local runner and mounting a local directory as a cache.

The following Docker Compose and CI runner integration example shows how to share the cache directory on the local disk:

# docker-compose.ci.yml
version: '3.8'

services:
  builder:
    image: docker:24-cli
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      # Mount the cache directory on the local runner inside the container
      - /tmp/ci-cache/.npm:/root/.npm
      - /tmp/ci-cache/go-build:/root/.cache/go-build
    environment:
      - DOCKER_BUILDKIT=1
    command: docker build --build-arg BUILDKIT_INLINE_CACHE=1 -t my-app:latest .
Enter fullscreen mode Exit fullscreen mode

Thanks to this method, we read the cache directly at the runner's SSD disk speed without any upload/download traffic to an external cloud storage (S3, etc.). This local disk caching strategy is a lifesaver, especially on servers located in Turkey during periods when international connection speeds fluctuate.

Post-Optimization Metrics: Return on Investment (ROI)

Every optimization made must have a concrete output. Proving this with metrics instead of just saying "the system got faster" is a requirement of engineering discipline. After implementing these 3 methods in my own projects, I tracked the gains both in terms of time and financial aspects.

In the table below, you can see the estimated gains that a medium-sized team running an average of 400 pipelines per month would achieve:

Metric Before Optimization After Optimization Monthly Net Gain
Average Build Time 14 minutes 3.5 minutes 10.5 minutes per pipeline
Total Monthly Run Time 5,600 minutes (93 hours) 1,400 minutes (23 hours) 70 hours of server time
CI/CD Server Cost (Estimated) $186 / month $46 / month $140 Saved
Developer Waiting Time 93 hours / month 23 hours / month 70 hours of focus time

Financially, $140 might not seem like a huge amount of money, but 70 hours of developer time is priceless. When combined with the cost of context switching when a developer's focus is disrupted while waiting for a pipeline, the bill to the company and the project can reach thousands of dollars. As I touched upon in my previous article [related: Software architecture and organizational flow], optimizing processes is actually the smartest way to use human resources most efficiently.

Pitfalls and Limitations to Avoid in Build Cache Management

Just like everything else, using a build cache has its side effects and limitations. Blindly trusting caching mechanisms is sometimes the biggest cause of the "it works locally but failed on the server" syndrome.

One of the most insidious problems I encountered was a corrupted cache layer constantly sabotaging new builds. To deal with such situations, I never skip these rules:

  • Cache Busting (Cache Versioning): Even if your dependency files don't change, it is necessary to completely clear the cache once in a while. You should be able to change the cache version with an environment variable (env) when triggering the pipeline (e.g., CACHE_VERSION=v2).
  • Leakage of Secret Files into Cache: Ensure that sensitive data such as .env files, SSL certificates, or SSH keys are never included in cache directories. Keep your .dockerignore and .gitignore files up to date.
  • Disk Space Management: Cache directories on the runner's local disk swell over time. Running a small cron job with systemd timers to clean up old cache folders weekly is essential for system health.

⚠️ Security Warning

In shared CI/CD runner environments, make sure that other projects cannot access the cache directory of a specific project. Otherwise, there is a risk of malicious code being injected into your project via cache poisoning attacks.

As a next step, you can start by choosing the slowest-running pipeline in your own projects. First, perform the Dockerfile optimization, then enable the package manager cache. You will feel the difference directly after the very first build.

Top comments (0)