DEV Community

Cover image for Optimizing Crystal Build time in Woodpecker CI: 415s to 196s with Caching
Michael Nikitochkin
Michael Nikitochkin

Posted on

Optimizing Crystal Build time in Woodpecker CI: 415s to 196s with Caching

#crystal #ci #woodpecker #performance

The Problem

Crystal test builds in Woodpecker CI were taking 415 seconds to complete.
Every pipeline run would recompile dependencies from scratch, even though most changes affected application code, not third-party libraries.

Crystal's compiler is thorough and safe, but recompiling everything on each CI run was costly - especially when running tests multiple times per day.

The Solution

Persistent Crystal cache storage was implemented using named volumes combined with a custom cache directory:

test:
  image: crystallang/crystal:${CRYSTAL_VERSION}
  environment:
    CRYSTAL_CACHE_DIR: /cache/crystal
  volumes:
    # Persistent cache for compiled Crystal modules
    - crystal-cache-${CRYSTAL_VERSION}:/cache/crystal
  commands:
    - rake test:build
    - rake test:run
Enter fullscreen mode Exit fullscreen mode

Note: Testing also included adding tmpfs: ["/tmp:size=2g"] but it provided no measurable improvement. The persistent cache is where the real optimization happens.

How It Works

1. Custom Cache Directory

By setting CRYSTAL_CACHE_DIR: /cache/crystal, the Crystal compiler stores compiled artifacts in a predictable location instead of the default temporary directory.

environment:
  CRYSTAL_CACHE_DIR: /cache/crystal
Enter fullscreen mode Exit fullscreen mode

This provides control over where Crystal stores:

  • Compiled standard library modules
  • Compiled dependency (shard) modules
  • Precompiled object files
  • LLVM intermediate representations

2. Named Volumes for Persistence

Woodpecker CI uses container volumes to persist data between pipeline runs. A named volume is mounted at the cache directory:

volumes:
  - crystal-cache-${CRYSTAL_VERSION}:/cache/crystal
Enter fullscreen mode Exit fullscreen mode

Key insight: Using ${CRYSTAL_VERSION} in the volume name maintains separate caches for different Crystal versions (e.g., crystal-cache-1.16.3 and crystal-cache-nightly). This prevents cache conflicts when testing against multiple Crystal versions in matrix builds.

3. What About tmpfs for /tmp?

Initial assumptions suggested that adding tmpfs for /tmp would help, since Crystal might write temporary files there during compilation. Testing was performed:

tmpfs:
  - /tmp:size=2g
Enter fullscreen mode Exit fullscreen mode

Result: No measurable improvement (~196s with or without tmpfs).

Why? Crystal's compilation model doesn't write much to temporary directories:

  • Most I/O goes to CRYSTAL_CACHE_DIR (which is already optimized with persistent volumes)
  • Crystal keeps most compilation state in memory
  • The compiler creates minimal intermediate temp files
  • Container overlay filesystem is "good enough" for the small amount of /tmp usage

4. Container Overlay Storage Optimization

Woodpecker agents using container overlay storage drivers (such as Podman's overlay2) benefit from efficient layer caching.
The named volume persists between runs, and only changed files need to be written - combining with the container runtime's copy-on-write mechanism for optimal performance.

Important note: Named volumes are local to each agent. If multiple Woodpecker agents are running, each maintains its own separate cache. This means:

  • First run on a new agent will be slow (cold cache)
  • Subsequent runs on the same agent will be fast (warm cache)
  • Pipelines may experience variable build times depending on which agent executes them

For shared caching across multiple agents, consider external cache storage solutions (S3, network volumes, or distributed cache systems).

The Impact

Before (no optimizations): 415 seconds per test build

After (persistent cache only): 196 seconds per test build

After (cache + tmpfs for /tmp): ~196 seconds (no significant change)

Improvement: 2.1x faster (53% time reduction) ⚡

Why Persistent Cache Helps, But tmpfs Doesn't

Persistent cache (/cache/crystal) - BIG WIN:

  • ✅ Avoids recompiling unchanged dependencies between builds
  • ✅ Speeds up subsequent builds dramatically (415s → 196s)
  • ✅ Survives across pipeline runs

tmpfs for /tmp - Minimal Impact:

  • ⚠️ Crystal doesn't write much to /tmp during compilation
  • ⚠️ Most I/O goes to CRYSTAL_CACHE_DIR (already on persistent volume)
  • ⚠️ Modern container overlay filesystem is "good enough" for the small amount of temp files

Why This Makes Sense

Crystal's compiler architecture is smart:

  1. Compiled artifacts go to cache directory - this is where the bulk of I/O happens
  2. Temporary files are minimal - Crystal doesn't create many intermediate temp files
  3. Most work is in-memory - the compiler keeps most data structures in RAM during compilation
  4. Cache hits dominate - on warm cache, there's very little new compilation happening

The persistent cache eliminated the expensive recompilation (hundreds of seconds). The remaining time is mostly:

  • Linking compiled objects
  • Running tests (database operations)
  • Test framework overhead

Adding tmpfs for /tmp doesn't help because there's simply not much disk I/O happening there.

What Gets Cached?

On the first run, Crystal compiles everything:

  • Standard library (~100MB of compiled code)
  • Third-party shards (dependencies)
  • Application code

On subsequent runs, Crystal reuses:

  • ✅ Unchanged standard library modules
  • ✅ Unchanged dependency code
  • ✅ Unchanged application code
  • ❌ Only recompiles what changed

Matrix Builds: One Cache Per Version

CI testing runs against multiple Crystal versions:

matrix:
  CRYSTAL_VERSION:
    - 1.16.3
    - 1.19.1
    - nightly
Enter fullscreen mode Exit fullscreen mode

Each version gets its own cache:

  • crystal-cache-1.16.3 - old version cache
  • crystal-cache-1.19.1 - stable version cache
  • crystal-cache-nightly - nightly version cache

This prevents cache corruption when compiler internals change between versions.

Implementation Details

Complete Woodpecker Configuration 

steps:
  test:
    image: crystallang/crystal:${CRYSTAL_VERSION}
    environment:
      DATABASE_URL: postgres://postgres:dbpgpassword@postgres:5432/api_test
      CRYSTAL_CACHE_DIR: /cache/crystal
    volumes:
      - crystal-cache-${CRYSTAL_VERSION}:/cache/crystal
    commands:
      - crystal env
      - rake test
Enter fullscreen mode Exit fullscreen mode

Repository Trust Requirement

Important: Using volumes: and tmpfs: in Woodpecker CI requires the repository to be marked as "Trusted".

Why trust is required:

  • volumes: - Allows mounting host volumes into containers, providing access to persistent storage
  • tmpfs: - Allows mounting in-memory filesystems, requiring elevated container privileges

Both features give containers more access to the host system. Woodpecker restricts these features to prevent potentially malicious pipeline configurations from compromising the CI infrastructure.

How to enable trust:

  1. Navigate to repository settings in Woodpecker UI
  2. Enable the "Trusted" checkbox
  3. Only repository administrators can modify this setting

Security consideration: Only enable trust for repositories with controlled access and reviewed pipeline configurations. Trusted pipelines can potentially access sensitive data on the CI host system.

Key Takeaways

  1. Named volumes survive pipeline runs - Woodpecker's volume mounting is key for persistence
  2. Caches are per-agent - each Woodpecker agent maintains its own cache, not shared across agents
  3. Repository must be marked as "Trusted" - required for using volumes: and tmpfs: features
  4. 2x speedup is achievable - especially for projects with many dependencies
  5. Focus on what matters - persistent cache is the killer feature for Crystal CI

Potential Issues and Solutions

Problem: Cache grows too large

Solution: Periodically clean old caches or set retention policies in Woodpecker agent configuration.

Problem: Cache corruption after Crystal upgrade

Solution: The ${CRYSTAL_VERSION} suffix naturally creates new caches for new versions.

Problem: Multiple agents don't share caches

Solution: This is by design - named volumes are local to each agent. Each agent maintains its own cache, which means:

  • First build on each agent will take the full 415 seconds (cold cache)
  • Subsequent builds on the same agent will take ~196 seconds (warm cache)
  • Build times vary depending on which agent picks up the job

For consistent performance across all agents, consider:

  • Use agent labels to pin jobs to specific agents
  • Implement external cache storage (S3, NFS, network volumes)
  • Accept variable build times as a trade-off for distributed load

Problem: Volumes not mounting or permission errors

Solution: Verify the repository has "Trusted" status enabled in Woodpecker settings. Without trust, volumes: and tmpfs: directives are silently ignored or produce permission errors.

Comparison with Other CI Systems

CI System Cache Strategy
GitHub Actions actions/cache with path /home/runner/.cache/crystal
GitLab CI cache: directive with key: ${CI_COMMIT_REF_SLUG}
Woodpecker Named volumes with version-specific keys

Woodpecker's approach is simpler - no cache upload/download steps, just persistent volumes.

Top comments (0)