DEV Community

Cover image for 🚀 Stop Chasing Small Docker Images: What Actually Matters for Go in Production
Chiman Jain
Chiman Jain

Posted on

🚀 Stop Chasing Small Docker Images: What Actually Matters for Go in Production

#devops #docker #cicd #go

A practical guide to reproducible builds, faster CI pipelines, and debuggable containers for Go engineers

Most Docker + Go tutorials end the same way:

“Use multi-stage builds, switch to Alpine, done.”

That advice works until it doesn’t.

At scale, different problems show up:

  • CI pipelines slow down unpredictably
  • Builds stop being reproducible
  • Debugging minimal containers becomes painful
  • Monorepos destroy Docker cache efficiency

This article focuses on what actually matters in production:
👉 reproducibility, caching, and operability

🔍 How Docker + Go Builds Actually Work

Before optimizing, it helps to visualize what’s happening.

📊 Build Flow Diagram 

        ┌───────────────┐
        │   Source Code │
        └──────┬────────┘
               │
               ▼
        ┌───────────────┐
        │  go.mod/sum   │
        └──────┬────────┘
               │
               ▼
        ┌──────────────────────┐
        │ go mod download      │
        │ (dependency layer)   │
        └──────┬───────────────┘
               │
               ▼
        ┌──────────────────────┐
        │ go build             │
        │ (compile layer)      │
        └──────┬───────────────┘
               │
               ▼
        ┌──────────────────────┐
        │ Final Image          │
        │ (distroless/scratch) │
        └──────────────────────┘
Enter fullscreen mode Exit fullscreen mode

⚠️ Key Insight:

If go.mod changes → everything below it rebuilds

🧠 Reproducible Builds: The Overlooked Problem

❌ What Goes Wrong

Same code, different builds:

  • Different architectures (amd64 vs arm64)
  • Embedded file paths
  • Environment-dependent outputs

✅ Fixing It

1. Remove Local Paths 

go build -trimpath -o app
Enter fullscreen mode Exit fullscreen mode

2. Lock Dependencies 

go mod download
Enter fullscreen mode Exit fullscreen mode

Optional stricter control:

GOPROXY=https://proxy.golang.org
GONOSUMDB=*
Enter fullscreen mode Exit fullscreen mode

3. Standardize Build Environment 

FROM golang:1.26 AS builder

ENV CGO_ENABLED=0

WORKDIR /app

COPY go.mod go.sum ./
RUN go mod download

COPY . .
RUN go build -trimpath -ldflags="-s -w" -o app
Enter fullscreen mode Exit fullscreen mode

📊 Multi-Arch Build Flow 

           ┌──────────────┐
           │   Source     │
           └──────┬───────┘
                  │
     ┌────────────┴────────────┐
     ▼                         ▼
┌──────────────┐        ┌──────────────┐
│ linux/amd64  │        │ linux/arm64  │
└──────┬───────┘        └──────┬───────┘
       ▼                       ▼
   Binary A               Binary B
 (different hash)      (different hash)
Enter fullscreen mode Exit fullscreen mode

👉 Focus on behavior consistency, not identical binaries.

⚡ Docker Caching: Why Monorepos Break It

📊 Layer Caching Model 

Layer 1: OS Base Image
Layer 2: go.mod / go.sum
Layer 3: Dependencies (go mod download)
Layer 4: Source Code
Layer 5: Build Output
Enter fullscreen mode Exit fullscreen mode

❌ Problem 

Change in go.mod
        ↓
Layer 2 invalidated
        ↓
Layer 3 re-runs (slow)
        ↓
Everything rebuilds
Enter fullscreen mode Exit fullscreen mode

✅ Optimized Caching Strategy

📊 Improved Flow 

        ┌──────────────┐
        │ go.mod/sum   │
        └──────┬───────┘
               ▼
   (cached via BuildKit mount)
        ▼
   Dependencies reused ✅

        ┌──────────────┐
        │ Source Code  │
        └──────┬───────┘
               ▼
        Build runs faster ⚡
Enter fullscreen mode Exit fullscreen mode

🔧 Practical Fixes

Use BuildKit Cache 

RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download
Enter fullscreen mode Exit fullscreen mode

Cache Build Artifacts 

RUN --mount=type=cache,target=/root/.cache/go-build \
    go build -trimpath -ldflags="-s -w" -o app
Enter fullscreen mode Exit fullscreen mode

Scope Dependencies 

COPY services/service-a/go.mod services/service-a/go.sum ./
RUN go mod download
Enter fullscreen mode Exit fullscreen mode

🐳 Minimal Images: The Trade-Off Nobody Talks About

📊 Image Comparison 

scratch      → smallest → hardest to debug ❌
distroless   → balanced → production-ready ✅
alpine       → larger   → easiest debugging 🔧
Enter fullscreen mode Exit fullscreen mode

🚨 Real-World Issues

scratch

  • No TLS certs → HTTPS fails
  • No shell → cannot debug
  • No timezone/DNS tools

distroless

  • Secure and minimal
  • But still no shell

alpine

  • Debuggable
  • But uses musl libc (can cause subtle issues)

✅ Practical Strategy 

Production   → distroless
Debug build  → alpine
Special case → scratch
Enter fullscreen mode Exit fullscreen mode

🧪 CI/CD Optimized Dockerfile

🚀 Production-Ready Template 

FROM golang:1.26 AS builder

WORKDIR /app

ENV CGO_ENABLED=0

COPY go.mod go.sum ./

RUN --mount=type=cache,target=/go/pkg/mod \
    go mod download

COPY . .

RUN --mount=type=cache,target=/root/.cache/go-build \
    go build \
    -trimpath \
    -ldflags="-s -w" \
    -o app

FROM gcr.io/distroless/base-debian12

WORKDIR /

COPY --from=builder /app/app /app

USER nonroot:nonroot

ENTRYPOINT ["/app"]
Enter fullscreen mode Exit fullscreen mode

🔧 Debug Variant 

FROM alpine:3.19

RUN apk add --no-cache ca-certificates

COPY --from=builder /app/app /app

ENTRYPOINT ["/app"]
Enter fullscreen mode Exit fullscreen mode

🧩 The Bigger Picture

Most engineers optimize for:

  • Image size
  • Build completion

But production systems care about:

Reproducibility  → Can I trust this build?
Debuggability    → Can I fix issues fast?
Performance      → Can CI scale?
Enter fullscreen mode Exit fullscreen mode

🎯 Final Takeaway

If your Docker setup feels:

  • slow
  • fragile
  • hard to debug

…it’s not Docker.

It’s how:

  • caching
  • dependencies
  • and runtime assumptions

interact.

Top comments (0)