- Book: Hexagonal Architecture in Go
- Also by me: Thinking in Go (2-book series) — Complete Guide to Go Programming + Hexagonal Architecture in Go
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You've seen this one. The CI green-lights the build. The
image pushes. The pod boots. And then the first request to
an external service hangs for thirty seconds and times out
with a DNS error that contradicts what nslookup inside the
container just told you.
The pod is on Alpine. The binary was built with CGO_ENABLED=1
on Debian. The C runtime that compiled it was glibc. The C
runtime trying to run it is musl. They are not the same
runtime, and Go's net package quietly notices.
The decision tree behind that one line in your Dockerfile,
CGO_ENABLED=0 or CGO_ENABLED=1, decides whether you can
ship to Alpine, to distroless, or to scratch. Most teams set
it by copy-paste from a tutorial and never revisit it until
something fails in production.
Why the cgo bit matters at all
Go has two DNS resolvers. The pure-Go resolver reads
/etc/resolv.conf and /etc/hosts itself and speaks DNS
over UDP/TCP from Go code. The cgo resolver calls into the
host's getaddrinfo, which is provided by the libc on the
machine.
When you import "net" and CGO_ENABLED=1, the Go runtime
picks between them at runtime based on a set of rules in
src/net/conf.go. The default is to prefer the cgo resolver
when cgo is available. That links the binary against
libc.so.6 from whatever distribution built it.
That binary will not run on Alpine. Alpine ships musl, not
glibc. The dynamic loader cannot find libc.so.6. You get
errors like not found when running a binary that exists,
or DNS lookups that succeed at the syscall level but fail at
the resolver level.
CGO_ENABLED=0 flips this. The binary is statically linked,
the cgo resolver is not compiled in, and /etc/resolv.conf
is parsed by Go. It runs on Alpine, distroless, scratch,
anything. The trade-off: anything that wraps a C library
stops compiling. That covers
mattn/go-sqlite3,
confluent-kafka-go,
SQLCipher, and most native image bindings. Pick what hurts less.
The four targets you actually ship to
Four base images cover almost every Go deployment. Each has
a different story about libc, certificates, and what tools
exist for debugging.
Scratch. Empty. No shell, no libc, no /etc/resolv.conf
template, no CA bundle. You ship one file: your binary. Only
works with CGO_ENABLED=0 and pure-Go binaries that do not
need any of those files at runtime.
Distroless (gcr.io/distroless/static-debian12 or
distroless/base-debian12). Two flavours matter for Go.
static has no libc and is the right base for
CGO_ENABLED=0 builds — it includes /etc/passwd, a CA
bundle, and tzdata. base includes glibc and is the right
base for CGO_ENABLED=1 builds. Both are signed, pinnable
by digest, and free of shells.
Alpine (alpine:3.19). Tiny, has a package manager, has
musl libc. You can apk add your way out of trouble. The
catch: musl is not glibc. A binary linked against glibc will
not run; a binary linked against musl will not run on
glibc-based hosts unless you accept the static-link tax.
Debian slim (debian:bookworm-slim). Big, glibc, full
package manager, easy to debug. The default if you do not
care about image size and you do care about being able to
apt install strace at three in the morning.
The decision tree
Three questions. Answer them in order.
Question 1: do you need cgo?
If you import any of these, you need cgo:
github.com/mattn/go-sqlite3github.com/confluentinc/confluent-kafka-gogocv.io/x/gocv- Any package that wraps a C library or includes
// #includedirectives at the top of.gofiles.
go list -deps -f '{{if .CgoFiles}}{{.ImportPath}}{{end}}' ./...
will print every package in your build graph that uses cgo.
Run it before you decide.
If the answer is yes, skip to question 3. You are choosing
between Alpine-musl and distroless-glibc.
If the answer is no, continue to question 2. You have the
luxury of static binaries.
Question 2 (no cgo): how minimal do you want?
CGO_ENABLED=0 GOOS=linux GOARCH=amd64 go build -ldflags="-s -w"
gives you a static binary. From here the base image is taste.
- Scratch if you want the smallest possible image and you have arranged for the CA bundle, timezone data, and any other files you need to be in the binary or the filesystem.
-
distroless/static if you want scratch's posture but
with
/etc/passwd, CA certs, andtzdataalready there. This is the right default for most Go services. -
Alpine if you want a shell for debugging and you are
willing to pay 5MB for it. With
CGO_ENABLED=0musl is not in the picture; the binary does not link against any libc at all.
The size gap between distroless/static and alpine is
small. The posture gap is bigger. Distroless has no shell
and no package manager, which removes a class of in-pod
escalation paths. Alpine keeps both, which is what you want
the night an on-call engineer needs to wget something from
inside the running pod.
Question 3 (cgo required): musl or glibc?
This is the question that bites.
If your build host is Debian or a golang:1.22-bookworm
image and you CGO_ENABLED=1 go build, the binary is linked
against glibc. The base image must also have glibc. That
means distroless/base, debian:slim, or ubuntu.
If you want Alpine as the base, you need to build the binary
on Alpine — or to cross-compile against musl. The simplest
path is a multi-stage Dockerfile where the build stage is
golang:1.22-alpine and the runtime stage is
alpine:3.19. Both have musl. The binary works.
The catch with Alpine + cgo is that some C libraries on
Alpine behave subtly differently from their glibc
counterparts. getaddrinfo on musl handles
/etc/resolv.conf's options line and AI_ADDRCONFIG
differently from glibc, which is documented in the
musl functional differences page
and tracked across multiple
golang/go DNS issues.
dlopen flags differ. Libraries tested only on glibc tend
to hit one of these edges in production before they hit it
in CI.
If you have the choice, distroless wins for cgo builds. You
pin it by digest, the image is signed, and the libc matches
what runs on your laptop. Alpine trades that for a smaller
image and a package manager, plus a libc you have to learn
around.
DNS, the resolver, and netdns
Even with CGO_ENABLED=0, the Go DNS resolver has knobs.
The pure-Go resolver is good but not bug-free. Three things
let you control it.
Build tag netgo. go build -tags netgo forces the
pure-Go resolver to be compiled in even when cgo is
available. Useful when you want a static binary but cannot
set CGO_ENABLED=0 for some other reason.
Build tag netcgo. The opposite — forces the cgo
resolver to be the only one. Rare; usually a sign something
upstream made a wrong call.
GODEBUG=netdns=go (or cgo). Runtime override. Set it
in your container env. netdns=go+1 adds resolver tracing
to stderr, which is the fastest way to debug a DNS issue in
production. You will see lines like go package net: using confirming which path is live.
the go resolver
The default rules are written in
src/net/conf.go.
Read them once. The short version: pure-Go on most Linux
unless /etc/nsswitch.conf mentions something exotic, in
which case it falls back to cgo.
Three Dockerfiles that work
Pick the one that matches your decision tree answer.
Pure-Go binary on distroless/static
# Build stage
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 GOOS=linux GOARCH=amd64 \
go build -ldflags="-s -w" -o /out/app ./cmd/app
# Runtime stage
FROM gcr.io/distroless/static-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]
This is the right default for most Go services. Static
binary, no libc to worry about, CA certs and tzdata
included, runs as a non-root user.
cgo-required binary on distroless/base
FROM golang:1.22 AS build
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=1 GOOS=linux GOARCH=amd64 \
go build -ldflags="-s -w" -o /out/app ./cmd/app
FROM gcr.io/distroless/base-debian12:nonroot
COPY --from=build /out/app /app
USER nonroot:nonroot
ENTRYPOINT ["/app"]
distroless/base includes glibc, /etc/passwd, CA certs,
and tzdata. Same posture as distroless/static but with
libc available. This is what you reach for when SQLite or a
similar cgo-bound dependency is in the build.
cgo-required binary on Alpine (both stages musl)
FROM golang:1.22-alpine AS build
RUN apk add --no-cache build-base sqlite-dev
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=1 GOOS=linux GOARCH=amd64 \
go build -ldflags="-s -w" -o /out/app ./cmd/app
FROM alpine:3.19
RUN apk add --no-cache ca-certificates sqlite-libs
RUN adduser -D -u 10001 app
COPY --from=build /out/app /app
USER app
ENTRYPOINT ["/app"]
Both stages use musl. The binary links against musl in
build, runs against musl in runtime. CA certs and the SQLite
shared library are installed via apk. This is the right
shape if you have decided Alpine is the base — keep both
stages on the same libc family.
The trap to avoid: a build stage on golang:1.22 (Debian,
glibc) and a runtime stage on alpine:3.19 (musl). The
binary will not start. The error is some variant of not when running a file that clearly exists.
found
How to verify you got it right
Three checks. Run them after every Dockerfile change.
Check what the binary linked against. Inside the
container or against the binary on your host:
# A static binary should print "not a dynamic executable"
file ./app
ldd ./app
For a CGO_ENABLED=0 binary you want to see not a dynamic
executable (or statically linked). For a cgo binary
you want to see libc.so.6 (glibc) or ld-musl-x86_64.so.1
(musl) and you want them to match the runtime image.
Check which DNS resolver is live. Boot the container
with GODEBUG=netdns=go+1 set, hit any external host, and
read the logs. You will see one of:
go package net: using the go resolver
go package net: using cgo DNS resolver
If you wanted pure-Go and it picked cgo, find out why.
Usually /etc/nsswitch.conf or an unusual resolv.conf
flag.
Check that DNS actually works under load. A pod that
resolves fine for one request can run out of UDP source
ports under high concurrency, especially with the pure-Go
resolver on a host with a busy conntrack table. Throw a
small load test at it before you congratulate yourself.
The CGO_ENABLED bit
One line in your Dockerfile. The consequences are everything
else: which libc the binary links against, which base image
it can run on, which DNS resolver it picks, and which class
of subtle production failures you sign up for.
If this was useful
This is the kind of build-time decision the Thinking in Go
series spends time on — not the headline language features,
but the choices that decide whether your service stays up.
Hexagonal Architecture in Go covers how to keep the
infrastructure layer (Dockerfiles included) at the edge of
the system, so swapping a base image or a database driver
does not ripple through the domain. Complete Guide to Go
Programming is the companion that walks through the
language and runtime details — including how the net package
picks its resolver — that this post assumes you know.



Top comments (0)