- Book: The Complete Guide to Go Programming
- Also by me: Hexagonal Architecture in Go — the companion book in the Thinking in Go series
- My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
- Me: xgabriel.com | GitHub
You have 5,000 image URLs and a function that thumbnails one of them.
Called in a loop, it takes eleven minutes. The machine has 8 cores,
seven of them idle the whole time. The work is embarrassingly
parallel: no URL depends on any other. You know Go can do this. The
question is how to wire it so you don't leak goroutines, don't deadlock
on a channel nobody drains, and don't lose the one result that
mattered.
That wiring has a name. Fan-out is splitting one stream of work across
N goroutines. Fan-in is merging their results back into one stream.
The pattern is old, the Go standard library gives you every piece, and
almost every non-trivial batch job you write ends up shaped like it.
The traps are all in the coordination.
The shape: one input channel, N workers, one output channel
Start with the skeleton. A producer sends jobs into a channel. N
workers read from that channel, do the work, and send results into a
second channel. A collector reads results.
type Job struct {
ID int
URL string
}
type Result struct {
ID int
Bytes int
Err error
}
The fan-out is one for loop that starts N goroutines, all ranging
over the same input channel:
func fanOut(
jobs <-chan Job,
results chan<- Result,
workers int,
wg *sync.WaitGroup,
) {
for i := 0; i < workers; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for j := range jobs {
results <- process(j)
}
}()
}
}
Each worker ranges over jobs. Go's channel semantics do the load
balancing for you: whichever worker is free grabs the next job. You
don't assign work; you let the workers pull it. When jobs is closed
and drained, every range loop ends, every worker calls wg.Done(),
and the goroutines exit.
Note the receive-only <-chan Job and send-only chan<- Result types
in the signature. That directionality is a compile-time guarantee that
a worker can't accidentally send into its own input or read its own
output. Use it.
Closing the output channel: the WaitGroup dance
Here is the part that trips people up. The collector wants to range
over results and stop when the work is done. For a range to stop,
results has to be closed. But no single worker can close it: if
worker 3 closes results while worker 5 is still trying to send, that
send panics with send on closed channel.
The rule in Go is: the channel is closed by the goroutine that knows
all senders are finished, and only that one. Nobody knows that from
inside a worker. So you spawn a small coordinator goroutine that waits
on the WaitGroup and closes the channel after every worker has
returned.
func run(urls []string, workers int) []Result {
jobs := make(chan Job)
results := make(chan Result)
var wg sync.WaitGroup
fanOut(jobs, results, workers, &wg)
// producer
go func() {
defer close(jobs)
for i, u := range urls {
jobs <- Job{ID: i, URL: u}
}
}()
// closer: waits for all workers, then closes results
go func() {
wg.Wait()
close(results)
}()
// fan-in: drain until results is closed
var out []Result
for r := range results {
out = append(out, r)
}
return out
}
Trace the shutdown, because the ordering is the whole point:
- The producer finishes sending and
close(jobs)runs. - Each worker's
range jobssees the closed, drained channel and returns, callingwg.Done(). - Once the last worker returns,
wg.Wait()unblocks andclose(results)runs. - The
for r := range resultsloop sees the closed, drained channel and returns.
Every close is done by exactly one goroutine that knows its senders
are done. The producer owns jobs. The closer goroutine owns
results. No worker closes anything. That single rule prevents both
the send on closed channel panic and the deadlock where nobody ever
closes results and the collector blocks forever.
Bounding concurrency: workers, not "a goroutine per job"
The instinct from other languages is go process(job) in a loop, one
goroutine per item. Go goroutines are cheap, so people assume this is
fine. For CPU-bound work it isn't. Spawn 5,000 goroutines that all
want a core and you thrash the scheduler and blow past whatever
connection or file-descriptor limit the work touches.
The worker-pool version above caps concurrency at workers. Pick that
number based on the bottleneck:
// CPU-bound: match the cores you can use.
workers := runtime.GOMAXPROCS(0)
// I/O-bound (HTTP, DB): usually higher than cores,
// tuned to what the downstream can take.
workers := 64
For CPU-bound work, runtime.GOMAXPROCS(0) reads the current setting
without changing it, which reflects the cores your process is allowed
to use. For I/O-bound work you want more goroutines than cores,
because most of them are parked waiting on the network, but you still
want a ceiling so you don't open 5,000 sockets at once. The whole
reason for the pool is that ceiling.
Preserving order (when you need it)
The fan-in above returns results in completion order. Fast jobs finish
first. If you're thumbnailing images and writing them to a bucket
keyed by ID, you don't care. If you're processing lines of a file and
have to write them back in the original order, you very much do.
Do not try to make the workers finish in order; that throws away the
parallelism. Instead, carry the index on the job (the Job.ID field
is already there) and reorder at the end. Since you know the count up
front, a preallocated slice indexed by ID is the cleanest way:
func runOrdered(urls []string, workers int) []Result {
// ... same jobs/results/wg setup and goroutines ...
ordered := make([]Result, len(urls))
for r := range results {
ordered[r.ID] = r // slot by original index
}
return ordered
}
Each result lands in its original slot regardless of when it arrives.
No sorting, no locks: every worker writes a distinct index, so the
writes don't race. You get parallel execution and ordered output at
the same time. When the count isn't known ahead of time, collect into
a slice and sort.Slice by ID at the end instead.
Propagating failure: cancel the rest when one job dies
A batch of 5,000 jobs where one fails raises a question the happy-path
code ignores: do you finish the other 4,999, or stop early? For a
best-effort batch, keep going and collect the errors (the Result.Err
field carries them). For an all-or-nothing job, cancel the moment one
worker hits a fatal error so you don't burn cycles on work you're going
to throw away.
errgroup from golang.org/x/sync wires this up. It combines the
WaitGroup, the first-error capture, and a context that cancels when
any worker returns an error.
func runGroup(ctx context.Context, urls []string) error {
g, ctx := errgroup.WithContext(ctx)
g.SetLimit(runtime.GOMAXPROCS(0)) // bound the fan-out
for i, u := range urls {
i, u := i, u // pre-1.22 capture; harmless on 1.22+
g.Go(func() error {
select {
case <-ctx.Done():
return ctx.Err()
default:
}
return store(ctx, i, u)
})
}
return g.Wait()
}
g.SetLimit(n) gives you the worker cap without hand-rolling the pool.
errgroup.WithContext hands back a ctx that is cancelled as soon as
any g.Go function returns non-nil, so the still-running workers see
ctx.Done() and bail. g.Wait() returns the first error. In a real
service this is the version to reach for: it collapses the WaitGroup
dance, the concurrency limit, and cancellation into three lines you
don't have to get subtly wrong.
The i, u := i, u line is only needed if you target Go before 1.22.
From 1.22 on, each loop iteration gets its own variables and the
capture is safe without it. Leaving it in does no harm.
The four failure modes, in one place
Every fan-out/fan-in bug you'll hit is one of these:
-
Deadlock on the output channel. You forgot the closer goroutine,
so
resultsnever closes and the collector blocks on arangeforever. Fix: one goroutine that doeswg.Wait()thenclose(results). -
send on closed channelpanic. A worker closed the output channel while another was still sending. Fix: no worker closes anything; only the coordinator does. -
Unbounded goroutines. One goroutine per job instead of a pool.
Fix: a fixed worker count, or
errgroup'sSetLimit. - Scrambled output. You assumed completion order equals input order. Fix: carry the index and slot results into a preallocated slice.
Get those four right and the pattern scales from a 20-line script to
the core of a batch service. The channels do the load balancing, the
WaitGroup does the shutdown, and the index does the ordering.
If you want the mechanics under this: how the scheduler parks and
wakes goroutines, why an unbuffered channel synchronizes sender and
receiver, what close actually does to a blocked range. That's the
runtime chapter of The Complete Guide to Go Programming. And once a
pattern like this grows into a real service, Hexagonal Architecture in
Go is about keeping the concurrency at the right boundary, so your
worker pool stays an adapter detail instead of leaking into your domain
logic. Both ship together as the Thinking in Go series.
Top comments (0)