Why Unbounded Worker Pools Blow Up Go Services Under Back‑pressure

#architecture #backend #go #performance

problem statement

in a recent deployment we added a background worker pool to a high‑traffic HTTP service. the pool pulled jobs from a shared channel without any limit on the number of goroutines. under normal load the service seemed fine, but a spike in upstream traffic caused the request queue to fill, the pool spawned thousands of goroutines, and the pod hit OOM kills within seconds. the crash cascaded to the entire deployment because the readiness probe never recovered.

why the pool exploded

no back‑pressure – a plain chan Job without a buffer or a semaphore allows any number of producers to push work.
missing context propagation – workers ignored the request context, so they kept processing even after the client timed out.
absence of telemetry – only occasional log lines were emitted; the CPU and memory usage spiked unnoticed until the pod was evicted.

a minimal reproducible snippet

type Job struct{ payload []byte }

func startWorkerPool(jobs <-chan Job) {
    for job := range jobs { // <- unbounded loop
        go func(j Job) {
            // heavy processing, no ctx
            _ = process(j.payload)
        }(job)
    }
}

func main() {
    jobs := make(chan Job) // unbuffered, producers block only on send
    go startWorkerPool(jobs)
    http.HandleFunc("/enqueue", func(w http.ResponseWriter, r *http.Request) {
        // simulate burst of 10k requests per second
        for i := 0; i < 1000; i++ {
            jobs <- Job{payload: []byte("data")}
        }
        w.WriteHeader(http.StatusAccepted)
    })
    http.ListenAndServe(":8080", nil)
}

production‑grade fixes

bulkhead with semaphore – limit concurrent workers to a sane number (e.g., runtime.GOMAXPROCS()*2).
bounded channel or work‑queue – use a buffered channel and drop or back‑pressure when the buffer is full.
context‑aware workers – pass r.Context() downstream; abort processing when the client disconnects.
metrics & alerts – expose goroutine, heap, and queue length via Prometheus; set alerts on 80% memory usage.
chaos testing – inject artificial latency on the upstream API and verify that the pool respects the limits without OOM.

trade‑offs to consider

throughput vs latency – a tighter semaphore reduces peak throughput but guarantees latency bounds and prevents node eviction.
queue size – a small buffer protects the node but may increase request rejection rates; choose a size based on expected burst and acceptable 503 rate.
cost of retries – if you drop jobs, ensure idempotent downstream processing or a durable retry store (e.g., Kafka) to avoid data loss.
observability overhead – adding per‑request tracing adds CPU, but without it you cannot see the back‑pressure chain.

conclusion

unbounded worker pools are a classic hidden failure mode that surfaces only under load spikes. limiting concurrency, propagating cancellation, and instrumenting the queue are non‑negotiable in production. how does your stack enforce bulkhead isolation for background processing, and what metrics do you rely on to detect a runaway goroutine storm?

DEV Community