A production memory-leak postmortem: two well-behaved libraries, one pathological interaction, and 50 million tiny objects the garbage collector was never allowed to free.
3:36 AM
The email from Railway arrived at 3:36 AM CDT on July 2nd:
Deploy Crashed!
Uh oh. Your deployment for taskqueue crashed within the staging environment.
Annoying, but survivable — staging crashes happen. What didn't add up came later: the production instance of the same service, which never crashed, was now sitting at 14–18 GB of RAM. This is a background worker that normally idles below 100 MB. It stayed at 14–18 GB for six days, working normally the entire time, until we redeployed it — at which point it dropped back to 21 MB like nothing had ever happened.
The memory graph looked like this: flat for weeks, a near-vertical cliff up to 14 GB, a six-day plateau, then a cliff back down at the redeploy. Not a slow leak. A step function.
Slow leaks are easy to reason about — something grows a little with every request. But what allocates fourteen gigabytes in minutes and then just… stops? And why did the memory never come back down after whatever-it-was ended?
What we knew
Our setup: a Go microservices dialer system on Railway. The service in question, taskqueue, is a background worker built on asynq, a Redis-backed job queue. It processes call logs, syncs recordings, that kind of thing.
Digging through Loki gave us the trigger within the hour. At exactly 08:36:00 UTC (= 3:36 AM CDT — the same minute as the crash email), every service in the fleet started screaming the same thing:
dial tcp 10.132.244.37:6379: i/o timeout
Redis was unreachable. Not just for one service — for all of them, in both environments simultaneously, each pointing at its own Redis instance. A platform-level network blip. It lasted about 22 minutes; by 08:58 the errors stopped and everything reconnected.
Here's the thing though. Six services hit the identical outage. Five of them logged some errors, reconnected, and moved on with flat memory. Only taskqueue — in both environments — exploded. Staging hit its container limit ~9 minutes in and got OOM-killed (the 3:36 email). Production had more headroom, absorbed 14 GB, and survived.
Five services shrugged. One ballooned. The only architectural difference: taskqueue is the only service that runs an asynq server.
Ruling out our own code
Before blaming a library, we audited our own code, and it came up clean: every worker pool was semaphore-bounded and joined, every batch load was small (verified in logs), asynq's concurrency was capped at 10, mutexes released on error paths. Nothing in our code could hold gigabytes.
We also had two crucial negative results from the logs:
- Memory stopped growing the moment Redis recovered. Whatever was allocating only did so while Redis was down.
- The service kept logging and processing normally for six days while carrying the 14 GB. The memory wasn't in use by anything — it was dead weight. And CPU sat slightly elevated the whole time.
That elevated CPU turned out to be a beautiful clue: it's the signature of a garbage collector repeatedly scanning a huge live heap it can't shrink. Which raises the real question —
Why didn't the GC clean it up?
The rule that explains this whole incident:
Go's garbage collector can only free memory that nothing references anymore. Reachable memory is, by definition, memory the program might still need — the GC must keep it, forever if necessary.
So this wasn't "Go being lazy about returning memory to the OS." Something had allocated millions of objects during those 22 minutes and kept a reference to every one of them.
We found it by reading two libraries' source code — specifically, the paths they take when Redis is down. Failure paths are the least-exercised, least-reviewed code in any dependency, and this bug needed two of them to line up.
Ingredient 1: asynq retries with zero backoff
asynq's processor polls Redis in a loop asking "any tasks for me?" Look at its error handling (v0.25.1, processor.go, condensed):
switch {
case errors.Is(err, errors.ErrNoProcessableTask):
// Queue is empty — normal. Sleep before polling again.
time.Sleep(p.taskCheckInterval/2 + jitter)
<-p.sema
return
case err != nil:
// ANY OTHER ERROR — e.g. "can't reach Redis"
p.logger.Errorf("Dequeue error: %v", err)
<-p.sema
return // ← no sleep. The outer for-loop calls this again IMMEDIATELY.
}
Empty queue? Sleep politely. Broken broker? Retry instantly, forever. While Redis was unreachable, asynq was hammering Dequeue in a hot loop.
On its own that's just wasted CPU — each failed attempt's memory should become garbage and get collected. Should.
Ingredient 2: go-redis leaks a "ticket" on every failed connection
go-redis maintains a connection pool. In v9.17.2, when the pool needs a connection, it creates a small bookkeeping object — a wantConn, holding a context, a cancel function, and a result channel — and appends it to an internal queue (internal/pool/pool.go):
w := &wantConn{
ctx: dialCtx, // context.WithTimeout → timer, cancelCtx
cancelCtx: cancel,
result: make(chan wantConnResult, 1),
}
...
p.dialsQueue.enqueue(w) // ← always enqueued
Now the bug. Search the package for who removes entries from dialsQueue: there is exactly one call site, and it runs only when a dial succeeds and hands its connection to a waiting requester. When a dial fails, the ticket gets marked "done"… and stays in the queue. Forever. The queue belongs to the pool; the pool belongs to the Redis client; the client lives as long as the process. Every ticket is permanently reachable. The GC is not allowed to touch it.
One more detail turns this from a trickle into a firehose: after PoolSize consecutive dial failures, go-redis stops dialing and fails fast, instantly returning the cached error:
if atomic.LoadUint32(&p.dialErrorsNum) >= uint32(p.cfg.PoolSize) {
return nil, p.getLastDialError()
}
Sensible optimization — except the ticket is enqueued before that check triggers the failure. So the failure path becomes microsecond-fast, asynq's no-backoff loop spins at full CPU speed, and every single spin permanently leaks one ticket.
Tight loop × small leak × 22 minutes = one very large number.
The receipts: 40,000 leaked tickets per second
Theory is cheap, so we reproduced it. Taskqueue on a dev machine, Prometheus + pprof wired up, and then we simulated the outage.
First attempt: docker stop redis. Nothing happened. Memory stayed flat at 35 MB. This was our favorite wrong result of the whole investigation, because the reason is so instructive: a stopped container refuses connections instantly, and connection refused is not a timeout error — go-redis handles it with internal retries and backoff, which paces the loop. The production outage was packets silently dropped — i/o timeout — and cached timeout errors skip the retry logic entirely.
The error type mattered more than the error.
Second attempt: drop the packets like the real outage did (iptables -j DROP on the Redis port). Within seconds:
21:14 RSS: 34 MB ← baseline
21:21 RSS: 1,703 MB
21:22 RSS: 2,082 MB
21:24 RSS: 3,950 MB
21:27 RSS: 4,906 MB ← ~1.2 GB/minute
Same error lines as production, down to the exact file and line. And the heap profile, ~100 seconds in, is the whole story in five rows:
Type: inuse_objects
7,167,240 pool.(*ConnPool).queuedNewConn ← wantConn structs + channels
4,442,693 context.WithDeadlineCause ← one timer-context per ticket
2,354,866 context.(*cancelCtx).Done ← cancel channels
... 98% of the heap, rooted under:
asynq.(*processor).exec → rdb.Dequeue → queuedNewConn
~4.4 million tickets in ~100 seconds — about 40,000 leaked tickets per second, at roughly 220 bytes of permanently-reachable heap each. Scale that to production's 22-minute outage and you get on the order of 50 million tickets ≈ 14 GB. The arithmetic closes.
The rest of the incident reproduced just as faithfully. Unblock Redis: growth freezes instantly, but live heap stays pinned — the plateau. Goroutine count flat at ~38 the whole time (this was never a goroutine leak — it was data). Restart the process: 40 MB. The only cure is the one we'd used in production without understanding it.
The twist: nobody's test matrix contained this bug
Here's the part that makes this a systems story rather than a "library X has a bug" story.
asynq v0.25.1 declares a dependency on go-redis v9.7.0 — a version that predates this entire async-dialing pool design. The wantConn queue didn't exist when asynq's error-handling loop was written. But our own go.mod pinned go-redis v9.17.2 for our other Redis usage, and Go's module resolution always picks the higher version for the whole binary. So asynq's hot error loop ran on top of a pool implementation it had never been tested against.
Neither library is "broken" in isolation. asynq's no-backoff retry was harmless on the go-redis it was built against. go-redis's ticket leak was slow-motion under normal request-driven traffic (the upstream reporter hit it gradually, under high concurrency). It took the combination — an unpaced retry loop driving a per-call leak at microsecond frequency — to produce a gigabyte a minute. The combination existed only in binaries like ours.
And it was a real upstream bug: go-redis issue #3678 reports exactly this object accumulation in v9.17.2, and PR #3680 (merged January 2026) fixes "zombie wantConn elements accumulating in the wantConnQueue" by purging done tickets on every enqueue. The fix ships in v9.20.1.
Our fix, after a week of forensics, two library source-audits, and a live reproduction:
- github.com/redis/go-redis/v9 v9.17.2
+ github.com/redis/go-redis/v9 v9.20.1
One line.
What we actually learned
1. Your dependencies' error paths are part of your system. Everything worked until Redis blinked for 22 minutes. The bug wasn't in any code we wrote — it lived in the interaction between two libraries' failure handling, a code path that only executes when infrastructure is already on fire.
2. "Go has a GC" does not mean "no memory leaks." Go leaks are almost never missing free() — they're accidental retention: an append to a process-lifetime slice, a map that only grows, a cache with no eviction. The GC faithfully preserved all 50 million of those tickets because the pool still pointed at them. Reachable is reachable.
3. Version skew between a library and its dependencies is a real risk surface. MVS silently upgraded asynq's Redis client by ten minor versions. Nobody tested that pairing, because that pairing existed only in downstream binaries.
4. The error type can matter more than the error. connection refused was harmless; i/o timeout was catastrophic — because one took the retry-with-backoff path and the other took the fail-fast path. If we'd only tested our reproduction with docker stop, we'd have wrongly exonerated the library.
5. Observability gaps decide how long incidents take to solve. We had logs (great — they found the trigger in an hour) but no /metrics and no pprof on the affected service. The root cause took days of source-code archaeology; with a heap profile it would have taken minutes — our eventual repro profile named the exact function on the first look. The service now ships pprof, Prometheus metrics (including asynq queue stats), and a dashboard whose memory panel plots RSS against live heap — the one comparison that instantly distinguishes "real leak" from "GC being lazy."
The next time a graph steps up 14 gigabytes and refuses to come down, we'll know in two minutes. Hopefully now, so will you.
Technical appendix: asynq v0.25.1 processor.go:152-193 (no-backoff error branch); go-redis v9.17.2 internal/pool/pool.go:577 (enqueue), pool.go:627 (only dequeue — success path), pool.go:370-372 (fail-fast); fixed by wantConnQueue.discardDoneAtFront() in v9.20.1. Reproduction: block the Redis port with iptables -j DROP (packet drop → dial timeout), not docker stop (→ connection refused, which is paced). Heap profiles captured via net/http/pprof, analyzed with go tool pprof -inuse_objects and -inuse_space.
Top comments (0)