Mutex Contention in Go: Finding It Before It Finds You

• Book: The Complete Guide to Go Programming
• Also by me: Hexagonal Architecture in Go — the companion book in the Thinking in Go series
• My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
• Me: xgabriel.com | GitHub

---

You've added more CPU. The graph didn't move. You doubled the
worker pool and throughput went up by ten percent, not a hundred.
Every core shows load, but the numbers say the service is doing
less work per core than it did with half the goroutines. Something
is serializing your parallel code, and the usual suspect is a lock
that every request has to walk through.

Go ships a profiler for exactly this. The mutex profile records
where goroutines block waiting to acquire a sync.Mutex or
sync.RWMutex. It is off by default, cheap to turn on, and it
points a finger at the single line holding your throughput hostage.
The trick is knowing how to read what it hands back, and what to do
once you know.

Turn the profiler on

The mutex profile is disabled until you set a sampling rate. One
call does it:

import "runtime"

func init() {
    runtime.SetMutexProfileFraction(1)
}

The argument is a sampling rate: 1 records every contention
event, 10 records roughly one in ten. On a hot service, 1 adds
measurable overhead, so start at 5 or 10 in production and use
1 locally or in a load test. Passing 0 leaves it off; passing a
negative number reads the current value without changing it.

To pull the profile over HTTP, register net/http/pprof:

import (
    "net/http"
    _ "net/http/pprof"
)

func main() {
    go func() {
        http.ListenAndServe("localhost:6060", nil)
    }()
    // ... the rest of your program
}

Now generate load, then grab the profile from a shell:

go tool pprof http://localhost:6060/debug/pprof/mutex

In a benchmark you don't need HTTP. -mutexprofile writes the
file directly:

go test -bench=. -mutexprofile=mutex.out
go tool pprof mutex.out

Reproduce the contention

Here is a cache that every request reads and some requests write.
A single sync.Mutex guards it. Under read-heavy load this is a
textbook hot lock.

type Cache struct {
    mu   sync.Mutex
    data map[string]int
}

func (c *Cache) Get(k string) (int, bool) {
    c.mu.Lock()
    v, ok := c.data[k]
    c.mu.Unlock()
    return v, ok
}

func (c *Cache) Set(k string, v int) {
    c.mu.Lock()
    c.data[k] = v
    c.mu.Unlock()
}

A benchmark that hammers Get from many goroutines:

func BenchmarkCache(b *testing.B) {
    c := &Cache{data: map[string]int{"x": 1}}
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            c.Get("x")
        }
    })
}

Run it with the mutex profile on:

go test -bench=BenchmarkCache -cpu=8 \
  -mutexprofile=mutex.out

Read the contention report

Open the profile and ask for the top entries:

$ go tool pprof mutex.out
(pprof) top

You'll see something close to this:

Showing nodes accounting for 4.20s, 100% of 4.20s total
      flat  flat%   sum%        cum   cum%
     4.20s   100%   100%      4.20s  100%  sync.(*Mutex).Unlock

Two things matter here. The unit is time spent blocked, not CPU
time. And the count points at Unlock, which surprises people. The
mutex profile attributes the delay to the goroutine that held the
lock and then released it, because that release is what unblocks
the waiters. So the line to fix is whatever the report shows sitting
under that Unlock in list or the graph, which is your Get and
Set above.

Use list to see it against source:

(pprof) list Cache.*Get

The annotated output puts the blocked time next to c.mu.Lock().
That confirms the diagnosis: goroutines are queueing on this one
mutex. Now you fix it.

First fix: RWMutex when reads dominate

Your Get only reads the map. Set writes it. A plain Mutex
forces readers to wait for each other even though two reads never
conflict. sync.RWMutex lets any number of readers hold the lock
at once and only blocks them when a writer wants in.

type Cache struct {
    mu   sync.RWMutex
    data map[string]int
}

func (c *Cache) Get(k string) (int, bool) {
    c.mu.RLock()
    v, ok := c.data[k]
    c.mu.RUnlock()
    return v, ok
}

func (c *Cache) Set(k string, v int) {
    c.mu.Lock()
    c.data[k] = v
    c.mu.Unlock()
}

Re-run the benchmark and the mutex profile. Under read-heavy load
the blocked time drops, because readers stop serializing against
each other.

RWMutex is not free, though, and it is not always a win. It
carries more bookkeeping than a Mutex, so under a short critical
section it can be slower than the plain lock it replaced. And if
writes are frequent, readers still stall behind every writer, so
you get the overhead without the payoff. Measure both. The rule of
thumb: reach for RWMutex when reads outnumber writes by a wide
margin and the critical section does real work. For a two-line map
lookup that's already fast, the extra bookkeeping can eat the gain.

Second fix: shard the lock

When both reads and writes are hot, one lock of any kind is the
ceiling. The answer is to stop having one lock. Split the data
across N shards, each with its own mutex, and pick the shard by
hashing the key. Two keys that land in different shards never
contend.

Start with the types and constructor: N shards, each an
independent map behind its own mutex.

import "hash/maphash"

const shardCount = 16

type shard struct {
    mu   sync.Mutex
    data map[string]int
}

type ShardedCache struct {
    seed   maphash.Seed
    shards [shardCount]*shard
}

func NewShardedCache() *ShardedCache {
    c := &ShardedCache{seed: maphash.MakeSeed()}
    for i := range c.shards {
        c.shards[i] = &shard{
            data: make(map[string]int),
        }
    }
    return c
}

The interesting part is shardFor: it hashes the key once and
maps it to a shard. Get and Set then lock only that shard.

func (c *ShardedCache) shardFor(k string) *shard {
    h := maphash.String(c.seed, k)
    return c.shards[h%shardCount]
}

func (c *ShardedCache) Get(k string) (int, bool) {
    s := c.shardFor(k)
    s.mu.Lock()
    v, ok := s.data[k]
    s.mu.Unlock()
    return v, ok
}

func (c *ShardedCache) Set(k string, v int) {
    s := c.shardFor(k)
    s.mu.Lock()
    s.data[k] = v
    s.mu.Unlock()
}

maphash gives you a fast, well-distributed hash from the standard
library, seeded once per cache. Sixteen shards turn one contention
point into sixteen, and the odds that two concurrent requests hit
the same shard drop with every shard you add. Powers of two are
handy because you can replace the modulo with a mask, but modulo is
fine and clearer to read.

Sharding has a cost too. You can no longer take a consistent
snapshot of the whole cache without locking every shard in order,
and iteration means walking all of them. If your access pattern is
key-at-a-time, that cost never comes due. If you need whole-map
operations, weigh it.

What the profile won't tell you

The mutex profile shows contention, not correctness. It will not
flag a mutex copied by value, a lock you forgot to release, or a
RLock upgraded to Lock in a way that deadlocks. For those, lean
on go vet's copylocks check and the race detector:

go test -race ./...

Contention and correctness are separate axes. The profiler answers
"where are goroutines waiting." The race detector answers "where is
access unsynchronized." A hot lock that's correct still caps your
throughput, and a fast path that's racy still corrupts data. You
want both clean.

One more thing worth internalizing: sharding and RWMutex treat
the symptom. Sometimes the real fix is to hold the lock for less
time, or not share the state at all. A per-goroutine accumulator
merged at the end beats any lock, because there's nothing to
contend on. Reach for that before you reach for sixteen mutexes.

The loop to run on Monday

1. Add runtime.SetMutexProfileFraction(1) behind a debug flag.
2. Run your real load, or a RunParallel benchmark that mimics it, with -mutexprofile.
3. go tool pprof, then top and list to find the line.
4. If reads dominate, try RWMutex and re-measure.
5. If reads and writes are both hot, shard and re-measure.
6. Run -race to confirm you didn't trade a slow-but-correct lock for a fast-but-broken one.

Contention is invisible until you turn the light on. The light is
one function call and a pprof command. Turn it on before your
throughput graph flatlines and someone starts guessing.

---

Lock contention is one of those Go topics that looks like a tuning
detail until it's the thing capping a whole service. The Complete
Guide to Go Programming goes deep on the runtime side of this — how
the scheduler parks and wakes goroutines, what sync.Mutex actually
costs, and why the mutex profile blames Unlock. Hexagonal
Architecture in Go is about the other half: keeping shared mutable
state behind the right boundary so a hot lock stays in one adapter
instead of leaking across your whole service.