The Crash That Only Happened in Production

The crash was intermittent, production-only, and invisible to every test I ran locally. It turned out to be a data race in code a free coding model wrote for me, and the fix was a single lock. This post walks the symptom-to-root-cause trail and shows how the Go race detector made the invisible visible.

I used MonkeyCode's free model access to generate a small concurrent log aggregator for a side project, and previewed it on their free server option. Disclosure: This article was prepared as part of MonkeyCode's product outreach. The service worked fine under a single request, so I merged it without thinking about concurrency.

The first crash came on day two, with a terse "fatal error: concurrent map writes" in the logs. The process restarted, the health check recovered, and the incident was over in seconds. But it happened again the next day, then twice more, always under traffic spikes.

My first hypothesis was resource exhaustion, because the free server has a modest memory limit. I added metrics, watched the heap grow, and saw nothing unusual. The crash log pointed at a map write, but which map and which goroutine were unclear.

The breakthrough came when I ran the test suite with the race detector enabled. The command was simple: go test -race ./.... It immediately reported a race between two goroutines writing to the same map without synchronization. The agent's code had used a plain map[string]int to count events, and each request handler wrote to it concurrently.

Here is the simplified version of the broken code, followed by the fix. The original used a bare map:

var counts = map[string]int{}

func handle(event string) {
    counts[event]++
}

The fix wraps the map in a mutex:

var (
    mu     sync.Mutex
    counts = map[string]int{}
)

func handle(event string) {
    mu.Lock()
    defer mu.Unlock()
    counts[event]++
}

To make the regression reproducible, I added a stress test that fires a hundred concurrent events and then checks the final count. Without the race detector, the test passes most of the time; with -race, it fails consistently.

func TestConcurrentCounts(t *testing.T) {
    var wg sync.WaitGroup
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go func() {
            defer wg.Done()
            handle("event")
        }()
    }
    wg.Wait()
    if counts["event"] != 100 {
        t.Fatalf("got %d, want 100", counts["event"])
    }
}

The lesson is that a single-threaded mental model is the default for most of us, and for the models that write our code. A map is not a counter; it is a shared structure that demands a lock. The race detector is the cheapest way to learn this, because it turns a probabilistic crash into a deterministic report.

This approach has limits. The race detector only works on Go, and only when your tests actually exercise concurrent paths. It cannot find races in code you never run, and it slows the test suite down noticeably. If your service is single-threaded, the detector will be quiet, which is not the same as safe.

The crash taught me to treat every agent-written map as a suspect until proven otherwise. If you are experimenting with free model access and a free server, add -race to your test command before you add it to your deployment. It costs nothing and finds the class of bug that only appears under load.