Go Profiling in Production: pprof, Flamegraphs and Memory Leaks

You deploy a Go service that handles 200 req/s in staging just fine. In production, latency climbs after six hours, memory keeps growing, and you have no idea which goroutine is the culprit. Go's built-in profiling toolchain can answer this — but only if you know how to use it correctly without adding overhead or security holes.

Enabling pprof Without Exposing It to the Internet

The most common advice is to add import _ "net/http/pprof" and start an HTTP server. That works, but shipping a /debug/pprof endpoint on your public API is a concrete security risk — it exposes memory contents, goroutine stacks, and CPU traces to anyone who can reach the port. Restrict it to localhost or a separate internal port with an explicit listener:

package main

import (
    "log"
    "net"
    "net/http"
    _ "net/http/pprof"
)

func startProfilingServer() {
    ln, err := net.Listen("tcp", "127.0.0.1:6060")
    if err != nil {
        log.Fatalf("pprof listen: %v", err)
    }
    go func() {
        if err := http.Serve(ln, nil); err != nil {
            log.Printf("pprof server: %v", err)
        }
    }()
}

Then from your workstation: ssh -L 6060:localhost:6060 prod-host and open http://localhost:6060/debug/pprof/ locally. No firewall changes, no ingress rules.

If you need in-process profiling for a CLI tool or a short-lived batch worker, use runtime/pprof directly — no HTTP server required:

import (
    "os"
    "runtime/pprof"
)

func main() {
    f, _ := os.Create("cpu.prof")
    pprof.StartCPUProfile(f)
    defer pprof.StopCPUProfile()

    runBatchJob()
}

Reading a CPU Profile Without Getting Lost

Once you have a profile file or a live endpoint, go tool pprof does the heavy lifting:

# From a file
go tool pprof cpu.prof

# From a live endpoint (30-second sample)
go tool pprof http://localhost:6060/debug/pprof/profile?seconds=30

Inside the interactive shell, top10 gives you the hottest functions by self time. But the number that actually matters is cumulative (cum) — the total time a function and everything it called consumed. A small self-time with a huge cumulative time means you have a hot inner loop further down the stack.

For flamegraphs, add -http=:8080 to open an interactive browser view:

go tool pprof -http=:8080 http://localhost:6060/debug/pprof/profile?seconds=30

The flamegraph view shows call stacks as horizontal bars. Width represents time. Look for wide bars near the top of the stacks — those are the hot leaves that the CPU is actually spending cycles on. If you see runtime.mallocgc eating 15–20% of CPU time, you have an allocation pressure problem, not a logic problem. The fix is fewer short-lived allocations, not faster algorithms.

Tracking Memory Leaks with the Heap Profile

Memory leaks in Go are usually not "classic" leaks (forgotten malloc). They're more subtle:

• Goroutines blocked waiting on a channel that never gets closed
• Global map[string]anything that grows without a TTL or eviction
• time.Ticker created inside a function that never calls .Stop()
• Slice backing arrays kept alive by a single-element subslice

The heap profile gives you two views:

# inuse_space = memory currently held by live objects (the leak view)
go tool pprof -inuse_space http://localhost:6060/debug/pprof/heap

# alloc_space = cumulative allocations since process start (the churn view)
go tool pprof -alloc_space http://localhost:6060/debug/pprof/heap

Inside pprof, use list <function_name> to annotate source lines with allocation counts. If one line in a hot path allocates a new []byte on every call, that's a sync.Pool candidate.

For goroutine leaks specifically, the goroutine endpoint is faster to check:

curl http://localhost:6060/debug/pprof/goroutine?debug=2

This dumps every goroutine's full stack trace to stdout. If you see 2,000 goroutines all stuck at chan receive and the channel should be drained by a single consumer, the consumer is dead or blocked — and every new request created a new goroutine that's now stuck waiting.

Continuous Profiling: Building Your Own Baseline

One-off profiles are reactive — you profile after the incident. Continuous profiling means sampling the heap at regular intervals so you have a baseline to compare against. Tools like Pyroscope or Google Cloud Profiler handle this at scale, but for a self-hosted service you can wire it up in 20 lines:

package profiling

import (
    "fmt"
    "os"
    "runtime/pprof"
    "time"
)

// StartPeriodicHeapDump writes a heap profile to dir every interval.
// Use interval >= 1 minute in high-traffic services to avoid GC pauses.
func StartPeriodicHeapDump(dir string, interval time.Duration) {
    go func() {
        ticker := time.NewTicker(interval)
        defer ticker.Stop()
        for range ticker.C {
            fname := fmt.Sprintf("%s/heap-%d.prof", dir, time.Now().Unix())
            f, err := os.Create(fname)
            if err != nil {
                continue
            }
            pprof.WriteHeapProfile(f)
            f.Close()
        }
    }()
}

Call StartPeriodicHeapDump("/var/log/myapp/profiles", 5*time.Minute) at startup. When an incident happens at 3am and memory is already at 4 GB, you have a heap dump from four minutes earlier that shows what it looked like at 800 MB — before the spike hit.

One caveat: writing a heap profile acquires the runtime.ReadMemStats lock, which briefly stops the world. At 1000+ req/s, stick to once per minute or route this to a replica.

What to Ignore in Flamegraphs

Flamegraphs can be noisy. A few patterns to filter mentally:

• runtime.schedule, runtime.findrunnable — normal GC and scheduler overhead. Only a problem if they exceed 25–30% of total CPU.
• syscall.Read / syscall.Write — IO wait. These appear in wall-clock CPU profiles but aren't actionable there. Use blocking profiles (/debug/pprof/block) for IO analysis.
• encoding/json.Marshal — if this is your top offender, switch to go-json or sonic, or reduce call frequency by caching marshaled responses.

For services that handle untrusted input at volume, resource exhaustion bugs (unbounded request body reads, missing read deadlines) typically don't appear in profiles taken at normal load — they only surface under attack traffic. A security hardening checklist reviewed before deployment catches the obvious ones before they become production incidents.

The Takeaway

Go's profiling toolchain is already built in — you just need to wire it safely and use it regularly. The practical setup is:

1. pprof HTTP handler bound to 127.0.0.1, accessed via SSH tunnel
2. Periodic heap dumps for pre-incident baselines
3. go tool pprof -http=:8080 for flamegraph visualization

For memory growth specifically, check the goroutine dump first — leaking goroutines are the single most common cause of steady memory growth in long-running Go services, and the goroutine endpoint shows them immediately. CPU problems come second, and flamegraphs make them obvious once you know how to read the width, not the height.

---

I run AYI NEDJIMI Consultants, a cybersecurity consulting firm. We publish free security hardening checklists — PDF and Excel.