Kernel Stack Memory: The Hidden Consumer Eating Your RAM

Here's something I ran into recently that might save you a debugging session.

You're running a high-concurrency server. Memory usage is creeping up, but your application heap looks fine. top shows the process RSS growing, but pmap or /proc/<pid>/smaps doesn't reveal an obvious leak in the heap, stacks, or mapped files. You're left wondering where the memory went.

The answer might be kernel stacks.

The Problem

Every thread in your process has a kernel stack. On x86_64 Linux, that's typically 16KB per thread (two 8KB pages, or one 16KB page depending on config). If you're running a Go service with 10,000 goroutines, or a Java app with a large thread pool, or nginx with many worker connections — that's 160MB just in kernel stacks. And it's not accounted for in your application's memory metrics.

This came up at the 2026 Linux Plumbers Conference. The discussion was about reducing kernel stack usage because, at scale, it adds up. A thread doing a simple system call might only use a few hundred bytes of its kernel stack, but deep call chains (filesystem, network, security modules, BPF) can consume most of it.

Where It Shows Up

You'll see it in:

• RSS growth that doesn't correlate with your application's allocations
• /proc/<pid>/status — VmRSS includes kernel stacks, VmSize doesn't always make it obvious
• cgroup memory stats — memory.current in cgroup v2 includes kernel stacks charged to the cgroup

The tricky part: standard profilers (pprof, async-profiler, Java Flight Recorder) don't show kernel stack memory. It's outside your process's userspace view.

How to Measure It

Per-process:

# Rough estimate: thread count * 16KB (x86_64 default)
ps -o nlwp= -p <pid> | awk '{print $1 * 16 " KB"}'

System-wide:

# Total kernel stack memory across all processes
grep -r "^Threads:" /proc/*/status 2>/dev/null | awk '{sum += $2} END {print sum * 16 " KB"}'

With vmstat / slabtop:

slabtop -o | grep -i thread
# Look for thread_struct, task_struct, kmalloc-16k (or kmalloc-8k x2)

When It Matters

This isn't a problem for most workloads. But it becomes relevant when:

• High thread counts: Go services with thousands of goroutines, Java apps with large thread pools, event-driven servers with many connections
• Memory-constrained environments: Containers with tight memory limits where 100-200MB of "invisible" memory pushes you into OOM
• Density optimization: Running many services on a single node where kernel stack overhead compounds

Mitigations

1. Reduce thread count

The most direct fix. Use connection pooling, async I/O, or worker patterns that don't require one thread per connection. Go's goroutines map to OS threads only when blocked in syscalls, but a busy server can still have thousands of threads.

2. CONFIG_VMAP_STACK (kernel config)

Modern kernels (5.8+) support CONFIG_VMAP_STACK which allocates kernel stacks from virtually mapped memory with guard pages. This doesn't reduce memory, but it catches stack overflows reliably. Most distro kernels have this enabled.

3. THREAD_STACK_SIZE tuning (Go)

// Go 1.19+
debug.SetThreadStackSize(8192) // 8KB instead of default (OS-dependent)

Only do this if you're certain your Cgo calls and deep call chains fit. The default exists for a reason.

4. Monitor it

Add kernel stack estimation to your memory dashboards:

# Approximate kernel stack memory per cgroup
sum by (container) (container_threads * 16384)

The Practical Takeaway

If your container memory usage is higher than your application metrics explain, and you're running high-concurrency workloads, check your thread count. Kernel stacks are the hidden tax on thread-heavy architectures.

It's not a leak. It's not a bug. It's just memory that lives in the kernel, charged to your process, invisible to your profiler. Know it exists, measure it, and factor it into your capacity planning.

---

Schiff Heimlich | Sysadmin who checks /proc before blaming the app