Memory-Mapped Caching in Go: mmap Mechanics, Page Fault Latency, and the Eviction Boundary Problem

Memory-Mapped Caching in Go: mmap Mechanics, Page Fault Latency, and the Eviction Boundary Problem

Application-layer caches built on map[string][]byte or Redis are the default answer for hot-path read amplification. They work until they don't: GC pressure from large heap allocations, network RTT jitter on Redis reads, or eviction policies that don't align with access patterns. A narrower alternative—memory-mapped files—trades explicit cache management for OS-level page management with zero-copy reads and off-heap storage. The tradeoff surface is poorly understood in Go services. This article works through the mechanics, the failure modes, and where the design boundary sits.

What mmap Actually Gives You

When you call mmap(2) on a file, the kernel maps file-backed pages into the process virtual address space. No data is read from disk at mapping time. Accesses to unmapped pages trigger a minor fault (page already in the OS page cache) or a major fault (page must be read from storage). The kernel manages this transparently. In Go you call into this via syscall.Mmap or the golang.org/x/sys/unix package.

package filecache

import (
    "fmt"
    "os"
    "syscall"
)

type MmapRegion struct {
    data []byte
}

func OpenRegion(path string) (*MmapRegion, error) {
    f, err := os.Open(path)
    if err != nil {
        return nil, fmt.Errorf("open: %w", err)
    }
    defer f.Close()

    fi, err := f.Stat()
    if err != nil {
        return nil, fmt.Errorf("stat: %w", err)
    }
    size := int(fi.Size())
    if size == 0 {
        return nil, fmt.Errorf("empty file")
    }

    data, err := syscall.Mmap(
        int(f.Fd()),
        0,
        size,
        syscall.PROT_READ,
        syscall.MAP_SHARED,
    )
    if err != nil {
        return nil, fmt.Errorf("mmap: %w", err)
    }
    return &MmapRegion{data: data}, nil
}

func (r *MmapRegion) Slice(offset, length int) []byte {
    return r.data[offset : offset+length]
}

func (r *MmapRegion) Close() error {
    return syscall.Munmap(r.data)
}

The slice returned by Slice points into the OS page cache, not the Go heap. The GC does not scan it. Allocation pressure from large in-process caches disappears. The data is accessible to any goroutine without a copy, and reads that hit warm pages are essentially a memcpy from L3 or RAM, not a syscall.

Where Go's Runtime Intersects

The slice header—pointer, length, cap—lives on the Go heap. The backing array does not. This creates a subtle escape problem: if you return a subslice from Slice and the caller stores it in an interface or passes it to json.Unmarshal, Go's escape analysis sees the header escape but not the data region. The GC won't move it (the runtime only moves heap objects in specific contexts, and mmap pages are not heap-managed), but the GC will not help you if a goroutine holds a stale slice into a region you've already Munmap'd. That's a use-after-free.

The correct contract is: the region owns the lifetime; callers borrow slices within a request-scoped duration and never store them beyond that scope. Enforce this structurally, not by convention.

A second interaction: syscall.Mmap causes the Go runtime's cgo signal handling path to activate for SIGBUS. If a file is truncated after mapping, any access to the now-invalid pages generates SIGBUS, which the Go runtime surfaces as a panic with a specific signal message rather than a clean error. Your service crashes. Validate file size before mapping and use atomic file replacement (write to temp, rename(2)) rather than in-place truncation.

Page Fault Latency Budget

The latency model for mmap reads is bimodal:

• Warm path (page in page cache): ~50–200ns depending on whether the page is in L1/L2/L3 or main memory. This is on par with a Redis pipelined read within the same host and orders of magnitude faster than a cross-AZ Redis call.
• Cold path (major fault): Kernel must issue an I/O to the underlying block device. On NVMe this is 100–300µs. On EBS gp3 in AWS it is 200µs–2ms depending on burst credit state. On io1/io2 it is more predictable but not fast enough for a 1ms latency budget on the hot path.

If your service has a 5ms P99 latency budget and you are serving a 2GB index file with a working set larger than available memory, major faults will blow that budget unpredictably. The OS will make eviction decisions based on global page reclaim pressure across all processes on the host, not your per-request deadline. This is the eviction boundary problem: you do not control when pages leave the page cache.

You can influence—but not guarantee—residency with madvise(2). MADV_WILLNEED hints the kernel to prefetch pages asynchronously. MADV_SEQUENTIAL adjusts readahead. Neither is a hard reservation.

func Prefetch(data []byte) error {
    _, _, errno := syscall.Syscall(
        syscall.SYS_MADVISE,
        uintptr(unsafe.Pointer(&data[0])),
        uintptr(len(data)),
        syscall.MADV_WILLNEED,
    )
    if errno != 0 {
        return errno
    }
    return nil
}

mlock(2) pins pages in RAM and prevents eviction entirely, but it requires elevated privileges (CAP_IPC_LOCK), consumes locked memory quota (ulimit -l), and prevents the OS from using that memory for anything else. In containers on ECS or EKS, mlock limits are often restrictive or unavailable. Do not rely on it in containerized services without explicit platform configuration.

Operational Failure Modes in Production

Memory overcommit on shared hosts. Kubernetes nodes and ECS hosts run multiple containers. The page cache is shared. When a co-resident service allocates heavily, the kernel's kswapd begins reclaiming pages. Your mmap region, being file-backed, is eligible for eviction without any signal to your process. The next access triggers a major fault, and your latency spikes. This is the same failure mode as noisy neighbor problems on Redis, just lower-level.

File rotation without remap. A common pattern for serving updated index files is to replace the file on disk and expect the mmap to pick up changes. With MAP_SHARED, the mapping reflects the underlying inode. If you replace the file via rename(2) (creating a new inode), the existing mapping continues to point at the old inode's pages. The process keeps serving stale data silently. You must Munmap, open the new file, and remap. During the remap window, you need a fallback read path or the region must be read-locked against concurrent access.

NUMA topology. On multi-socket servers (less common in cloud but present on bare metal or large instance types), page allocation is NUMA-aware. Pages for the mmap region may reside on a remote NUMA node from the goroutine accessing them, adding 50–100ns per access due to cross-socket memory latency. You can observe this via numastat and influence placement with mbind(2), but the window for that optimization is narrow in typical AWS deployments.

Where mmap-Backed Caching Fits

The pattern works well for:

• Static or slowly rotating read indexes: ML feature stores, precomputed lookup tables, IP geolocation databases, compiled rule sets. Large, mostly read, rotated infrequently. The working set fits in memory on the target instance type.
• Zero-copy serving of binary blobs: Serving protobuf-encoded records or columnar data segments directly off a file without deserializing into heap structures. Combine with a fixed-offset index structure (array of [offset uint32, length uint32]) for O(1) lookup without malloc.
• Shared memory between processes on the same host: MAP_SHARED with a named file or POSIX shared memory allows multiple processes to read the same pages without redundant copies. Relevant if you run a sidecar or co-located agent.

It is the wrong tool when:

• Your cache working set is significantly larger than available RAM and access patterns are not strongly localized.
• You need fine-grained TTL-based invalidation on individual keys—there is no key abstraction, only byte ranges.
• You are running on EBS-backed storage with variable I/O latency and cannot tolerate the P99 impact of major faults.
• You need write visibility—concurrent writers to a MAP_SHARED region require your own synchronization; the OS does not provide it.

Decision Framework

Before adopting mmap as a caching layer, answer these questions:

1. Does the working set fit in instance memory with headroom? If the ratio of working set to available RAM exceeds ~0.6, major fault rate will be unpredictable under memory pressure. Measure with perf stat -e major-faults or /proc/vmstat under realistic load.
2. What is the file update frequency? If rotation happens more than once per minute, remap coordination adds complexity that likely exceeds the benefit. A Redis hash or an in-process sync.Map with a TTL wheel is simpler.
3. Is your latency budget fault-tolerant to P99 spikes? If your SLO is P99 < 2ms and you cannot bound major fault latency via instance type selection and mlock, the design is fragile.
4. What is the operational context? EKS or ECS with shared node pools increases noisy neighbor risk. A dedicated instance or a memory-optimized instance type (r7g, x2idn) with a reserved memory allocation changes the math.
5. Can you tolerate the Go lifetime contract? If slice lifetimes are difficult to bound in your codebase—heavy use of goroutines that outlive request context, interfaces that capture byte slices—the use-after-unmap risk may not be worth it.

When all five conditions are favorable, mmap-backed caches eliminate GC pressure, reduce allocator contention, and provide sub-microsecond read latency on warm pages with no network hop. When they are not, the opacity of OS page reclaim will produce latency anomalies that are difficult to attribute without kernel-level profiling tools.