Stop Letting Your Model Wander Between Sockets: Practical NUMA Pinning for Local LLM Inference on Linux

A two-socket box looks like one computer until you give it a memory-hungry workload. Then it becomes two computers connected by a wire that is slower than the one inside each half — and your inference server has no idea which half it is standing on.

I have watched a 32B GGUF load "successfully" and then decode at half the expected rate, with top showing plenty of idle CPU and free showing plenty of free RAM. Nothing was broken. The model was simply split across two NUMA nodes, and half of every token's memory traffic was crossing the inter-socket link.

This post is about the layer nobody tunes: where the pages actually live. We will look at what Linux memory policy really does to an mmap'd model file, how to pin a service declaratively with systemd, and how to prove the pinning worked instead of assuming it.

NUMA in one paragraph (the part that matters)

On a NUMA machine, each CPU socket has memory attached directly to it, and access to the other socket's memory goes through an interconnect. The kernel knows this because firmware describes it: relative cost is exposed as node distances.

$ numactl --hardware
available: 2 nodes (0-1)
node 0 cpus: 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 32 33 34 35 ...
node 0 size: 128884 MB
node 0 free: 96120 MB
node 1 cpus: 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 48 ...
node 1 size: 129019 MB
node 1 free: 127330 MB
node distances:
node   0   1
  0:  10  21
  1:  21  10

Distance 10 is "local" — the kernel's own reference point. A larger number means farther and more expensive. Those numbers are not a benchmark; they are the topology the scheduler and the allocator reason about. But they only matter if pages are placed deliberately.

lscpu gives you the same inventory in a different shape:

$ lscpu | grep -i numa
NUMA node(s):          2
NUMA node0 CPU(s):     0-15,32-47
NUMA node1 CPU(s):     16-31,48-63

Note the SMT sibling layout: node 0 holds CPUs 0-15 and their hyperthreads 32-47. If you pin a 16-thread job to "node 0" by listing physical CPUs, you can accidentally land only on the primary threads and leave the siblings idle — --cpunodebind takes the whole node, siblings included, which is usually what you want.

The policy model: task policy, not file policy

This is the crux, and it is where most advice goes wrong.

Linux has several scopes for memory policy. For a service, the one that matters is the task (thread) policy, because:

• It is inherited across fork() and preserved across execve(). That is why wrapping a binary in numactl works at all — the policy set before exec is still in force inside your inference server.
• It governs page allocation for memory-mapped files opened MAP_SHARED, regardless of access type. Model weights loaded by mmap are exactly that: file-backed, shared, read-mostly mappings.
• It is applied only when a new page is allocated. For anonymous memory, that is the first touch. For file-backed pages, it is when the page is first read into page cache.

That last point is the whole game. Placement is decided at fault time, by whichever thread happens to touch the page first. If your loader uses a thread pool and those threads are spread across both sockets, your weights land in a scatter pattern that no later tuning will fix without migrating pages.

And the default policy is local allocation: allocate on the node of the CPU that triggered the fault, falling back to a nearby node if that one is full. So with no policy at all, the layout of your model in memory is an accident of thread scheduling.

Two details from the docs that are easy to miss: a VMA policy (mbind()) applies to anonymous pages, and for a MAP_SHARED file mapping the pages follow the task policy instead. And if a cpuset restriction exists, it takes priority over memory policy. Set both, and the tighter one wins.

Diagnose before you tune

Start by watching where pages go, in pages (the default unit) or megabytes (-n, -m):

# system-wide: are we missing locally?
$ numastat -n
# per-process, by PID or by command-line pattern
$ numastat -p llama-server

The fields that matter:

• numa_hit — allocated on the intended node. Good.
• numa_miss — allocated here, but the process preferred elsewhere. Every one of these has a matching numa_foreign on another node.
• local_node / other_node — allocated on this node while the thread was running on this node / on a different node. A large other_node column is remote traffic you are paying for.
• interleave_hit — relevant only when you are using an interleave policy.

For a running server:

PID=$(pgrep -f llama-server | head -1)

# cpuset view (what the scheduler is allowed to use)
grep -E '^(Cpus_allowed_list|Mems_allowed_list)' /proc/$PID/status

# per-node distribution of that process's resident pages
numastat -p $PID

Mems_allowed_list tells you the cpuset restriction; it does not tell you the memory policy. The policy shows up as a distribution: if node 0 and node 1 both hold large shares of an mmap'd model that otherwise fits in one node, you were never pinned.

/proc/$PID/numa_maps annotates individual mappings, which is useful when you want to know which mapping got which treatment — but read it as a hint, not as a policy dump.

Four placement strategies, and when each is correct

1. Bind: one model, one node (strict)

numactl --cpunodebind=0 --membind=0 llama-server \
  --model /srv/models/qwen2.5-32b/qwen2.5-32b-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 -t 16 -c 8192

--membind is MPOL_BIND: memory comes only from the listed nodes. When several nodes are listed, allocation starts at the lowest node ID and moves on only when that node is full. Nothing is ever allocated outside the mask.

That strictness is the point and the danger: if the model plus KV cache does not fit, allocations fail rather than silently going remote. You will see ENOMEM-style failures or a process that dies during load instead of a slow process that works. Size the node honestly first:

# node capacity, in MB, from the same inventory you read earlier
numactl --hardware | grep -E 'node [0-9]+ size'

2. Preferred: soft affinity (usually the better default)

numactl --cpunodebind=0 --preferred=0 llama-server --model /srv/models/model.gguf

MPOL_PREFERRED tries the preferred node first and falls back to nearby nodes when it runs low. You keep locality in the normal case without turning a slightly-too-large model into a load failure. For a homelab box where you occasionally bump context size, this is the setting I would reach for.

3. Interleave: the model is bigger than any single node

numactl --interleave=0,1 llama-server --model /srv/models/model.gguf -t 32

MPOL_INTERLEAVE round-robins pages across the listed nodes. It optimizes for bandwidth over latency — and the kernel docs are blunt that this does not make a single page faster: accesses to one page are still limited to one node's bandwidth. What you gain is that both memory controllers and both halves of the interconnect share the load instead of one socket becoming the bottleneck.

Do not combine interleave with --cpunodebind to a single node. You would be forcing every thread onto one socket while deliberately spreading its memory across both — the worst of both layouts.

4. Local: let first-touch decide, but deterministically

numactl --localalloc llama-server --model /srv/models/model.gguf

MPOL_LOCAL is local allocation with fallback. It only becomes interesting once you have constrained the CPUs, because then every fault comes from the same node and first-touch placement is predictable instead of random.

Make it declarative with systemd

Wrapping ExecStart in numactl works, but it is easy to lose: someone edits the unit, the wrapper disappears, and the regression is invisible. systemd has had native NUMA policy since v243, and it composes with the cpuset controller.

# /etc/systemd/system/llama-pinned.service
[Unit]
Description=llama.cpp server (pinned to NUMA node 0)
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
User=llm
Group=llm
WorkingDirectory=/srv/models/qwen2.5-32b

# Memory policy: MPOL_BIND to node 0
NUMAPolicy=bind
NUMAMask=0

# CPU affinity derived from the same node mask
CPUAffinity=numa

# Hard cpuset fence (cgroup v2 cpuset controller)
AllowedMemoryNodes=0

ExecStart=/usr/local/bin/llama-server \
  --model /srv/models/qwen2.5-32b/qwen2.5-32b-q4_k_m.gguf \
  --host 127.0.0.1 --port 8080 -t 16 -c 8192

Restart=on-failure
RestartSec=3
LimitNOFILE=65535

[Install]
WantedBy=multi-user.target

Notes on those directives, straight from the man pages:

• NUMAPolicy= accepts default, preferred, bind, interleave, or local. NUMAMask= takes the node list, with the same syntax as a CPU list, or all. For preferred, expect exactly one node.
• CPUAffinity=numa derives the allowed CPU range from NUMAMask=, so the CPU and memory masks cannot drift apart.
• AllowedCPUs= / AllowedMemoryNodes= drive the cgroup v2 cpuset controller. The man page is explicit that they do not guarantee usage, because a parent unit can restrict further — the effective values are reported as EffectiveCPUs= and EffectiveMemoryNodes=.

Also worth knowing: StartupAllowedCPUs= and StartupAllowedMemoryNodes= apply the restriction only during startup and shutdown, which is genuinely useful if you want the loader to have the whole machine while the steady-state server stays on one node.

If you would rather not write a unit, run a one-off with the same properties:

sudo systemd-run --unit=llama-lab \
  --property=NUMAPolicy=preferred --property=NUMAMask=0 \
  --property=AllowedMemoryNodes=0 \
  /usr/local/bin/llama-server --model /srv/models/model.gguf --port 8080

Proving it worked

Three checks, in increasing order of trustworthiness.

# 1. What systemd thinks it applied
systemctl show -p EffectiveCPUs -p EffectiveMemoryNodes llama-pinned.service

# 2. What the kernel says the process is actually allowed to use
PID=$(systemctl show -p MainPID --value llama-pinned.service)
grep -E '^(Cpus_allowed_list|Mems_allowed_list)' /proc/$PID/status

# 3. Where the pages ended up, after a warm-up request
curl -s http://127.0.0.1:8080/completion \
  -d '{"prompt":"warm up","n_predict":16}' >/dev/null
numastat -p $PID

Check 3 is the one that counts. If the model is 20 GB and node 0 shows roughly 20 GB resident for that process while node 1 shows near zero, you are pinned. If both columns are large, your policy did not take — usually because a cpuset was wider than you thought, or because ExecStart still points at a wrapper that overrides it.

That warm-up request matters more than it looks. Pages are placed when first touched, and pages already faulted in stay where they were allocated. A policy installed after the model is loaded does not retroactively move anything.

Measuring honestly

Do not trust a single number. Pull the same benchmark twice, unpinned and pinned, with repeats:

# baseline
llama-bench -m /srv/models/model.gguf -t 16 -p 512 -n 128 -r 3

# pinned
numactl --cpunodebind=0 --membind=0 \
  llama-bench -m /srv/models/model.gguf -t 16 -p 512 -n 128 -r 3

Two things will bite you if you skip them:

1. Thread count must fit the node. Pinning 32 threads to a 16-core node does not measure NUMA; it measures oversubscription. Match -t to the node's physical cores.
2. First-touch effects hide in short runs. A 128-token generation after a cold load measures page faults as much as it measures memory locality. Warm up, then repeat.

For a live server, the cheap signal is numastat -p deltas: if numa_foreign keeps climbing on the node you pinned to, the process is asking for memory it cannot get locally and something upstream is fighting your policy.

The failure modes I keep seeing

• "It says one node." Then there is nothing to pin — and no, that is not always a mistake. Many consumer boards report a single node; multi-die server CPUs may expose more depending on the firmware memory-interleaving mode. numactl --hardware is the ground truth, not the marketing spec sheet.
• CPU pinning without memory pinning. With the default policy, memory follows the faulting CPU, so CPU-only pinning often looks like it works. It breaks the moment a helper thread, a loader pool, or a socket worker faults pages before your affinity is in place.
• bind on a node that is already busy. Strict binding plus a co-tenant service on the same node equals allocation failures at 3 a.m. Use preferred, or fence the co-tenant with AllowedMemoryNodes= too.
• Forgetting -- with numactl. numactl --cpunodebind=0 --membind=0 -- process -l exists precisely because the child's own flags can collide with numactl's. If your server has a -l or -i flag, you need the separator.
• Containers. A container gets a cpuset, not a policy. Pass the nodes in: podman run --cpuset-mems=0 --cpuset-cpus=0-15 ... maps onto cpuset.mems and cpuset.cpus, and remember that the cpuset wins over any policy set inside.
• GPU weights are a different problem. If the model lives in VRAM, host memory policy only affects staging buffers, pinned host allocations, and the loader — all still ordinary kernel allocations that follow the task policy of the allocating thread. Check nvidia-smi topo -m for the PCIe/NVLink topology before blaming NUMA for GPU-side numbers.

Rollback

Everything here is runtime state attached to a process or a unit. Drop the NUMAPolicy=/NUMAMask= lines and run systemctl daemon-reload, or remove the numactl wrapper, and the next start is back to the kernel default. To un-migrate pages that were placed under a policy you no longer want, migratepages can move them, but restarting the server is almost always simpler and less surprising.

The habit worth keeping is the diagnostic order: read the topology, read the policy that is actually in force, then read where the pages landed. Two of those three are one command each, and they will save you from optimizing a scheduler that was never the problem.

Sources

• numactl(8) — https://man7.org/linux/man-pages/man8/numactl.8.html
• numastat(8) — https://man7.org/linux/man-pages/man8/numastat.8.html
• set_mempolicy(2) — https://man7.org/linux/man-pages/man2/set_mempolicy.2.html
• Kernel documentation, NUMA Memory Policy — https://docs.kernel.org/admin-guide/mm/numa_memory_policy.html
• systemd.exec(5), NUMAPolicy=, NUMAMask=, CPUAffinity= — https://www.freedesktop.org/software/systemd/man/latest/systemd.exec.html
• systemd.resource-control(5), AllowedCPUs=, AllowedMemoryNodes= — https://www.freedesktop.org/software/systemd/man/latest/systemd.resource-control.html
• cgroup v2 documentation (cpuset controller) — https://docs.kernel.org/admin-guide/cgroup-v2.html
• proc_pid_status(5), Cpus_allowed_list, Mems_allowed_list — https://man7.org/linux/man-pages/man5/proc_pid_status.5.html