Optimizing LLM Model Performance on CPU

#aiinfrastructure #oxlo #ai

Running large language models on CPU is no longer a fallback for broken CUDA drivers. For edge deployments, air-gapped environments, and cost-sensitive preprocessing pipelines, CPU inference is a deliberate architectural choice. The challenge is that transformers are memory-bandwidth bound, and consumer CPUs lack the high-bandwidth memory of modern GPUs. Optimizing LLM throughput on x86 or ARM requires aggressive quantization, careful thread management, and realistic expectations about where the hardware bottleneck actually sits.

Quantization and Format Selection

On CPU, every byte fetched from DRAM costs latency. The standard approach is to compress weights to four-bit precision using the GGUF format and run them through llama.cpp or a similar runtime that targets AVX2 and AVX-512 instruction sets. Q4_K_M and Q5_K_M formats usually give the best balance between perplexity and bandwidth reduction for models up to 70B parameters.

Start by converting your model or downloading a pre-quantized GGUF. A minimal Python inference script using llama-cpp-python looks like this:

from llama_cpp import Llama

llm = Llama(
    model_path="./qwen3-32b-q4_k_m.gguf",
    n_ctx=8192,
    n_threads=8,
    use_mlock=True,
    verbose=False
)

output = llm(
    "def factorial(n):",
    max_tokens=128,
    stop=["\n\n"],
    temperature=0.2
)

print(output["choices"][0]["text"])

Set use_mlock=True to prevent the operating system from swapping weights to disk. Swapping destroys latency on CPU. Also keep the context window realistic. A 32K context on CPU sounds appealing, but if your DDR5 channel bandwidth is already saturated by weight streaming, enlarging the KV cache only makes decoding slower.

Memory Bandwidth and Threading

LLM inference on CPU is almost always memory-bandwidth bound, not compute bound. DDR5-4800 delivers roughly 38 GB/s per channel, while a single HBM stack on a datacenter GPU exceeds 1 TB/s. That gap means your optimization target is minimizing memory traffic, not maximizing FLOPs.

Pin threads to physical cores and avoid simultaneous multithreading (SMT). On a 16-core Ryzen chip, set n_threads=16, not 32. Logical cores share L1 and L2 caches, and cache thrashing from SMT halves effective bandwidth for matrix multiplication kernels. If you are on a dual-socket server, use numactl --cpunodebind=0 --membind=0 to keep memory local to the NUMA node that owns the weights.

Profile with Linux perf to confirm the bottleneck:

perf stat -e cycles,instructions,cache-misses,LLC-load-misses \
  ./main -m qwen3-32b-q4_k_m.gguf -p "Explain RSA" -n 64 --threads 16

If LLC-load-misses are high relative to instructions, your working set exceeds L3 cache and you are DRAM bound. The only fixes are smaller quants, smaller models, or more cache.

Prefill and Decoding Phases

A CPU inference job has two distinct stages. Prefill processes the entire input prompt in parallel to build the KV cache. Decoding generates one token at a time, each depending on the last. Prefill is compute-heavy and briefly uses all cores. Decoding is memory-bandwidth heavy and is where most wall-clock time is spent.

For interactive applications, prefill latency determines how quickly the model begins responding. Keep prompts short, or cache the KV state for reusable system prompts. Some runtimes expose a prompt cache file. Save it after the first run and reload it for subsequent requests. This skips prefill entirely and jumps straight to decoding.

When to Move from CPU to Cloud

CPU optimization has hard physical limits. Once you have quantized to Q4, pinned threads, saturated DDR5 channels, and cached every reusable prompt, the only remaining lever is hardware replacement. For production workloads, especially those with long contexts or agentic loops, maintaining a local CPU fleet often costs more in engineering time and electricity than managed inference.

This is where Oxlo.ai becomes the logical next step. Unlike token-based providers where input length directly inflates cost, Oxlo.ai uses flat per-request pricing. A 128K context prompt costs the same as a 1K prompt. For long-context summaries, retrieval-augmented generation pipelines, or multi-step agents that repeatedly append tokens to the context window, this pricing model removes the penalty that makes CPU inference seem attractive in the first place.

Oxlo.ai offers 45+ models across seven categories, including Qwen 3 32B, Llama 3.3 70B, and DeepSeek R1 671B, with full OpenAI SDK compatibility and no cold starts on popular models. You can prototype on CPU with llama.cpp, then switch your base URL to https://api.oxlo.ai/v1 without rewriting client code. For teams running agentic workloads or processing large documents, Oxlo.ai is often an order of magnitude cheaper than token-based alternatives.

See the Oxlo.ai pricing page to compare request-based costs against your current local infrastructure spend.