Running LLMs Locally on Consumer Hardware — Part 1: The Stack and First Benchmarks

Sven Welack — Fri, 31 Jul 2026 09:32:51 +0000

This is the first in a series of build-log posts documenting a local LLM project, in which models are run on owned consumer hardware rather than through a cloud API. The present entry covers the hardware, the software stack, and the benchmarks by which a primary model was selected.

The hardware

Two machines are used, both consumer-grade. All benchmarks reported below were obtained on the primary desktop.

Machine	CPU	RAM	GPU
Primary desktop	Ryzen 5950X	~80 GB DDR4	AMD RX 6900XT (16 GB)
Secondary box	Ryzen 5600G	32 GB	NVIDIA GTX 1060 (6 GB)

The software stack

Ollama serves as the model runner across two GPU vendors: ROCm 5.7 for the AMD card on the primary desktop, and CUDA for the NVIDIA card on the secondary box.

The primary model is Gemma 4 26B, a mixture-of-experts model with roughly 3.8B active parameters, quantized to Q4_K_M and occupying approximately 18 GB on disk. On the RX 6900XT it is run with an automatic GPU/CPU layer split, as the Q4 weights together with the KV cache exceed the 16 GB of available VRAM. Several Ollama settings were enabled to recover headroom: flash attention, and an 8-bit (q8_0) KV cache, the latter approximately halving the cache footprint. A free cloud tier is retained for occasional heavier tasks, though the objective is to run as much as possible locally.

Selecting a model: benchmarks

Before a primary model was chosen, the installed models were benchmarked. Two properties were of interest: throughput and output quality.

Throughput was measured on the primary desktop with a 500-word essay prompt (ollama run <model> --verbose):

Model	Tokens/sec	Duration	Tokens out
gemma4:26b	18.86	50.11s	945
gemma4-26b (64K ctx)	17.96	51.99s	934
mistral:7b-instruct	34.81	10.17s	354
llama3.2	57.11	3.99s	228

The smaller models are substantially faster; their token counts, however, are lower, and in practice their responses were correspondingly shallower.

Quality was assessed with a five-task suite spanning logic, coding, summarization, creative writing, and instruction-following:

Model	Score	Note
gemma4-26b (64K)	50/50	Flawless instruction-following
glm-4.7-flash	42/50	Missed only a complex string-formatting task
qwen3-coder:30b	40/50	Overthinks logic; hallucinated a fake country
llama3.2	28/50	Failed logic entirely
mistral:7b	26/50	Failed logic and negative constraints

A harder ten-task variant — incorporating a lipogram, a theory-of-mind question, and a riddle — was subsequently administered, on which Gemma 4 26B scored 99/100 while sustaining approximately 17 tokens/sec. This represents the local optimum: strong quality at a workable speed.

The principle that follows is that throughput and quality trade off against one another, and the fastest available model is rarely the appropriate default for substantive work. Gemma 4 26B is slower than the 7B models yet markedly more accurate, and it was therefore adopted as the primary model.

What follows

With a runner, a model, and a hardware baseline established, subsequent entries turn to the extraction of useful work from the setup. We intend to address context length and KV-cache trade-offs; the distinction between prefill and generation, and why a machine without a GPU can sustain conversation yet falter on large prompts; and the practice of keeping models resident to avoid cold-start reload penalties.

The measured figures are reported as-is, dead ends included. Part 2 will follow.

DEV Community: Sven Welack

Running LLMs Locally on Consumer Hardware — Part 1: The Stack and First Benchmarks

The hardware

The software stack

Selecting a model: benchmarks

What follows