Much of the discourse around running Large Language Models locally assumes access to high-end hardware workstation GPUs with 24 GB of VRAM or Apple Silicon unified memory.
In practice, a substantial portion of the developer community works on budget laptops or desktop systems constrained to 4 GB of VRAM. On this hardware tier, running local models is an exercise in managing razor-thin memory margins. Practitioners are forced to navigate a difficult trade-off: Does one run a small model at native 16-bit precision, or quantize a larger architecture down to 4-bit or 8-bit weights?
To investigate this systematically on consumer hardware,quantbench was developed as an open-source benchmarking tool built from the ground up for reproducible LLM measurement on constrained GPUs.
This article shares the engineering motivation behind building quantbench, the methodological pitfalls encountered along the way, and empirical observations across the Qwen2.5 model family.
Benchmarking Objectives and Challenges
Evaluating LLM quantization on local hardware often encounters several methodological challenges:
Confounding Runtime Variables: Comparing a quantized GGUF checkpoint running in C++ (
llama.cpp) directly against an uncompressed FP16 model in PyTorch introduces two simultaneous variables: the quantization format and the execution engine itself. Isolating the effect of quantization requires keeping the runtime engine constant.Incomplete Metric Sets: Benchmarks frequently measure token throughput while omitting first-token latency (TTFT), baseline memory overhead, or perplexity. Generation speed provides little value if quality degrades significantly, or if memory allocations silently page to host RAM.
Single-Prompt Evaluation Bias: Evaluating quality using a single prompt introduces substantial domain bias. Models can exhibit minimal perplexity on boilerplate code while showing higher loss on multi-step reasoning.
Core Architectural Requirements
To address these challenges, quantbench was designed around four specific criteria:
Single-Variable Control: The execution engine is held constant (PyTorch + Hugging Face Transformers) with identical greedy decoding loops, isolating weight format (FP16, INT8, NF4) as the single independent variable.
Process Isolation: Each run executes in an independent subprocess to prevent CUDA memory fragmentation and ensure consistent baseline VRAM measurements.
Multi-Domain Quality: Perplexity is measured across multiple textual domains using token-weighted log-likelihood aggregation.
Hardware Telemetry & Spill Detection: Device metrics are monitored via NVML to identify cases where memory allocations reach physical capacity and spill across the PCIe bus.
Methodological Controls
1. Evaluating Execution Order Effects
To observe and control for the potential effects of execution order on throughput and thermal behavior, benchmarks are executed across 3 complete passes in alternating sequence:
-
Pass 1: Ascending order (
0.5B ➔ 1.5B ➔ 3B) -
Pass 2: Descending order (
3B ➔ 1.5B ➔ 0.5B) -
Pass 3: Ascending order (
0.5B ➔ 1.5B ➔ 3B)
Aggregating the resulting metrics via the median and the 5th percentile (p5 tail throughput) prevents order-induced variations from skewing the results of any individual model variant.
2. Multi-Domain Token-Weighted Perplexity
To evaluate model quality across distinct linguistic tasks, four separate domains were selected:
Narrative Prose: Descriptive creative writing (
prompt.txt)Code: An algorithmic Python implementation (
code.txt)Analytical Reasoning: A step-by-step logic puzzle (
reasoning.txt)Factual: Scientific text (
nature.txt)
Per-prompt perplexities vary considerably by domain. Because prompts differ in token length, computing an arithmetic average of per-prompt perplexities would introduce statistical distortion. Overall perplexity is aggregated via token-weighted log-likelihood:
PPL_overall = exp( - sum(log_probabilities) / total_evaluated_tokens )
Experimental Results
The summary of the 27 runs is rendered below:

Empirical Observations
1. Performance of the 3B NF4 Configuration
Within the scope of this test matrix, Qwen2.5-3B in NF4 (4-bit NormalFloat) demonstrated a favorable balance between memory consumption and measured perplexity:
Perplexity: 3B NF4 achieved an overall perplexity of 5.890, compared to 6.212 for the 1.5B FP16 model and 8.069 for the 0.5B FP16 model. Across the tested prompt set, the 4-bit compressed 3B model retained lower perplexity than smaller architectures running at uncompressed precision.
VRAM Profile: The model occupied 2,445 MiB of peak VRAM, leaving approximately 1.6 GB of headroom within the 4096 MiB physical limit.
Throughput: It sustained a median generation speed of ~18–20 tokens/s with a first-token latency (TTFT) around 88–100 ms.
2. Latency Characteristics of bitsandbytes INT8
A consistent pattern emerged in the INT8 evaluation via bitsandbytes (LLM.int8()):
Quality Retention: INT8 preserved perplexity levels closely matching the FP16 baselines (e.g., 0.5B FP16: 8.069 vs. INT8: 8.061; 3B FP16: 5.475 vs. INT8: 5.603).
Throughput Profile: However, decode speeds decreased significantly, ranging between 6 and 10 tokens/s across all model sizes, alongside elevated TTFT values (e.g., ~150 ms on 1.5B).
This throughput profile reflects the internal mechanics of LLM.int8(): weights remain in 8-bit storage, but dynamic dequantization occurs during computation, accompanied by specialized routing for activation outliers. In an uncompiled PyTorch path on consumer-tier hardware, this dequantization cycle introduces a noticeable latency tax.
3. PCIe Memory Spillover on 3B FP16 ([SPILL])
When evaluating Qwen2.5-3B in FP16, the model reached 4,083 MiB of allocation against the 4,096 MiB device capacity. Rather than raising a fatal OutOfMemoryError, excess tensor pages were managed across the PCIe interface into system memory.
quantbench automatically flagged this state with a [SPILL] warning:
Generation speed dropped to 2.6–2.8 tokens/s.
First-token latency (TTFT) increased to >420 ms.
Because PCIe transfer speeds are substantially lower than dedicated GDDR6 bandwidth, memory spilling introduces an immediate performance drop that highlights the importance of tracking allocation thresholds explicitly.
Conclusion
A common assumption in local deployment is that running a smaller model at uncompressed precision (FP16) is inherently safer or higher quality than applying aggressive compression to a larger model.
The empirical observations from this benchmark demonstrate that unquantized precision is not inherently superior to quantized scale:
Compressing a 3-billion-parameter architecture down to 4-bit NormalFloat (NF4) achieved a lower overall perplexity across the evaluated domains than running 1.5B or 0.5B models at uncompressed FP16.
Parameter scale and architectural capacity provide a foundation of linguistic competence that largely survives 4-bit quantization, whereas smaller parameter counts exhibit higher perplexity regardless of numerical precision.
For memory-constrained hardware (such as 4 GB consumer GPUs), quantization should not be viewed merely as an undesirable compromise, but rather as an enabling mechanism to access more capable model architectures that would otherwise exceed physical hardware limits.
While precision trade-offs must always be validated against specific downstream workloads, these results indicate that on memory-limited hardware, prioritizing parameter scale through modern 4-bit formats offers a practical and effective strategy.
Source code quantbench
Top comments (1)
The result that stuck with me is the 3B NF4 model beating the 1.5B FP16 model on perplexity — 5.890 against 6.212 — while using only 2,445 MiB of peak VRAM and holding 18–20 tokens/s. That is a clean answer to the question you open with: quantizing a larger model down can beat running a smaller one at full precision.
What makes me trust the numbers is the method, not just the headline: holding the engine constant with PyTorch and greedy decoding, running each variant in its own subprocess, and the three alternating passes to control for order effects. The [SPILL] flag catching the 3B FP16 run at 4,083 MiB, crawling at 2.6–2.8 tokens/s with TTFT over 420 ms, is a nice touch too.
Did the ranking hold across all four perplexity domains, or did code or reasoning flip it anywhere?