DEV Community: plasmon

Ollama, LM Studio, and GPT4All Are All Just llama.cpp — Here's Why Performance Still Differs

plasmon — Wed, 08 Apr 2026 13:56:20 +0000

Ollama, LM Studio, and GPT4All Are All Just llama.cpp — Here's Why Performance Still Differs

When running local LLMs on an RTX 4060 8GB, the first decision isn't the model. It's the framework.

llama.cpp, Ollama, LM Studio, vLLM, GPT4All — plenty of options. But under an 8GB VRAM constraint, the framework choice directly affects inference speed. A 0.5GB difference in overhead changes which models you can load at all. One extra API abstraction layer adds a few ms of latency.

What follows is a comparison on identical hardware with identical models.

Frameworks and Evaluation Criteria

Framework Overview

frameworks = {
    "llama.cpp (CLI)": {
        "version": "b8233 (2026-03)",
        "backend": "CUDA + Metal + CPU",
        "quantization": "GGUF (Q2_K ~ FP16)",
        "API": "CLI / llama-server (OpenAI-compatible)",
        "strength": "Minimal overhead, maximum control",
    },
    "Ollama": {
        "version": "0.6.x",
        "backend": "llama.cpp (bundled)",
        "quantization": "GGUF (via Ollama Hub)",
        "API": "REST API + CLI",
        "strength": "Docker-like simplicity, easy model management",
    },
    "LM Studio": {
        "version": "0.3.x",
        "backend": "llama.cpp (bundled)",
        "quantization": "GGUF (GUI search)",
        "API": "OpenAI-compatible API + GUI",
        "strength": "GUI, beginner-friendly, function calling support",
    },
    "vLLM": {
        "version": "0.7.x",
        "backend": "Custom CUDA kernels + PagedAttention",
        "quantization": "AWQ, GPTQ, FP8, GGUF (v0.4.2+)",
        "API": "OpenAI-compatible API",
        "strength": "Batch processing optimization, server-oriented",
    },
    "GPT4All": {
        "version": "3.x",
        "backend": "llama.cpp (bundled)",
        "quantization": "GGUF",
        "API": "GUI + Python SDK",
        "strength": "Simplest setup, offline-first",
    },
}

The critical fact: Ollama, LM Studio, and GPT4All all use llama.cpp internally. The differences are purely in wrapper design. Only vLLM has its own CUDA kernels.

Evaluation Axes

evaluation_axes = {
    "Inference speed (t/s)": "Generation speed with identical model and quantization",
    "VRAM overhead": "VRAM consumed by the framework itself, excluding the model",
    "Cold start time": "Time to complete model loading",
    "API compatibility": "OpenAI API compatibility and quality",
    "Function calling": "Tool-use support and accuracy",
    "Setup difficulty": "Steps from install to first inference",
}

Inference Speed Comparison

Test Conditions

test_config = {
    "GPU": "RTX 4060 Laptop (8GB VRAM)",
    "model": "Qwen2.5-7B-Instruct Q4_K_M (4.7GB)",
    "prompt": "Explain the difference between TCP and UDP in 200 words",
    "max_tokens": 256,
    "temperature": 0.7,
    "context_length": 4096,
    "measurement": "Median of 3 runs",
}

Results

Framework            Prompt eval  Generation  TTFT    VRAM overhead
                     (t/s)        (t/s)       (ms)    (excl. model)
────────────────────────────────────────────────────────────────
llama.cpp (CLI)       ~800        32.1        120     ~0.3 GB
llama-server          ~780        31.5        135     ~0.4 GB
Ollama                ~750        30.2        180     ~0.5 GB
LM Studio             ~720        29.8        250     ~0.6 GB
GPT4All               ~680        28.5        300     ~0.7 GB
vLLM                  N/A*        N/A*        N/A*    ~1.5 GB+

* vLLM OOM with default settings on 8GB VRAM
  (PagedAttention KV cache pre-allocation consumes additional VRAM)

Analysis

speed_analysis = {
    "llama.cpp vs Ollama": {
        "gap": "32.1 vs 30.2 = 5.9%",
        "cause": "Ollama's REST API layer + model management daemon overhead",
        "practical_impact": "Negligible. Convenience offsets the difference.",
    },
    "llama.cpp vs LM Studio": {
        "gap": "32.1 vs 29.8 = 7.2%",
        "cause": "GUI + additional API abstraction layers",
        "practical_impact": "GUI benefits outweigh speed loss for most use cases",
    },
    "llama.cpp vs GPT4All": {
        "gap": "32.1 vs 28.5 = 11.2%",
        "cause": "Python SDK overhead + non-optimized default settings",
        "practical_impact": "Acceptable for beginners, room for optimization",
    },
    "vLLM": {
        "issue": "Cannot run 7B models on 8GB VRAM",
        "cause": "PagedAttention KV cache pre-allocation consumes additional VRAM",
        "use_case": "Tunable via gpu_memory_utilization, but practically needs 16GB+",
    },
}

# Bottom line: llama.cpp is fastest, but the gap is 6-11%
# On 8GB VRAM, the real differentiator is overhead (0.3GB vs 1.5GB)
# That overhead gap determines your maximum model size

When VRAM Overhead Becomes Fatal on 8GB

On 8GB VRAM, framework overhead directly dictates your maximum model size.

# Maximum model size per framework
max_model_size = {
    "llama.cpp": {
        "overhead": 0.3,
        "cuda_context": 0.3,
        "available_for_model": 8.0 - 0.3 - 0.3,  # 7.4 GB
        "max_model": "Qwen2.5-32B Q4_K_M (18GB) -> 7.4GB on GPU + 10.6GB CPU offload",
        "max_full_gpu": "Mistral-Nemo-12B Q4_K_M (7.2GB) -> barely fits",
    },
    "Ollama": {
        "overhead": 0.5,
        "cuda_context": 0.3,
        "available_for_model": 8.0 - 0.5 - 0.3,  # 7.2 GB
        "max_full_gpu": "7B Q4_K_M (4.7GB) -> comfortable, 12B -> tight",
    },
    "LM Studio": {
        "overhead": 0.6,
        "cuda_context": 0.3,
        "available_for_model": 8.0 - 0.6 - 0.3,  # 7.1 GB
        "max_full_gpu": "7B Q4_K_M (4.7GB) -> comfortable, 12B -> difficult",
    },
    "vLLM": {
        "overhead": 1.5,
        "cuda_context": 0.3,
        "available_for_model": 8.0 - 1.5 - 0.3,  # 6.2 GB
        "max_full_gpu": "Even 7B models have no headroom",
        "note": "Not recommended for 8GB",
    },
}

The overhead difference between llama.cpp and vLLM is 1.2GB. That 1.2GB could buy you:

Additional KV cache allocation to extend context length
Room to co-locate a BGE-M3 embedding model alongside your LLM
Higher GPU offload ratio for the model, speeding up inference

On 8GB VRAM, framework selection isn't a preference. It's an architectural decision.

Function Calling Support

As covered in my function calling article (separate article), tool use is the killer feature for local LLMs. Here's where each framework stands:

function_calling_support = {
    "llama.cpp (llama-server)": {
        "supported": True,
        "method": "OpenAI-compatible tools parameter",
        "GBNF_grammar": True,  # Enforces JSON output grammatically
        "quality": "Model-dependent. High accuracy with Qwen2.5-7B-Instruct + GBNF grammar",
        "limitation": "Requires manual server startup",
    },
    "Ollama": {
        "supported": True,
        "method": "OpenAI-compatible tools parameter (v0.4+)",
        "GBNF_grammar": False,  # No raw GBNF, but format parameter supports JSON Schema
        "quality": "Same as llama.cpp (identical backend)",
        "limitation": "No GBNF grammar, but structured output via format parameter with JSON Schema",
    },
    "LM Studio": {
        "supported": True,
        "method": "OpenAI-compatible tools parameter",
        "GBNF_grammar": True,  # JSON Schema enforcement
        "quality": "Testable through GUI, which is the main advantage",
        "limitation": "Backend equivalent to llama.cpp",
    },
    "vLLM": {
        "supported": True,
        "method": "OpenAI-compatible tools + Guided Decoding",
        "quality": "High accuracy via Guided Decoding",
        "limitation": "Needs gpu_memory_utilization tuning on 8GB, practically 16GB+ recommended",
    },
    "GPT4All": {
        "supported": False,
        "note": "No function calling support. Chat only.",
    },
}

GPT4All doesn't support function calling. It's unusable for agentic workflows. vLLM's Guided Decoding is powerful but impractical on 8GB. For function calling on 8GB VRAM, you're limited to the llama.cpp family -- direct, Ollama, or LM Studio.

Recommendations by Use Case

recommendations = {
    "Maximum performance (developers)": {
        "pick": "llama.cpp (CLI / llama-server)",
        "reasons": [
            "Minimal overhead (0.3GB)",
            "GBNF grammar enforces structured output",
            "Direct control over all parameters",
            "Per-layer GPU/CPU offload granularity",
        ],
        "downside": "Requires technical knowledge, no GUI",
    },
    "Convenient daily use": {
        "pick": "Ollama",
        "reasons": [
            "Docker-pull simplicity (ollama pull model)",
            "Background daemon, always available",
            "OpenAI-compatible API for drop-in replacement",
            "Within 6% of llama.cpp speed",
        ],
        "downside": "No GBNF grammar (JSON Schema via format param available), slightly larger overhead",
    },
    "GUI-driven experimentation": {
        "pick": "LM Studio",
        "reasons": [
            "Model search and download entirely in GUI",
            "Chat UI for real-time testing",
            "Function calling testable through the interface",
        ],
        "downside": "Higher memory footprint due to GUI layer",
    },
    "Easiest possible start (non-engineers)": {
        "pick": "GPT4All",
        "reasons": [
            "Install -> launch -> chat in minimal steps",
            "Fully offline",
            "No unnecessary configuration options",
        ],
        "downside": "No function calling, slowest speed, limited customization",
    },
    "Production / server deployment": {
        "pick": "vLLM (16GB+ GPU recommended) or llama-server",
        "reasons": [
            "vLLM: PagedAttention for efficient batch processing",
            "llama-server: Lightweight server that works on 8GB",
        ],
        "downside": "vLLM impractical on 8GB",
    },
}

The Verdict for 8GB

Question: What's the optimal framework for 8GB VRAM?

Answer: Depends on use case. But technically optimal is raw llama.cpp.

Why:
1. Minimum overhead (0.3GB) -> maximum usable VRAM
2. Fastest speed (+6-11% over other frameworks)
3. GBNF grammar enforces structured output -> highest function calling reliability
4. Per-layer GPU/CPU offload control

However:
- For daily use, Ollama's convenience outweighs the speed gap
- If you need a GUI, LM Studio is the only option
- vLLM is impractical on 8GB (needs 16GB+)
- GPT4All is unsuitable for agentic tasks (no function calling)

The total speed spread across all frameworks is within 11%.
Model selection matters far more than framework selection.
The gap between Qwen2.5-3B (2.0GB) and Qwen2.5-7B (4.7GB)
dwarfs the gap between llama.cpp and GPT4All.

If you're spending time agonizing over frameworks, spend it benchmarking models instead.

References

llama.cpp -- github.com/ggerganov/llama.cpp
Ollama -- ollama.ai
LM Studio -- lmstudio.ai
vLLM -- vllm.ai
GPT4All -- gpt4all.io
"Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) arXiv:2309.06180

99.8% of LLM Inference Power Isn't Spent on Computation

plasmon — Wed, 08 Apr 2026 10:14:44 +0000

99.8% of LLM Inference Power Isn't Spent on Computation

When people debate LLM inference bottlenecks, bandwidth and VRAM dominate the conversation. But of the five walls identified by LIMINAL (Davies et al., arXiv:2507.14397), the hardest one to break through is power.

Bandwidth scales by widening the bus (HBM4 did exactly that). Capacity scales by stacking more dies. But power is directly chained to physics. The era when process shrinks automatically reduced power consumption ended around 2006, when Dennard scaling collapsed.

# The collapse of Dennard Scaling
dennard_scaling = {
    "1970-2006": {
        "rule": "Smaller transistors -> lower voltage -> constant power per area",
        "result": "Performance/W improved for free with every node shrink",
        "benefit": "Moore's Law + Dennard's Law in sync -> exponential perf gains",
    },
    "2006-present": {
        "reality": "Voltage can't drop further (subthreshold leakage)",
        "result": "Shrinking transistors no longer reduces per-transistor power",
        "mitigation": "Dark silicon, heterogeneous design, power-constrained design",
    },
}
# Since 2006, chips physically cannot fire all transistors simultaneously.
# Unused sections are intentionally powered off (dark silicon) to manage thermals.

GPU Power Draw Over Time: A One-Way Escalator

Data Center GPUs

# NVIDIA GPU TDP (Thermal Design Power) progression
gpu_tdp = {
    "V100 (2017)":      {"tdp": 300, "unit": "W", "process": "12nm", "hbm": "HBM2"},
    "A100 (2020)":      {"tdp": 400, "unit": "W", "process": "7nm",  "hbm": "HBM2E"},
    "H100 (2022)":      {"tdp": 700, "unit": "W", "process": "4nm",  "hbm": "HBM3"},
    "H200 (2024)":      {"tdp": 700, "unit": "W", "process": "4nm",  "hbm": "HBM3E"},
    "B200 (2025)":      {"tdp": 1000, "unit": "W", "process": "4nm", "hbm": "HBM3E"},
    "Next gen (2026-27)": {"tdp": "1200-1500?", "unit": "W", "process": "3nm", "hbm": "HBM4"},
}

# V100 to B200: process shrank 12nm -> 4nm (3 generations) but TDP rose 300W -> 1000W (3.3x)
# Process efficiency improved, but transistor count increases more than canceled it out
# B200's 1000W requires liquid cooling. Air cooling hits its ceiling around 350W.

Is Efficiency Keeping Up?

# Performance/W trend (inference throughput basis)
efficiency_trend = {
    "V100":  {"perf_per_watt": 1.0,  "relative": "baseline"},
    "A100":  {"perf_per_watt": 2.5,  "relative": "2.5x vs V100"},
    "H100":  {"perf_per_watt": 4.2,  "relative": "4.2x vs V100"},
    "B200":  {"perf_per_watt": 6.0,  "relative": "6.0x vs V100 (estimated)"},
}

# Perf/W improved 6x over 8 years
# Absolute performance improved 30-50x over the same period
# Translation: most of the performance gains came from "burning more watts"
# Efficiency gains explain only ~20% of the performance improvement

The implication is blunt: LLM inference performance growth depends on dumping more power in. Efficiency alone doesn't cut it.

Power Cost Per Token

Decode Power Breakdown

# Power decomposition for decoding a 70B model (FP16)
decode_power_breakdown = {
    "Weight reads (HBM)": {
        "data": "140 GB per token",
        "HBM_power": "HBM3E: ~20 pJ/bit -> 140e9 * 8 * 20e-12 = 22.4 mJ/token",
        "note": "HBM energy cost scales linearly with bandwidth",
    },
    "Matrix ops (GPU cores)": {
        "ops": "~140 GFLOP per token (weight matmul)",
        "GPU_power": "H100: ~0.3 pJ/FLOP -> 140e9 * 0.3e-12 = 0.042 mJ/token",
        "note": "Actual computation energy is 1/500th of the data reads",
    },
    "KV cache reads": {
        "at_32K": "~8 GB per token (all layers)",
        "power": "8e9 * 8 * 20e-12 = 1.28 mJ/token",
        "note": "Scales linearly with context length",
    },
}

# The stunning ratio:
# Data movement: 23.7 mJ/token (99.8%)
# Computation:    0.042 mJ/token (0.2%)
# Nearly ALL power in LLM inference goes to "moving data around"

This ratio breaks most people's intuition. GPUs are thought of as "compute" devices, but during LLM inference, 99.8% of the power goes to everything except computation -- reading data from memory and shuffling it across the chip.

Datacenter-Scale Power Consumption

# Power estimate for a GPT-4 class service
datacenter_power = {
    "assumptions": {
        "model": "~1.8T parameters (MoE, estimated)",
        "queries_per_day": 100_000_000,  # 100M queries/day
        "avg_tokens_per_query": 500,
        "gpu": "H100 (700W)",
        "throughput_per_gpu": "~150 tokens/s (estimated, with batching)",
    },
    "calculation": {
        "total_tokens_per_day": "50B tokens",
        "tokens_per_second": "~578,703 t/s",
        "gpus_needed": "578703 / 150 = ~3,858 GPUs",
        "power": "3858 * 700W = 2.7 MW (GPUs only)",
        "with_cooling_network": "2.7 MW * 1.5 (PUE) = ~4.0 MW",
        "annual_power": "4.0 MW * 8760h = ~35 GWh/year",
    },
    "cost": {
        "electricity": "35 GWh * $0.05/kWh = ~$1.75M/year (electricity only)",
        "per_query": "$1.75M / 365 / 100M = ~$0.000048/query",
        "per_1k_tokens": "~$0.001 (electricity cost portion only)",
    },
}

GPU power alone: 35 GWh per year. That's roughly equivalent to the annual consumption of 3,500 US households. And this is inference only, for one service. Training costs 10x more.

Three Constraints the Power Wall Imposes on LLM Inference

Constraint 1: Scale-Out Hits a Ceiling

# Datacenter power constraints
datacenter_constraints = {
    "typical_rack_power": "20-30 kW per rack",
    "h100_per_node": "8 GPUs (DGX H100 node)",
    "node_power_h100": "8 * 700W + CPU/NVSwitch/NIC = ~10 kW (per node)",
    "b200_per_node": "8 GPUs (DGX B200 node)",
    "node_power_b200": "8 * 1000W + CPU/NVSwitch/NIC = ~14 kW (per node)",

    "problem": "Adding more GPUs doesn't help if the datacenter's power supply is the bottleneck",
    "reality": "Most existing datacenters are 20MW class. New builds take 3-5 years",
    "trend": "Microsoft and OpenAI are planning 1GW-class datacenters (one nuclear reactor's worth)",
}

The power wall puts a hard physics cap on the "just buy more GPUs" strategy.

Constraint 2: Thermal Limits on Consumer Devices

# RTX 4060 8GB: power and thermal reality
rtx4060_thermal = {
    "TDP": "115W (laptop variant)",
    "GPU_die_area": "~159 mm2",
    "power_density": "115 / 159 = 0.72 W/mm2",

    "During LLM inference": {
        "typical_power": "60-80W (not full load, but memory access is heavy)",
        "memory_controller": "Sustaining 272 GB/s bandwidth alone costs ~15W",
        "note": "Inference taxes the memory controller harder than the compute units",
    },

    "Practical limits": {
        "thermal_throttling": "Kicks in at high GPU temperatures, drops inference speed",
        "battery_operation": "Not viable (60Wh battery = under 1 hour)",
        "sustained_inference": "Fine with adequate cooling, constrained in thin laptops",
    },
}

Local LLM speed is "bandwidth-bound" -- everyone knows that. What's less discussed is that using bandwidth itself costs power. Maintaining 272 GB/s eats ~15W at the memory controller alone. As models grow and bandwidth demand climbs, power consumption follows proportionally.

Constraint 3: Tokens Per Watt

# Power efficiency across hardware
tokens_per_watt = {
    "RTX 4060 (Qwen2.5-32B Q4_K_M)": {
        "speed": "10.8 t/s",
        "power": "~70W",
        "efficiency": "10.8 / 70 = 0.154 t/s/W",
    },
    "RTX 4060 (Qwen3.5-4B Q4_K_M)": {
        "speed": "~50 t/s (estimated)",
        "power": "~40W (smaller models draw less power)",
        "efficiency": "50 / 40 = 1.25 t/s/W",
    },
    "M4 Mac mini (Qwen2.5-32B Q4_K_M)": {
        "speed": "~8 t/s",
        "power": "~30W (Apple Silicon efficiency)",
        "efficiency": "8 / 30 = 0.27 t/s/W",
    },
    "H100 (Llama-3-70B FP16, batch=32)": {
        "speed": "~768 t/s (32 parallel × 24 t/s, weight reads shared across batch)",
        "power": "700W",
        "efficiency": "768 / 700 = 1.10 t/s/W",
    },
}

# H100 batched at 1.10 t/s/W — comparable to RTX 4060 small model (1.25)
# On a single request, H100 does ~24 t/s (bandwidth-bound: 3.35TB/s / 140GB) -> 0.034 t/s/W
# A small model on RTX 4060 is 37x more power-efficient than single-request H100

Here's where it gets interesting. For workloads that can't batch -- personal use, real-time conversation -- a small local model beats datacenter GPUs on power efficiency. Qwen3.5 4B on an RTX 4060 at 1.25 t/s/W is 37x more efficient than an H100 serving a single request at 0.034 t/s/W. Even against batched H100 (1.10 t/s/W), the 40W laptop GPU with a 4B model wins. A 700W datacenter GPU losing to a laptop on power efficiency.

Three Approaches to Attacking the Power Wall

I wrote about a three-layer approach to the bandwidth wall (separate article) previously. The power wall has the same structure.

Layer 1: Chip-Level Power Efficiency
  Process shrinks (5nm -> 3nm -> 2nm): 10-20% improvement per generation
  Power delivery improvements (BSPDN backside power): 30% IR drop reduction
    -> smaller voltage margins -> power savings
  -> Reliable but slow. Post-Dennard improvements are incremental.

Layer 2: Architecture-Level Power Efficiency
  Sparse Attention: skip unnecessary ops -> direct power savings
  Quantization (INT8/INT4): fewer bits -> 1/4 to 1/16 compute power
  MoE (Mixture of Experts): top-2-of-8 activation -> memory bandwidth power at 1/4
  -> Software-level, immediately deployable

Layer 3: Changing the Compute Paradigm
  PIM (Processing-In-Memory): eliminate data movement -> attack the 99.8%
  Photonic computing: matrix ops via light interference -> near-zero power
  Analog compute (BrainScaleS-2 etc.): eliminate digital conversion
  -> Research stage, but the only fundamental fix

Effective power efficiency = L1 x L2 x L3
Layer 2 alone with MoE + INT4 quantization:
  Memory bandwidth power reduced to 1/4 (MoE top-2-of-8) x 1/4 (INT4) = 1/16
  A 70B model's effective power drops to 4.4B-model territory

Layer 2 delivers the fastest results. And it's already accessible to local LLM users. If you're running Q4_K_M quantized models, you're already benefiting from Layer 2. Choosing MoE models (Mixtral, DeepSeek-V3) is also a correct move from a power efficiency standpoint.

Local LLMs and Power Efficiency

The Consumer GPU Advantage

# The power efficiency paradox
power_paradox = {
    "conventional_wisdom": "Datacenter GPUs are more power-efficient",
    "reality": {
        "with_batching": "H100 at batch=32: 1.10 t/s/W — comparable to RTX 4060 small model",
        "single_request": "RTX 4060 small model is 37x more efficient (1.25 vs 0.034 t/s/W)",
    },
    "reason": {
        "H100_at_700W": "Most power consumed by idle memory banks and interconnect",
        "RTX_4060_at_40W": "Small model keeps the whole system at high utilization",
    },
    "conclusion": "For personal use, local LLMs are rational from a power perspective too",
}

Datacenter GPUs are designed around the assumption of concurrent multi-request processing. When a single user sends a single request, most of those 700W go to waste. Running a 4B model on an RTX 4060 is far more power-efficient for that use case.

Maximizing Power Efficiency on an RTX 4060 in Practice

1. Prefer smaller models
   -> Qwen3.5 4B (3.4GB, ~40W) delivers 8x the t/s/W of Qwen2.5-32B (18GB, ~70W)
   -> If the task allows it, always reach for the smallest viable model

2. Choose MoE models
   -> Same parameter count, but fewer active parameters means less power draw
   -> Mixtral 8x7B has 3-4x the parameter efficiency of a dense 47B

3. Keep context short
   -> KV cache reads consume power
   -> Use RAG to retrieve only what's needed; don't dump the full document

4. Idle the GPU when inference isn't needed
   -> RTX 4060 idle draw is ~10W
   -> Stopping background inference saves 50-60W instantly

References

"LIMINAL: Exploring The Frontiers of LLM Decode Performance" (2025) arXiv:2507.14397
Dennard, R. H. et al. "Design of Ion-Implanted MOSFET's with Very Small Physical Dimensions" (1974) IEEE JSSC
"The Efficiency Misnomer" -- Patterson et al. (2021) arXiv:2110.11822
NVIDIA B200 specifications (2025) -- TDP 1000W, HBM3E 8TB/s

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

plasmon — Wed, 08 Apr 2026 09:33:08 +0000

Q4 KV Cache Fit 32K Context into 8GB VRAM — Only Math Broke

The biggest VRAM hog in LLM inference isn't always the model weights.

Once context length grows, KV cache memory consumption overtakes the model itself. Llama-3-8B (Q4_K_M, 4.9GB) at 32K context burns roughly 4GB on KV cache alone. That's 9GB total. An RTX 4060 8GB can't hold it.

# KV cache memory calculation
def kv_cache_memory(
    n_layers: int,
    n_heads_kv: int,
    head_dim: int,
    context_length: int,
    dtype_bytes: int = 2,  # FP16
) -> float:
    """KV cache memory usage in GB"""
    # K + V, two tensors
    bytes_total = 2 * n_layers * n_heads_kv * head_dim * context_length * dtype_bytes
    return bytes_total / (1024 ** 3)

# Llama-3-8B (GQA: 8 KV heads)
llama3_8b = kv_cache_memory(
    n_layers=32,
    n_heads_kv=8,      # GQA: 32 attention heads -> 8 KV heads
    head_dim=128,
    context_length=32768,  # 32K
    dtype_bytes=2,      # FP16
)
# -> 4.0 GB

# Qwen2.5-32B (GQA: 8 KV heads)
qwen25_32b = kv_cache_memory(
    n_layers=64,
    n_heads_kv=8,
    head_dim=128,
    context_length=32768,
    dtype_bytes=2,
)
# -> 8.0 GB — KV cache alone fills 8GB. No room left for the model.

You can quantize the model to fit in VRAM, but if the KV cache stays FP16, it doesn't matter. The moment you stretch the context, VRAM overflows.

That's where KV cache quantization comes in.

What KV Cache Quantization Actually Is

How It Differs from Model Quantization

Model weight quantization (GGUF Q4_K_M, etc.) is well understood. KV cache quantization solves a fundamentally different problem.

# Two types of quantization compared
quantization_types = {
    "Model weight quantization": {
        "target": "Pre-trained parameters (offline)",
        "timing": "One-time conversion before inference",
        "quality_impact": "Well-studied. Q4_K_M is practical for most tasks",
        "tools": "llama.cpp GGUF, GPTQ, AWQ, bitsandbytes",
        "vram_effect": "Reduces proportional to model size",
    },
    "KV cache quantization": {
        "target": "Attention intermediate states generated dynamically during inference",
        "timing": "Real-time quantization on every token generation",
        "quality_impact": "Active research area. Highly task-dependent",
        "tools": "llama.cpp (--cache-type-k, --cache-type-v), vLLM (FP8)",
        "vram_effect": "Reduces proportional to context length",
    },
}

The critical distinction: model weight quantization compresses static data, done once before inference. KV cache quantization compresses data generated in real time during inference, happening on every forward pass. That means overhead.

Using It in llama.cpp

llama.cpp supports KV cache quantization through --cache-type-k and --cache-type-v options.

# FP16 KV cache (default)
llama-cli -m model.gguf -c 32768

# Q8_0 KV cache (halves memory, minimal quality impact)
llama-cli -m model.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0

# Q4_0 KV cache (quarters memory, noticeable quality impact)
llama-cli -m model.gguf -c 32768 --cache-type-k q4_0 --cache-type-v q4_0

Memory Math on an RTX 4060 8GB

Llama-3-8B Q4_K_M + KV Cache Quantization

# VRAM usage estimates on RTX 4060 8GB
configs = {
    "FP16 KV (default)": {
        "model": 4.9,        # Q4_K_M
        "kv_32k": 4.0,       # FP16
        "overhead": 0.5,     # CUDA context, etc.
        "total": 9.4,
        "fits_8gb": False,
    },
    "Q8_0 KV": {
        "model": 4.9,
        "kv_32k": 2.0,       # Half of FP16
        "overhead": 0.5,
        "total": 7.4,
        "fits_8gb": True,    # Barely
    },
    "Q4_0 KV": {
        "model": 4.9,
        "kv_32k": 1.0,       # Quarter of FP16
        "overhead": 0.5,
        "total": 6.4,
        "fits_8gb": True,    # Comfortable
    },
}

# FP16 can't do 32K context at all
# Q8_0 barely fits
# Q4_0 leaves 1.6GB headroom -> room to co-locate an embedding model

Practical Configurations Compared

Config 1: Llama-3-8B Q4_K_M + FP16 KV
  Context: ~16K max (VRAM 7.4GB)
  Use case: Short conversations, code generation

Config 2: Llama-3-8B Q4_K_M + Q8_0 KV
  Context: Up to 32K (VRAM 7.4GB)
  Use case: Medium-length document processing, longer conversations
  Quality: Nearly identical to FP16 (details below)

Config 3: Llama-3-8B Q4_K_M + Q4_0 KV
  Context: 32K (VRAM 6.4GB) -> can co-locate BGE-M3
  Use case: RAG + long context
  Quality: Degradation visible on some tasks

Config 4: Qwen2.5-32B Q4_K_M + Q4_0 KV (partial offload)
  Model: 18GB -> GPU 7.5GB + CPU 10.5GB
  KV (8K): Q4_0 at 0.5GB -> GPU
  Use case: Short context but need high-quality answers

The essence of KV cache quantization: it reshapes the tradeoff between model size and context length. With FP16 KV, you're stuck with "small model x short context." With Q4_0 KV, you unlock "small model x long context" and "large model x short context + RAG."

Where Quality Breaks Down

Findings from the KIVI Paper

KIVI (Liu et al., 2024, arXiv:2402.02750) provides the most systematic study of KV cache quantization.

# KIVI: Key-Value Cache Quantization — key findings
kivi_findings = {
    "method": "Key uses per-channel quantization, Value uses per-token quantization",
    "rationale": {
        "Key": "Value distribution varies widely across channels -> per-channel is appropriate",
        "Value": "Value distribution varies widely across tokens -> per-token is appropriate",
    },
    "results": {
        "2bit_KV": "Downstream task accuracy drops by at most ~2%",
        "2bit_KV_longbench": "LongBench: 44.27 vs FP16's 44.52 (0.56% gap)",
        "VRAM_savings": "2-bit vs FP16: 87.5% KV cache reduction (1/8). Paper reports 2.6x total peak memory reduction",
    },
    "critical_note": "Key and Value need different quantization axes. Quantizing both the same way causes quality to collapse",
}

KIVI's core finding: Key and Value quantization must be designed separately. Each Key channel's value range is stable across tokens but varies wildly between channels. Value is the opposite. Apply the same quantization scheme to both, and one of them falls apart.

Which Tasks Break First

# KV cache quantization tolerance by task type
task_sensitivity = {
    "High tolerance (Q4 still practical)": [
        "Simple Q&A (fact retrieval)",
        "Summarization (short to short)",
        "Classification tasks",
        "Code completion (short context)",
    ],
    "Medium tolerance (Q8 recommended)": [
        "Long document summarization (16K+ tokens)",
        "Multi-turn conversation (10+ turns)",
        "Document reference in RAG",
        "Translation (especially technical docs)",
    ],
    "Low tolerance (FP16 recommended)": [
        "Mathematical reasoning (CoT)",
        "Exact numerical citation",
        "Long-range information retrieval (needle-in-haystack)",
        "Code logical consistency (long functions)",
    ],
}

A pattern emerges. Tasks that require precise information retention are the most sensitive to quantization. Summarization and classification survive on the "gist" of the input, but math and needle-in-haystack depend on exact bit-level values of specific tokens.

This mirrors model weight quantization behavior. Q4_K_M handles casual conversation fine but shows degradation on math benchmarks. The same law applies to KV cache quantization.

Implementation Pattern: Dynamic Switching Based on Context Length

The ideal setup switches KV cache quantization level based on context length. Short context gets FP16 for maximum quality. As it grows, drop to Q8_0 or Q4_0 to prevent overflow.

# Auto-select KV cache config based on context length
def select_kv_config(
    model_vram_gb: float,
    gpu_vram_gb: float = 8.0,
    target_context: int = 32768,
    n_layers: int = 32,
    n_heads_kv: int = 8,
    head_dim: int = 128,
) -> dict:
    """Select KV cache config based on available VRAM"""
    overhead = 0.5  # CUDA context, etc.
    available = gpu_vram_gb - model_vram_gb - overhead

    kv_sizes = {}
    for dtype, factor in [("f16", 2), ("q8_0", 1), ("q4_0", 0.5)]:
        bytes_per_token = 2 * n_layers * n_heads_kv * head_dim * factor
        max_ctx = int(available * (1024**3) / bytes_per_token)
        kv_sizes[dtype] = {
            "max_context": max_ctx,
            "vram_at_target": (bytes_per_token * target_context) / (1024**3),
        }

    # Pick the highest quality that fits
    for dtype in ["f16", "q8_0", "q4_0"]:
        if kv_sizes[dtype]["max_context"] >= target_context:
            return {
                "cache_type": dtype,
                "max_context": kv_sizes[dtype]["max_context"],
                "kv_vram": kv_sizes[dtype]["vram_at_target"],
                "total_vram": model_vram_gb + kv_sizes[dtype]["vram_at_target"] + overhead,
            }

    return {"cache_type": "q4_0", "max_context": kv_sizes["q4_0"]["max_context"],
            "note": "target_context exceeds available VRAM even with Q4_0"}

# Llama-3-8B Q4_K_M on RTX 4060 8GB
config = select_kv_config(model_vram_gb=4.9, target_context=32768)
# -> {"cache_type": "q8_0", "max_context": ~33000, "kv_vram": 2.0, "total_vram": 7.4}
# Q8_0 barely reaches 32K

# Qwen3.5-4B Q4_K_M on RTX 4060 8GB
# Note: Qwen3.5-4B uses a hybrid architecture (32 layers, only 8 full-attention layers, rest are Gated DeltaNet)
# KV cache only applies to the 8 attention layers
config_small = select_kv_config(model_vram_gb=2.7, n_layers=8, n_heads_kv=4, target_context=32768)
# -> FP16 fits 32K easily (2.7 + 0.5 + 0.5 = 3.7GB, tons of room)

With a small enough model, you might not need KV cache quantization at all. Qwen3.5-4B (~2.7GB) has a hybrid architecture where only 8 of 32 layers use traditional attention with KV cache. FP16 KV at 32K context uses barely 0.5GB. Sometimes "use a smaller high-accuracy model at full precision" beats "cram a bigger model in with quantization."

llama.cpp KV Cache Quantization Internals

Q8_0 vs Q4_0 Under the Hood

# llama.cpp KV cache quantization schemes
kv_quant_details = {
    "q8_0": {
        "bit_width": 8,
        "block_size": 32,
        "method": "absmax symmetric quantization",
        "computation": "For each 32-element block, find max(abs(x)) and store one scale factor",
        "quality": "Near-identical to FP16 (KIVI paper shows negligible degradation at Q8)",
        "speed": "Decode speed equal to or slightly faster than FP16 (reduced memory transfer)",
        "recommendation": "Safe drop-in replacement for default",
    },
    "q4_0": {
        "bit_width": 4,
        "block_size": 32,
        "method": "absmax symmetric quantization (4-bit)",
        "computation": "Compress each 32 elements to 4-bit. One scale factor",
        "quality": "Task-dependent. Fine for simple tasks, degrades on math and long-range retrieval",
        "speed": "Decode can be faster when bandwidth-bound (reduced transfer volume)",
        "recommendation": "Only when VRAM constraints are severe",
    },
}

Impact on Decode Speed

KV cache quantization has a surprising side effect. When memory bandwidth is the bottleneck during decode, smaller KV cache means less data to transfer, which can make decoding faster.

# Estimated decode speed impact on RTX 4060 (272 GB/s)
decode_impact = {
    "FP16 KV, 32K context": {
        "kv_read_per_token": "4.0 GB (KV across all layers)",
        "theoretical_speed": "272 / (4.9 + 4.0) = 30 t/s",
        "note": "Reads model weights + entire KV for attention computation",
    },
    "Q8_0 KV, 32K context": {
        "kv_read_per_token": "2.0 GB",
        "theoretical_speed": "272 / (4.9 + 2.0) = 39 t/s",
        "improvement": "+30%",
    },
    "Q4_0 KV, 32K context": {
        "kv_read_per_token": "1.0 GB",
        "theoretical_speed": "272 / (4.9 + 1.0) = 46 t/s",
        "improvement": "+53%",
    },
}
# KV cache quantization doesn't just "make it fit" — it makes it faster
# Though dequantization CPU/GPU overhead means real numbers will be lower

Not just VRAM savings — bandwidth savings too. This is effectively a Layer 2 optimization (reducing read volume) against the bandwidth wall (separate article). A software-side weapon that increases effective bandwidth without changing hardware.

Combining with Other KV Cache Optimizations

KV cache quantization isn't meant to be used alone. It hits hardest when combined with other optimizations.

Method                  VRAM Reduction  Quality Impact  Implementation
────────────────────────────────────────────────────────────────────────
GQA (Grouped Query)     50-75%          None            Decided at model design time
KV Quant (Q8_0)         50%             Negligible      One llama.cpp flag
KV Quant (Q4_0)         75%             Task-dependent  One llama.cpp flag
Sliding Window          Fixed cap       Long-range loss  Decided at model design time
Sparse Attention        Significant     Task-dependent  Requires custom implementation
Paged Attention         Defrag only     None            Automatic in vLLM, etc.

Combined example: Llama-3-8B (GQA 4x) + Q8_0 KV
  GQA 75% reduction x Q8_0 50% reduction = 12.5% of baseline
  FP16 full cache: 16GB -> 2.0GB
  32K context fits comfortably on an 8GB GPU

GQA (Grouped Query Attention) is already baked into most recent models, so users don't need to think about it. Llama-3, Qwen2.5, Mistral all benefit from it. Stacking KV quantization on top cuts it by another half to quarter.

When You Shouldn't Quantize on 8GB

I've been making the case for KV cache quantization, but the opposite perspective matters too.

# Cases where KV cache quantization is unnecessary
unnecessary_cases = {
    "Small model + short context": {
        "example": "Qwen3.5-4B (~2.7GB, only 8 attention layers) + 8K context",
        "KV_FP16": "~0.13 GB (8 attention layers × 4 KV heads)",
        "total": "~3.3 GB",
        "verdict": "Under half of 8GB. Quantization is pointless",
    },
    "Task-specialized models": {
        "example": "Function calling only (short I/O)",
        "KV_FP16": "< 0.1 GB",
        "verdict": "Short context means KV cache is never the problem",
    },
    "Accuracy-critical tasks": {
        "example": "Math reasoning, code review (correctness is paramount)",
        "verdict": "Better to use a smaller model and keep KV at FP16",
    },
}

The realistic approach is to combine this with a multi-model routing strategy (separate article) — switching models based on the task:

Function calling -- Qwen3.5-4B (~2.7GB) + FP16 KV. Short context + few attention layers, no quantization needed
Long-context RAG -- Llama-3-8B (4.9GB) + Q8_0 KV. 32K context with quality preserved
Knowledge Q&A -- Qwen2.5-32B (18GB, CPU/GPU offload) + Q4_0 KV. Short context only

Don't apply the same KV config to every model. Pick based on task and context length.

References

"KIVI: A Tuning-Free KV Cache Quantization Plugin for Large Language Models" (2024) arXiv:2402.02750
llama.cpp KV cache quantization: --cache-type-k, --cache-type-v options
"PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection" (2026) arXiv:2603.21576
"Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023) arXiv:2309.06180

HBM4 Didn't Break the Memory Wall — It Just Moved It

plasmon — Wed, 08 Apr 2026 02:06:11 +0000

HBM4 Didn't Break the Memory Wall — It Just Moved It

HBM bandwidth has doubled every generation.

HBM2E (2020): 410 GB/s per stack — 1024-bit, 3.2 Gb/s/pin
HBM3  (2022): 819 GB/s per stack — 1024-bit, 6.4 Gb/s/pin
HBM3E (2024): 1.2 TB/s per stack — 1024-bit, up to 9.8 Gb/s/pin (JEDEC max, varies by vendor)
HBM4  (2026): 2.0 TB/s per stack — 2048-bit, 8.0 Gb/s/pin

Notice anything off?

HBM4's JEDEC base pin speed is 8.0 Gb/s. That's lower than HBM3E's max JEDEC spec of 9.8 Gb/s (Samsung's implementation; SK Hynix HBM3E runs at 8 Gb/s). Bandwidth doubled, but the base spec pin speed didn't go up. The majority of the bandwidth gain comes from doubling the interface width (1024 to 2048 bits).

They didn't make the pipe faster. They made it wider. That was HBM4's design decision, and it was driven by physics hitting back.

Why Pin Speed Hit a Ceiling

The Signal Integrity Wall

# HBM per-pin speed progression
pin_speed_history = {
    "HBM2E": {"speed": 3.2, "unit": "Gb/s", "year": 2020},
    "HBM3":  {"speed": 6.4, "unit": "Gb/s", "year": 2022},
    "HBM3E": {"speed": 9.8, "unit": "Gb/s", "year": 2024},
    "HBM4":  {"speed": 8.0, "unit": "Gb/s", "year": 2026},
}

# Pin speed growth rate
# HBM2E→HBM3: 2.0x (doubled in 2 years)
# HBM3→HBM3E: 1.53x (1.5x in 2 years)
# HBM3E→HBM4: 0.82x (decline)

Signaling through microbumps above 10 Gb/s gets ugly. TSV (Through-Silicon Via) parasitic capacitance and impedance mismatch cause jitter to blow up. HBM3E's 9.8 Gb/s was already pushing against that physical limit.

SK Hynix's 12-layer HBM4 sample shipped in March 2025 hit 11.7 Gb/s — meeting NVIDIA's Rubin requirements. The JEDEC base spec landed at 8 Gb/s, but vendor implementations routinely exceed base specs (same pattern as HBM3E). Mass production speeds are expected to land between 8-12 Gb/s.

# Why going wider is the safer bet
design_tradeoff = {
    "Increase pin speed": {
        "Upside": "No area increase",
        "Risk": "Jitter blowup, yield loss, power increase",
        "Limit": "~10 Gb/s (TSV parasitic capacitance)",
        "Cost": "Signal integrity circuits get complex fast",
    },
    "Widen the interface": {
        "Upside": "Double bandwidth while keeping signal quality intact",
        "Risk": "Die area increase, packaging cost increase",
        "Limit": "Physical bump pitch (40μm → 36μm → ?)",
        "Cost": "Die area = cost",
    },
}
# HBM4 chose the latter. Safe, but it gets the bandwidth numbers

What 2 TB/s Means for LLM Inference

A100 → H100 → B200 → Next-Gen Bandwidth Progression

# GPU accelerator HBM bandwidth over time
gpu_bandwidth = {
    "A100 80GB":  {"hbm": "HBM2E", "stacks": 5, "bw": "2.0 TB/s", "year": 2020},
    "H100 80GB":  {"hbm": "HBM3",  "stacks": 5, "bw": "3.35 TB/s", "year": 2022},
    "H200":       {"hbm": "HBM3E", "stacks": 6, "bw": "4.8 TB/s", "year": 2024},
    "B200":       {"hbm": "HBM3E", "stacks": 8, "bw": "8.0 TB/s", "year": 2025},
    "Next-gen (est.)": {"hbm": "HBM4", "stacks": 8, "bw": "16 TB/s", "year": "2026-27"},
}

# B200→next-gen: 2x bandwidth
# How much does this actually speed up LLM inference?

Recalculating the Decode Bandwidth Bottleneck

Taking the bandwidth-bound decode calculation from the KV cache article (separate article) and re-running it for the HBM4 generation.

# Llama-3-70B decode bandwidth requirements
model_params = {
    "parameters": 70e9,
    "bytes_per_param": 2,  # FP16
    "model_size": 140e9,   # 140 GB
}

# Generating 1 token requires reading the entire model (weight-bound decode)
# Theoretical max decode speed = bandwidth / model size

decode_speed = {
    "A100 (2 TB/s)":    f"{2000 / 140:.0f} t/s = 14 t/s",
    "H100 (3.35 TB/s)": f"{3350 / 140:.0f} t/s = 24 t/s",
    "H200 (4.8 TB/s)":  f"{4800 / 140:.0f} t/s = 34 t/s",
    "B200 (8 TB/s)":    f"{8000 / 140:.0f} t/s = 57 t/s",
    "HBM4 gen (16 TB/s)": f"{16000 / 140:.0f} t/s = 114 t/s",
}

# HBM4 generation: 70B model at 114 t/s
# 8x faster than A100's 14 t/s
# Still nowhere near 10,000 t/s

What LIMINAL Found

A team led by NVIDIA Research (Davies et al., arXiv:2507.14397) built an analytical model called LIMINAL and came to a sobering conclusion:

liminal_findings = {
    "model_accuracy": "7.6% mean absolute error vs. measured",
    "key_conclusion": "Reaching 10,000+ t/s requires fundamental algorithmic breakthroughs, not just hardware scaling",
    "four_barriers": [
        "Compute",
        "Memory capacity",
        "Memory bandwidth",
        "Collective communication",
    ],
    "implication": "HBM4 won't break through the bandwidth wall. It just pushes it back a bit",
}

10,000 t/s means generating a token every 0.1ms. For a 70B model (140GB at FP16), that requires 10,000 × 140 = 1,400 TB/s of bandwidth. Eight HBM4 stacks deliver 16 TB/s — roughly 1/88th of what's needed. Two orders of magnitude short. LIMINAL makes it clear: no amount of HBM generational scaling alone will bridge this gap.

What This Means for Consumer GPUs

RTX 4060 → Next-Gen Bandwidth Outlook

# Consumer GPU (GDDR) bandwidth over time
consumer_bandwidth = {
    "RTX 3060":  {"memory": "GDDR6",  "bw": "360 GB/s", "year": 2021},
    "RTX 4060":  {"memory": "GDDR6",  "bw": "272 GB/s", "year": 2023},
    "RTX 5060": {"memory": "GDDR7", "bw": "448 GB/s", "year": 2025},
    "RTX 6060 (est.)": {"memory": "GDDR7X?", "bw": "~550-600 GB/s", "year": "2027-28"},
}

# Note: RTX 4060 actually regressed in bandwidth (360→272)
# NVIDIA prioritizes cost over bandwidth on consumer cards
# Even with GDDR7, RTX 5060's 448 GB/s is roughly one HBM2E stack

# The datacenter-consumer bandwidth gap
gap = {
    "2024": f"B200 8 TB/s / RTX 4060 272 GB/s = {8000/272:.0f}x gap",
    "2027": f"HBM4 gen 16 TB/s / RTX 6060 est. 550 GB/s = {16000/550:.0f}x gap",
}
# The gap stays around 30x. Consumer will never catch up to datacenter

What Local LLM Users Should Take Away

# If bandwidth determines decode speed, is local LLM's future bleak?

local_llm_perspective = {
    "The pessimistic view": {
        "fact": "272 GB/s caps Qwen2.5-32B Q4_K_M (~20GB) at ~14 t/s",
        "fact2": "GDDR7 at 448 GB/s only gets you to ~22 t/s",
        "conclusion": "You can't win on bandwidth",
    },
    "The realistic view": {
        "counter1": "Quantization keeps advancing (Q2_K, 1.5-bit) — smaller models need less bandwidth",
        "counter2": "Small models are getting scary good (Qwen3.5 4B-class accuracy gains) — you may not need the big model",
        "counter3": "Multi-model orchestration — use bandwidth efficiently",
        "counter4": "Block selection optimization (software PRISM-like approaches) — read less data per token",
    },
    "core_truth": "You'll lose the bandwidth arms race. Win by not needing the bandwidth in the first place",
}

Datacenter GPU bandwidth grows at roughly 2x every two years. Consumer GPU bandwidth can actually go backwards when cost is prioritized (RTX 3060 to 4060: 360 to 272 GB/s). This gap is structural — it won't close. HBM4's 2 TB/s per stack is for datacenter in 2026, and it will never come to consumer hardware. HBM requires an interposer and packaging that simply doesn't fit a consumer GPU form factor.

The battlefield for local LLM isn't raw bandwidth. It's maximizing useful work per byte transferred — and that's a software and model design problem.

The Memory Wall Is Being Attacked from Three Directions

Three layers of approach exist for tackling the memory bandwidth bottleneck right now.

Layer 1: Raw hardware bandwidth increase
  HBM3→HBM3E→HBM4 (2x every 2 years)
  GDDR6→GDDR7 (up to ~1.6x on spec, actual products vary)
  → Reliable, but decelerating at the 10 Gb/s/pin physics wall

Layer 2: Reduce how much you read
  Quantization (FP16→Q4_K_M→Q2_K: 4-8x reduction)
  Sparse Attention (skip most of KV cache — reduction varies by method)
  Block selection (PRISM: 16x reduction)
  → Software-driven, immediate impact

Layer 3: Eliminate reads entirely
  PIM (Processing-In-Memory: compute where the data lives)
  Cache optimization (keep hot patterns in SRAM)
  → Requires hardware changes, but attacks the root cause

Effective bandwidth = L1 × L2 × L3
If HBM4 + Sparse Attention + PIM all come together:
  16 TB/s × 10x reduction (est.) × 2x PIM efficiency (est.) = 320 TB/s effective
  That gives a 70B model 2,285 t/s — 163x over today's A100
  Note: 10x and 2x are rough estimates from combining techniques, not measured values

Still short of 10,000 t/s. But the leap from 14 t/s to 2,000+ t/s is impossible with any single technology — it only becomes visible when you multiply all three layers together.

HBM4 didn't "double bandwidth." It updated Layer 1 of a three-layer stack. The other two layers are software and architecture problems — and that's where local LLM still has room to fight. Even an RTX 4060 at 272 GB/s can multiply its effective bandwidth several times over by maxing out Layer 2. You can lose on raw numbers and still win on how you use what you've got.

References

"LIMINAL: Exploring The Frontiers of LLM Decode Performance" (2025) arXiv:2507.14397
JEDEC JESD270-4 HBM4 Standard (2025)
SK Hynix HBM4 12-layer sample (March 2025) — 11.7 Gb/s, for NVIDIA Rubin
"PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection" (2026) arXiv:2603.21576

Running Just One LLM on 8GB VRAM Is a Waste

plasmon — Tue, 07 Apr 2026 22:56:53 +0000

Liquid syntax error: Unknown tag 'endraw'

Light Just Cut KV Cache Memory Traffic to 1/16th

plasmon — Tue, 07 Apr 2026 22:53:20 +0000

Light Just Cut KV Cache Memory Traffic to 1/16th

The bottleneck in long-context LLM inference isn't compute. It's memory bandwidth.

Every decode step in a Transformer scans the entire KV cache to generate a single token. That's O(n) memory reads for context length n, every single step. No matter how fast your GPU's ALUs get, this O(n) memory wall doesn't budge.

A March 2026 ArXiv paper (arXiv:2603.21576, Park & Park) proposes PRISM, which offloads KV cache block selection to photonic circuits, making memory access O(1).

The result: 16x memory traffic reduction at 64K tokens. Block selection energy efficiency: 10,000x. Accuracy: 100%.

Why Memory Bandwidth Is the LLM Inference Bottleneck

The Structural Problem with Decoding

# What happens in one Transformer decode step
def decode_one_token(query, kv_cache):
    # query: current token (1)
    # kv_cache: all past tokens (n)

    # Step 1: compute similarity between query and entire KV cache
    scores = query @ kv_cache.keys.T  # O(n) memory reads

    # Step 2: Softmax
    weights = softmax(scores)

    # Step 3: weighted sum
    output = weights @ kv_cache.values  # O(n) memory reads

    return output

# As context length grows, Steps 1 and 3 scale linearly
# Compute is also O(n), but the real bottleneck is memory read speed

The Bandwidth Hell in Numbers

# Bandwidth consumption at 64K context
# Qwen2.5-7B: hidden=3584, layers=28, GQA (4 KV heads, head_dim=128)
context_length = 64_000  # 64K tokens
kv_dim = 4 * 128         # GQA: 4 KV heads × 128 dim = 512
num_layers = 28
bytes_per_element = 2    # FP16

# KV cache reads per decode step
kv_read_per_step = (
    context_length * kv_dim * 2  # K + V
    * num_layers * bytes_per_element
)
# = 64000 * 512 * 2 * 28 * 2 = 3.67 GB

# RTX 4060 bandwidth: 272 GB/s
# At 64K, bandwidth alone allows ~75 t/s — GQA helps enormously

# Without GQA (MHA model like LLaMA-1 7B: hidden=4096, layers=32, all heads KV):
# 64000 * 4096 * 2 * 32 * 2 = 33.6 GB → bandwidth ceiling = 8 t/s
# GQA reduced KV traffic by ~9x

# But larger models still hit the wall even with GQA:
# 70B model (GQA, KV dim=1024, 80 layers): 64000*1024*2*80*2 = 16.8 GB
# RTX 4090: 16.8 / 1008 = 0.017s = ~60 t/s ceiling

GPU compute doubles every generation, but memory bandwidth grows far more slowly. This divergence is the fundamental wall for long-context LLM inference.

This is the same problem that PIM (Processing-in-Memory) attacks — move compute to where the data lives. PRISM takes a different angle: reduce the amount of data you need to read in the first place.

Existing Solutions and Their Limits

Top-K / Sparse Attention

# Existing KV cache reduction methods
existing_approaches = {
    "Top-K Attention": {
        "method": "Read only the top-K most similar KV blocks",
        "problem": "Finding which blocks are top-K requires scanning all of them",
        "complexity": "O(K) after selection, but selection itself is O(n)",
    },
    "Sliding Window": {
        "method": "Only reference the most recent W tokens",
        "problem": "Completely discards long-range information",
        "complexity": "O(W) but sacrifices accuracy",
    },
    "H2O (Heavy-Hitter Oracle)": {
        "method": "Keep tokens with highest cumulative attention scores",
        "problem": "Score tracking itself requires O(n) memory management",
        "complexity": "Reduces work but the O(n) wall remains",
    },
}

The common problem: determining which KV blocks matter still requires an O(n) memory scan. No matter how efficiently you process the selected blocks, the selection step's O(n) doesn't go away.

Existing Photonic Accelerators

Research exists on accelerating attention computation with optical circuits. But these approaches accelerate dense matrix operations (dense attention) with light — the O(n) memory scaling stays the same.

PRISM's insight is different. Instead of dense matrix multiplication, it uses light for block selection — a coarse similarity search.

PRISM Architecture

The Core Idea: O(1) Block Selection with Light

Conventional Block Sparse Attention:
  Query → [Electronic: similarity with all blocks O(n)] → Top-K selection → precise compute

PRISM:
  Query → [Photonic: similarity with all blocks simultaneously O(1)] → Top-K selection → precise compute
           ↑
           This is what changes

Why does light make this O(1)? It exploits the physical properties of light itself.

Broadcast-and-Weight

# Electronic block selection
def electronic_block_select(query, block_keys, k):
    """Compute similarity with n blocks sequentially → O(n)"""
    scores = []
    for block_key in block_keys:  # n memory reads
        score = dot_product(query, block_key)
        scores.append(score)
    return top_k(scores, k)

# Photonic block selection (PRISM's principle)
def photonic_block_select(query_light, block_modulators, k):
    """Broadcast light to all blocks simultaneously → O(1)"""
    # Step 1: Convert query to optical signal
    # Step 2: Broadcast light to all microring resonators simultaneously
    # Step 3: Each resonator modulates light with its block key (weighting)
    # Step 4: Read all results simultaneously with photodetectors
    # → All block similarities resolved in one clock cycle
    return top_k(all_scores, k)  # O(1)

Unlike electrons, light can carry multiple wavelengths simultaneously through a single waveguide (Wavelength Division Multiplexing — WDM). Each wavelength handles a different block's similarity computation, enabling physically parallel processing of all blocks.

TFLN Microring Resonators

PRISM uses Thin-Film Lithium Niobate (TFLN) microring resonators.

# TFLN microring specs (from paper design + TFLN literature)
tfln_specs = {
    "material": "LiNbO3 thin film",
    "modulation_speed": "100+ GHz class (electro-optic effect)",
    "insertion_loss": "Low loss (vs silicon)",
    "precision": "4-6 bit (sufficient for block selection)",
    "size": "Tens of μm diameter (thousands fit on one chip)",
}

# Why TFLN (paper's design rationale)
advantages = {
    "vs_silicon": "Orders-of-magnitude higher electro-optic coefficient → far better modulation efficiency",
    "vs_InP": "Larger wafer sizes, more suitable for mass production",
    "vs_thermal_tuning": "ns response (thermal: μs response)",
}

The key insight: block selection is a coarse similarity search — 4-6 bit precision is plenty. Full-precision attention computation runs on electronic circuits (GPU) for only the selected blocks. Light handles the low-precision work at extreme speed; electronics handle the high-precision work on a small subset.

Benchmarks: 16x Reduction at 100% Accuracy

Needle-in-a-Haystack Test

# PRISM accuracy evaluation (from paper)
prism_accuracy = {
    "model": "Qwen2.5-7B",
    "method": "PRISM (k=32 blocks)",
    "context_lengths": [4096, 8192, 16384, 32768, 65536],
    "accuracy": [100, 100, 100, 100, 100],  # 100% across all lengths
    "test": "Needle-in-a-Haystack (NIAH)",
}

# k=32 selects only a fraction of all blocks
# Yet 100% accuracy → block selection judgment is precise

Memory Traffic Reduction

# Traffic reduction by context length
traffic_reduction = {
    "4K":  {"full_attention": "1x", "prism": "~2x reduction"},
    "16K": {"full_attention": "1x", "prism": "~6x reduction"},
    "64K": {"full_attention": "1x", "prism": "16x reduction"},
}
# Longer context = bigger reduction
# PRISM's photonic circuit cost is fixed (O(1)), so gains grow with n

Energy Efficiency

# GPU vs PRISM energy comparison (block selection only, per paper estimates)
# Note: NOT total LLM inference cost — only the KV cache block selection operation
energy_comparison = {
    "metric": "Energy per block selection (paper Table, approximate)",
    "gpu_baseline": {
        "4K": "~1 mJ",
        "64K": "~16 mJ",
        "scaling": "O(n) — linear with context length",
    },
    "prism": {
        "4K": "~0.1 μJ",
        "64K": "~0.1 μJ",
        "scaling": "O(1) — independent of context length",
    },
    "ratio_at_64K": "~10,000x (4 orders of magnitude) — block selection only",
}

A 4-order-of-magnitude efficiency gap in block selection sounds wild, but the math is straightforward: GPU energy scales with n, photonic circuit energy doesn't. As context grows, this gap widens further. However, total LLM inference energy is dominated by FFN and attention compute on electronics, so the system-level impact is more modest. The 16x memory traffic reduction is the more practically significant number.

What This Means for an RTX 4060 User

PRISM is a research-stage photonic chip. You can't buy one today. But the direction this research points matters directly to local LLM users.

The Current Long-Context Wall

# Long-context inference reality on RTX 4060 8GB
rtx4060_long_context = {
    "vram": "8GB",
    "bandwidth": "272 GB/s",
    "practical_limits": {
        "Qwen2.5-7B Q4_K_M": {
            "4K": "Runs fine",
            "8K": "Works but noticeably slower",
            "16K": "KV cache eats VRAM, severely degraded speed",
            "32K": "OOM or swap thrashing",
        },
    },
    "bottleneck": "272 GB/s bandwidth saturates at long context",
}

What PRISM Suggests You Can Do Today

You can't use PRISM itself, but the same principle — efficient block selection — has software approximations available right now.

# Software-based block selection approaches
software_block_selection = {
    "Quest (2024)": {
        "method": "Store per-block statistics (min/max), compare with query to skip irrelevant blocks",
        "effect": "Major KV cache access reduction (per paper)",
        "accuracy": "Near full-attention accuracy (per paper)",
        "available": True,
    },
    "RetrievalAttention (2024)": {
        "method": "Store KV cache in a vector DB, use approximate nearest-neighbor search for selection",
        "effect": "Major speedup at long context",
        "accuracy": "Task-dependent",
        "available": True,
    },
    "vLLM PagedAttention": {
        "method": "Manage KV cache in pages, prevent memory fragmentation",
        "effect": "Better effective VRAM utilization",
        "accuracy": "Lossless",
        "available": True,
    },
}

# Relationship to PRISM:
# Software methods = still O(n) but with smaller constants
# PRISM = fundamentally O(1)
# Same direction: don't read everything, read only the blocks that matter

The Future of Photonic Computing and LLM Inference

How CPO and PRISM Relate

CPO (Co-Packaged Optics) accelerates chip-to-chip data transfer using light. PRISM performs computation itself using light. Different layers of the same stack.

CPO:   Chip ←light→ Chip  (accelerate transfer with light)
PIM:   Compute inside memory  (eliminate transfer)
PRISM: Compute with light  (make selection O(1), reduce transfer)

Different approaches, but all fighting the same enemy — the memory bandwidth wall

Distance to Production

prism_roadmap_estimate = {
    "current_stage": "Paper published (simulation + individual component demos)",
    "missing": [
        "Full system integration test",
        "Physical interface to GPU",
        "Manufacturing process establishment",
        "Software stack (drivers, compilers)",
    ],
    "optimistic": "3-5 years for research prototype",
    "realistic": "5-10 years for limited commercial deployment",
    "comparison": {
        "CPO": "Intel/TSMC targeting limited production 2026-2027",
        "PIM": "Samsung HBM-PIM announced 2021, joint testing with AMD",
        "PRISM-type": "Academic stage",
    },
}

Production is far off. But LLM context lengths keep growing rapidly (GPT-4: 8K → GPT-4 Turbo: 128K → Gemini 1.5: 1M → Claude 4.x: 1M), and the memory bandwidth wall gets worse at the same rate. Electronic bandwidth improvements can't keep up with that growth curve. Light's O(1) scaling holds the principled answer.

Three Weapons Against the Memory Bandwidth Wall

What this article covers:

The bottleneck in long-context LLM inference is memory bandwidth: Every decode step scans the full KV cache — O(n) cost
PRISM makes block selection O(1) with light: WDM and microring resonators enable physically parallel processing of all blocks
16x traffic reduction at 64K, 10,000x block selection energy efficiency: Light's advantage grows with context length
100% accuracy preserved: Coarse selection (4-6 bit) runs on light, precise computation runs on electronics — division of labor
Software versions of the same principle work today: Quest, RetrievalAttention, and other block selection optimizations

Against the memory wall, three distinct approaches are advancing simultaneously: CPO (optical transfer), PIM (in-memory compute), PRISM (optical selection). It's not about which one wins — they'll likely combine at different layers. Light works for transfer and computation alike. That flexibility will shape the next decade of semiconductors.

References

"PRISM: Breaking the O(n) Memory Wall in Long-Context LLM Inference via O(1) Photonic Block Selection" (2026) arXiv:2603.21576
"Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference" (2024) arXiv:2406.10774
"RetrievalAttention: Accelerating Long-Context LLM Inference via Vector Retrieval" (2024) arXiv:2409.10516

ツール呼び出しでも大きいモデルは勝てなかった

plasmon — Tue, 07 Apr 2026 21:48:06 +0000

ツール呼び出しでも大きいモデルは勝てなかった

LLMにツールを持たせる（function calling / tool use）。これはエージェントの基盤技術であり、RAGの次の進化であり、ローカルLLMの実用性を左右する機能だ。

では、どのモデルがfunction callingで最も正確か。13モデルをQ4_K_M量子化でテストした2026年のベンチマーク（JD Hodges, 2026）の結果は、予想を裏切るものだった。

97.5%の精度を出したのは3.4GBのモデルだった。25GBのモデルは85%で負けた。

少なくともこのテスト環境では、大きいモデルが強いという前提は成り立たなかった。

ベンチマーク結果: 13モデル全比較

全モデルQ4_K_M量子化（モデルの精度を維持しつつVRAM使用量を約75%削減する4bit圧縮）、LM Studio経由で統一環境テスト。40ケースのfunction calling精度。サイズはLM Studio上でのロードVRAM（ファイルサイズとは異なる）。

順位	モデル	サイズ	精度
1	Qwen3.5 4B	3.4GB	97.5%
2	GLM-4.7-Flash	18GB	95.0%
2	Nemotron 3 Nano 4B	4.2GB	95.0%
4	Mistral Nemo 12B	7.5GB	92.5%
5	Qwen3 8B	5GB	85.0%
5	GPT-OSS 20B	12GB	85.0%
5	Nemotron 3 Nano 30B-A3B	25GB	85.0%
8	DeepSeek-R1-Distill 14B	9GB	57.5%
8	Phi-4 Mini	2.5GB	57.5%
10	Gemma 3 4B QAT	3.2GB	55.0%
11	Mistral Small 3.2 24B	15GB	42.5%
12	Hammer 2.1 7B	4.2GB	20.0%
13	xLAM-2 8B FC-R	4.9GB	15.0%

3つの衝撃

衝撃1: 3.4GBが97.5%
Qwen3.5 4Bは40ケース中39ケースで成功。失敗は1ケースのみ。3.4GBはRTX 4060 8GBのVRAMの半分以下。推論モデルとEmbeddingを同時に動かしても余裕がある。

衝撃2: サイズだけでは精度を予測できない

サイズ	モデル	精度
3.4GB	Qwen3.5 4B	97.5% (1位)
4.2GB	Nemotron 3 Nano 4B	95.0% (2位)
7.5GB	Mistral Nemo 12B	92.5% (4位)
18GB	GLM-4.7-Flash	95.0% (2位)
25GB	Nemotron 3 Nano 30B-A3B	85.0% (5位)

このテスト環境（LM Studio + Q4_K_M）では、モデルサイズだけではfunction calling精度を予測できない。15GBのMistral Smallが42.5%で、3.4GBのQwen3.5に55ポイント差で負ける。ただしn=13かつLM Studioの推論パス依存なので、モデル固有の能力についての一般的結論を出すにはデータが不足している。

注意: 下位モデルにはLM Studio互換性問題がある
xLAM-2 8B FC-R（15%）とHammer 2.1（20%）の低スコアは、モデル品質ではなくLM Studioのchat template非互換が原因。xLAM-2は実際にはBerkeley Function Calling Leaderboard（BFCL）で1位のモデルだが、独自のツール呼び出し形式がLM StudioのOpenAI互換APIで正しく変換されなかった。このベンチマークは「LM Studio経由での実用精度」を測っており、モデル固有の能力を測っていない点に注意が必要。

なぜ小さいモデルがfunction callingで勝つのか

構造化出力の特殊性

function callingはLLMのタスクの中でも特殊だ。自由文生成ではなく、厳密な構造（JSON、関数名、引数型）に従った出力が必要。

要件	function calling	自由文生成
出力形式	JSON（厳密なスキーマ準拠）	自然言語（柔軟）
型安全性	引数の型が正確必須	不要
ハルシネーション許容度	ゼロ（存在しない関数を呼べない）	ある程度許容
知識依存度	低い（形式遵守が支配的）	高い（世界知識が品質に直結）

ポイント: function callingは知識量よりフォーマット遵守能力に依存する。大きいモデルの強みは知識量（世界知識、推論能力）にあるが、function callingではそれが活きない。代わりに、学習データの品質とinstruction followingの精度が効く。

Qwen3.5 4Bが強い理由は推測だが:

Qwen3.5世代のinstruction tuningがfunction calling形式に最適化されている
4Bパラメータでも構造化出力には十分な表現力がある
パラメータ数を絞ることで、学習データの信号がノイズに埋もれにくい

パラメータ数神話の崩壊（続）

これは8GB VRAMのモデル選択ルール（別記事）で述べた法則の延長だ。

法則 (再確認):
  パラメータ数 ≠ タスク性能

追加法則 (function calling):
  構造化出力タスクでは、小さいモデルが大きいモデルに勝つことがある
  理由: 知識量ではなくフォーマット遵守能力が支配的

RTX 4060 8GBでのfunction calling実装

ベンチマーク結果を実装に落とし込む。

推奨構成

目的	モデル	VRAM	精度	余剰VRAM
最高精度	Qwen3.5 4B Q4_K_M	~3.4GB	97.5%	~4.6GB（Embedding同時起動可）
バランス	Nemotron 3 Nano 4B Q4_K_M	~4.2GB	95.0%	~3.8GB
マルチターン	Mistral Nemo 12B Q4_K_M	~7.5GB	92.5%	~0.5GB（単独使用推奨）

最小構成コード

# llama.cpp + Qwen3.5-4B で function calling
import subprocess
import json

TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "get_weather",
            "description": "Get current weather for a location",
            "parameters": {
                "type": "object",
                "properties": {
                    "location": {"type": "string"},
                    "unit": {"type": "string", "enum": ["celsius", "fahrenheit"]}
                },
                "required": ["location"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "search_documents",
            "description": "Search internal documents by query",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "max_results": {"type": "integer", "default": 5}
                },
                "required": ["query"]
            }
        }
    }
]

def call_with_tools(user_message: str, tools: list) -> dict:
    """ローカルLLMでfunction callingを実行"""
    # ChatML形式のプロンプト構築
    tools_json = json.dumps(tools, ensure_ascii=False)
    prompt = f"""<|im_start|>system
You are a helpful assistant with access to tools.
Available tools: {tools_json}
When you need to use a tool, respond with JSON:
{{"name": "function_name", "arguments": {{"key": "value"}}}}
Only call a tool if the user's request requires it.
<|im_end|>
<|im_start|>user
{user_message}<|im_end|>
<|im_start|>assistant
"""

    result = subprocess.run(
        ["llama-cli", "-m", "qwen3.5-4b-q4_k_m.gguf",
         "-p", prompt, "-n", "200", "--temp", "0.1",
         "--grammar-file", "json.gbnf"],  # JSON文法制約
        capture_output=True, text=True
    )

    try:
        return json.loads(result.stdout.strip())
    except json.JSONDecodeError:
        return {"error": "Failed to parse", "raw": result.stdout}

# 実行例
result = call_with_tools(
    "東京の天気を教えて",
    TOOLS
)
# 期待出力: {"name": "get_weather", "arguments": {"location": "Tokyo"}}

GBNF文法によるJSON保証

llama.cppのGBNF（Generalized Backus-Naur Form）文法制約を使うと、LLMの出力をJSON形式に強制できる。これにより構造化出力の信頼性がさらに上がる。

# json.gbnf (基本的なJSON文法)
root   ::= object
value  ::= object | array | string | number | "true" | "false" | "null"
object ::= "{" ws (string ":" ws value ("," ws string ":" ws value)*)? ws "}"
array  ::= "[" ws (value ("," ws value)*)? ws "]"
string ::= "\"" [^"\\]* "\""
number ::= "-"? [0-9]+ ("." [0-9]+)?
ws     ::= [ \t\n]*

GBNF制約なしの場合、モデルが自由にテキストを混ぜてパース失敗が頻発する:

# GBNF なし — モデルが余計なテキストを混ぜる
{"name": "get_weather", "arguments": {"location": "Tokyo"}} Let me check the weather for you...

GBNF制約ありの場合、各トークンが文法規則に準拠するよう制約されるため、構文的に不正なJSONは生成されにくい。ただしトークン上限に達した場合は不完全なJSONになる可能性があり、意味的な正確性（正しい関数名や引数値）は保証されない。推論速度に若干のオーバーヘッドが発生する。

function calling × Agentic RAG の統合

function callingが使えると、前回のAgentic RAG記事（別記事）で紹介したエージェント型RAGがさらに安定する。

# function calling + Agentic RAG の統合実装
ARAG_TOOLS = [
    {
        "type": "function",
        "function": {
            "name": "keyword_search",
            "description": "キーワードベースでドキュメント検索",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "k": {"type": "integer", "default": 5}
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "semantic_search",
            "description": "意味ベクトルでドキュメント検索",
            "parameters": {
                "type": "object",
                "properties": {
                    "query": {"type": "string"},
                    "k": {"type": "integer", "default": 5}
                },
                "required": ["query"]
            }
        }
    },
    {
        "type": "function",
        "function": {
            "name": "answer",
            "description": "ユーザーに最終回答を提供",
            "parameters": {
                "type": "object",
                "properties": {
                    "text": {"type": "string"}
                },
                "required": ["text"]
            }
        }
    }
]

def agentic_rag_loop(question: str, max_steps: int = 3) -> str:
    """function calling ベースの Agentic RAG"""
    context = []

    for step in range(max_steps):
        # 現在のコンテキストでfunction callを要求
        tool_call = call_with_tools(
            f"Question: {question}\n"
            f"Retrieved so far: {json.dumps(context[:3])}\n"
            f"Call a search tool or answer directly.",
            ARAG_TOOLS
        )

        if tool_call.get("name") == "answer":
            return tool_call["arguments"]["text"]

        elif tool_call.get("name") == "keyword_search":
            results = keyword_search(tool_call["arguments"]["query"])
            context.extend(results)

        elif tool_call.get("name") == "semantic_search":
            results = semantic_search(tool_call["arguments"]["query"])
            context.extend(results)

    # max_stepsに達したら強制回答
    return call_with_tools(
        f"Based on: {context}\nAnswer: {question}",
        [ARAG_TOOLS[2]]  # answerツールのみ
    )["arguments"]["text"]

Qwen3.5 4B（97.5%精度）でこのループを回せば、function callingの失敗でエージェントが暴走するリスクを最小化できる。しかも3.4GBなので、BGE-M3 Embedding（1.5GB）と同時に起動しても5GB以内。RTX 4060 8GBに余裕で収まる。

避けるべきモデルと選ぶべきモデル

避けるべきモデル

LM Studio環境で精度が出なかったモデル:

モデル	サイズ	精度	理由
Mistral Small 3.2 24B	15GB	42.5%	VRAMに見合わない精度
DeepSeek-R1-Distill 14B	9GB	57.5%	推論特化、構造化出力は苦手

※ xLAM-2 8B FC-R（15%）とHammer 2.1（20%）はLM Studioのchat template非互換が原因の可能性が高いため、モデル品質の評価としては参考にならない。

用途別推奨

用途	推奨モデル	サイズ	精度	理由
Agentic RAG	Qwen3.5 4B	3.4GB	97.5%	最高精度 + Embedding同時起動可
マルチターンAgent	Mistral Nemo 12B	7.5GB	92.5%	シーケンシャル処理に強い
コスト最小	Nemotron Nano 4B	4.2GB	95.0%	高精度 + 省VRAM
速度最優先	GLM-4.7-Flash	18GB	95.0%	52 t/s (ただしVRAM大)
8GBで知識+ツール	Qwen3.5 4B + 32B切替	可変	可変	ツール呼び出しは4B、知識回答は32B

最後のパターンが興味深い。ツール呼び出し判定には3.4GBの小さいモデル、実際の知識回答には32Bの大きいモデル、という2段構成。ツール選択の精度と知識の深さを両立できる。

function callingはローカルLLMの次のフロンティア

この記事の要点:

function calling精度はモデルサイズだけでは予測できない: 3.4GBのQwen3.5 4Bが25GBモデルを上回った（LM Studio環境）
構造化出力は知識量より形式遵守能力に依存する傾向がある: 大きいモデルの強み（知識量）が活きにくいタスク
RTX 4060 8GBはfunction callingの好環境: Top3モデルがすべて8GB以内に収まる
GBNF文法制約でJSON構文エラーを大幅に削減可能: llama.cppの強力機能
ベンチマーク結果は推論サーバー依存: LM Studioの互換性が下位モデルのスコアに影響している

ローカルLLMの価値は「大きいモデルを動かすこと」から「適切なモデルを適切なタスクに使うこと」に移行している。function callingはその象徴的な例だ。ただし、推論サーバーとの互換性が結果を大きく左右するため、自分の環境で検証することが前提になる。

参考文献

JD Hodges. "I Tested 13 Local LLMs on Tool Calling | 2026 Eval Results" (2026)
"Small Language Models for Efficient Agentic Tool Calling" (2025) arXiv:2512.15943
"TinyAgent: Function Calling at the Edge" (2024) arXiv:2409.00608
"ToolACE: Winning the Points of LLM Function Calling" (2024) arXiv:2409.00920

RAGの検索精度を3軸で測ったら最適解が条件で全く変わった

plasmon — Tue, 07 Apr 2026 00:54:28 +0000

RAGの検索精度を3軸で測ったら最適解が条件で全く変わった

RAG（Retrieval-Augmented Generation）を組むとき、embeddingモデル・検索アルゴリズム・チャンクサイズの3つを「なんとなく」で選んでいないだろうか。

「BGE-M3が安定」「ベクトル検索で十分」「チャンクは500-1000文字」。よく見るアドバイスだ。しかし、この3軸を日本語テクニカルコーパスで実測したら、デフォルト設定の落とし穴が見えてきた。

E5-small（384次元）がBGE-M3（1024次元）より高品質で9倍速い。BM25は形態素解析を入れるだけでJ-Mean@3が5.50→8.97に跳ね上がり、300文字チャンクが600文字・1200文字に圧勝した。

最大のインパクトは「アルゴリズムの選択」ではなく、「日本語トークナイザが壊れていた」という基盤の問題だった。この記事では実測データを全て公開した上で、あなたの条件に合った構成の選び方を整理する。

実験の設計

コーパス

日本語テクニカル記事（Zenn/Qiita）217件から生成した約1,500チャンク。AI・半導体・ハードウェア領域の技術記事が中心。英語コンテンツは含まない。

評価方法: Judge-based

検索結果の品質をLLMジャッジ（mmarco multilingual cross-encoder）が10点スケールで評価する。従来のrecall/precisionではなく、「検索されたチャンクが質問に対してどれだけ有用か」を直接測る。

主要指標:

J-MRR（Judge Mean Reciprocal Rank）: 最も関連性の高い結果が上位に来ているか
J-Mean@k: 上位k件の平均品質スコア（10点満点）
High-hit@k: 高品質チャンクがk件中に含まれる確率

3班分離

パイプライン（検索実行）・ジャッジ（品質評価）・分析（統計処理）を分離し、評価バイアスを抑制した。

軸1: Embeddingモデル — 小さい方が速くて品質も高かった

4つのembeddingモデルを同一コーパス・同一クエリで比較した。

モデル	次元	J-Mean@3	NDCG@3	速度 (texts/s)	埋め込み時間
E5-small	384	9.03	0.929	234	6.5秒
E5-large	1024	9.04	0.932	27	52.7秒
BGE-M3	1024	8.76	0.879	25	57.5秒
MiniLM	384	8.32	0.845	521	2.8秒

E5-smallはE5-largeと品質がほぼ同じ（J-Mean@3で0.01差）で、速度が9倍、VRAM使用量が約1/5。BGE-M3は品質でもE5-smallに負けている。

MiniLMは最速（521 texts/s）だが、k=10での品質低下が顕著（NDCG@10: 0.726 vs E5-small: 0.933）。上位3件だけ使う用途なら選択肢になるが、広く情報を集めたい場合は不向き。

なぜ小さいモデルが勝つのか

直感的には、1024次元の方が384次元より多くの情報をエンコードできるはずだ。しかし、1,500チャンクの日本語テック記事というコーパスに対して、1024次元は過剰だった可能性が高い。限られたドメインのテキストでは、384次元でも意味的な差異を十分に表現でき、むしろ空間が密に使われることで検索精度が安定する。

この解釈はスケーリング実験でも支持された。ノイズチャンクを追加して最大19,477チャンクまで拡大したテストで、E5-smallはBGE-M3よりもノイズ耐性が高かった（19K時点でMRR 0.856 vs 0.742）。

ただし注意: これは約20Kチャンクまでの結果。10万チャンク超の大規模コーパスでは逆転する可能性がある。

軸2: 検索アルゴリズム — 壊れたトークナイザが全ての元凶だった

5つの検索戦略を比較した。最も重要な発見がここで出た。ただし、その意味を正確に理解するには順を追って見る必要がある。

まず、形態素解析なしの結果（embeddingモデル: BGE-M3）

戦略	J-MRR	J-Mean@3	備考
dense（ベクトル検索）	0.918	6.81	BGE-M3での意味的類似度検索
hybrid RRF	0.825	5.91	dense + BM25のスコア融合
BM25（.split()）	0.802	5.50	スペース分割トークナイザ

ベクトル検索がBM25に圧勝。よく見る「BM25は古い」「denseが強い」という話と一致する。ここまでは定説通り — に見えた。

形態素解析を入れた瞬間

戦略	J-MRR	J-Mean@3	備考
BM25（fugashi）	1.000	8.97	形態素解析トークナイザ
hybrid RRF（fugashi）	1.000	8.76	dense + BM25(fugashi)
dense（BGE-M3）	0.918	6.81	変化なし

BM25のJ-Mean@3が5.50から8.97に跳ねた。これは+63%の改善だが、「BM25がベクトル検索に勝った」と解釈するのは早い。

正しい比較: 同一embeddingモデルで見る

上の表には落とし穴がある。denseの6.81はBGE-M3での結果だ。軸1の実験で確認した通り、E5-smallのdense検索はJ-Mean@3 = 9.03を出している。

戦略	J-Mean@3	備考
E5-small dense	9.03	軸1実験より
BM25 fugashi	8.97	形態素解析あり
BGE-M3 dense	6.81	軸2のベースライン
BM25 .split()	5.50	形態素解析なし

E5-small denseとBM25 fugashiはほぼ互角（差はわずか0.06）。つまり、BM25が勝ったのではなく、壊れたトークナイザを修正したら、BM25が本来の性能を取り戻したというのが正確な描写だ。

MRRが1.000（29クエリ全問で最上位にヒット）になったのも、コーパスが1,500チャンク・日本語テック記事という狭いドメインで、技術用語のキーワード一致が効きやすい条件だったことが大きい。汎用ドメインや大規模コーパスでは異なる結果になり得る。

本当の教訓: デフォルト設定が日本語を壊している

日本語にはスペースがない。.lower().split()で分割すると、文全体が1トークンになるか、改行位置でしか切れない。つまりBM25がまともに動いていなかった。

fugashi（MeCab）で形態素解析すると、「半導体」「アーキテクチャ」「推論」のような技術用語が正しく分割される。BM25は本来キーワード一致に強いアルゴリズムで、正しいトークナイザさえあればベクトル検索と互角以上になる。

これは日本語（および中国語など、スペースで単語が区切られない言語）に特有の問題だ。英語では.split()で概ね正しくトークン化されるため、同じ問題は起きにくい。LangChainやLlamaIndexのデフォルト設定で日本語BM25を使うと、能力の半分も出ていない可能性がある。

「ハイブリッド検索が最強」は条件つき

LangChainやLlamaIndexのドキュメントでは「hybrid検索を使いましょう」が定石だが、今回の実測ではBM25単独（fugashi）がhybrid RRFをわずかに上回った。ベクトル検索の結果を混ぜることで、BM25の完全一致のシグナルが薄まっているためと考えられる。

ただし、これは技術用語の完全一致が重要な狭いコーパスでの結果だ。意味的に近い文を探したい場合（例: 同義語での検索、抽象的な質問への回答）や、コーパスが大規模化した場合は、denseやhybridの方が適している可能性が高い。

軸3: チャンクサイズ — 小さいチャンクが一貫して勝った

3つの固定サイズチャンクを比較した。

サイズ	チャンク数	J-MRR	J-Mean@3	NDCG@3
300文字	2,877	0.855	6.48	0.566
600文字	1,497	0.825	5.91	0.493
1,200文字	718	0.785	3.97	0.355

300→1200で品質がほぼ半減。チャンクが大きくなるほど、関連情報がノイズに埋もれて検索精度が落ちる。

NVIDIAのベンチマーク（2025）でも256-512トークンが最適とされており、300文字（約150-200トークン）は妥当な範囲内にある。また、NAACL 2025で発表されたVectaraの研究では、固定サイズ分割がセマンティック分割を一貫して上回った。

LLMの長文脈対応とチャンクサイズは別問題

「LLMのコンテキスト窓が128Kになったからチャンクを大きくすべき」という意見もあるが、これは検索と生成を混同している。検索の精度は「必要な情報だけを的確に取り出せるか」で決まる。大きなチャンクは関連情報を含む確率は上がるが、無関係な情報も同時に持ち込む。検索精度と生成品質は別の最適点を持つ。

MMR（多様性検索）: 品質が壊滅した

Maximal Marginal Relevance（MMR）は検索結果の多様性を上げる手法として知られるが、今回のテストでは品質が壊滅した。

戦略	J-Mean@3	Source Diversity@3
BM25 fugashi	8.97	0.51
hybrid MMR (λ=0.7)	1.12	0.84
hybrid MMR (λ=0.5)	0.98	0.86

多様性は0.51→0.86に向上したが、品質が10点満点中1点に崩壊。λを0.7に上げても回復しない。

記事生成のように多角的な情報が必要な用途では多様性は重要だが、MMRは現時点で品質を犠牲にしすぎる。クラスタリングベースの多様性確保など、別のアプローチが必要。

あなたの条件で選ぶ: 構成選定マトリクス

あなたの条件	Embedding	検索	チャンク	根拠
日本語 + 技術用語多い + ~10Kチャンク以下	E5-small	BM25 + 形態素解析 or dense	300文字	本実験の実測構成。BM25 fugashiとE5-small denseは品質ほぼ互角、速度・VRAMはE5-small優位
日本語 + 汎用テキスト + 大規模 (10K-100K+)	BGE-M3 or E5-large	hybrid RRF + 形態素解析	600文字	大規模ではdenseの意味検索が効いてくる可能性（未検証）
英語中心	E5系 or OpenAI ada-002	dense or hybrid	512トークン	英語ではBM25の形態素解析問題が発生しない
多言語混在	BGE-M3	hybrid RRF	600文字	BGE-M3の多言語対応が活きる
とにかく速度重視	MiniLM	BM25	300文字	品質はやや落ちるが最速
多様性が必要	—	MMR以外を検討	—	MMRは品質壊滅。クラスタリングベース等を要検討

最も重要な1つだけ変えるなら

日本語RAGでまず最初に見直すべきはBM25のトークナイザだ。

今回の全実験で最大の効果量は、BM25のトークナイザを.split()からfugashi形態素解析に変えた時のJ-Mean@3: 5.50→8.97（+63%）。これは「BM25の最適化」ではなく「壊れていたトークナイザの修正」だ。正しくトークン化されたBM25はベクトル検索と互角の品質を出す。embeddingモデルの変更（+3%）やチャンクサイズの変更（+10%）に比べて、影響の大きさが桁違いに大きい。

LangChainのデフォルト構成でhybrid searchを使っている場合、BM25側のトークナイザが.split()のままになっていないか確認してほしい。日本語テキストに対してスペース分割は機能しない。fugashi（MeCab）やSudachi、Janeomeなどの形態素解析器を入れるだけで、検索品質が回復する。

この実験の限界

コーパスが狭い: 日本語テック記事1,500チャンクでの結果。英語テキスト、汎用コーパス、10万チャンク級では異なる結果になり得る。一般に英語のopen-domain QAベンチマークでは、hybrid検索がBM25単独を上回る傾向が報告されている
Judge biasの可能性: LLMジャッジは短く焦点の絞られた回答を好む傾向がある。小チャンクが「実際に良い」のか「ジャッジが好む」だけなのかは区別できていない
End-to-endの検証がない: 検索品質の改善が最終的な生成品質（記事の質）に繋がるかは未検証
MMRのλ範囲が不十分: λ=0.5と0.7のみテスト。0.9や0.95での結果は未確認

大規模・広範なコーパスでの追試を計画している。結果が出たら続報を書く。

実装: 最小構成のコード

E5-small + BM25 fugashi + 300文字チャンクの最小実装を示す。

形態素解析トークナイザ

import fugashi

tagger = fugashi.Tagger()

def tokenize_ja(text: str) -> list[str]:
    """日本語テキストを形態素解析でトークン化する"""
    tokens = []
    for word in tagger(text):
        surface = word.surface.strip()
        if not surface:
            continue
        # 1文字ひらがな（助詞等）を除外
        if len(surface) == 1 and surface in "はがのをにへとでやかもば":
            continue
        tokens.append(surface.lower())
    return tokens

BM25検索（形態素解析版）

from rank_bm25 import BM25Okapi

# ドキュメントのトークン化
tokenized_docs = [tokenize_ja(doc) for doc in documents]
bm25 = BM25Okapi(tokenized_docs)

# 検索
query_tokens = tokenize_ja("E5-smallの検索精度")
scores = bm25.get_scores(query_tokens)

.lower().split()をこのtokenize_jaに置き換えるだけで、検索品質が大幅に改善する。

Embeddingモデルの変更

from sentence_transformers import SentenceTransformer

# BGE-M3の代わりにE5-smallを使う
model = SentenceTransformer("intfloat/multilingual-e5-small")

# E5はプレフィックスが必要
query_vec = model.encode("query: E5-smallの検索精度")
doc_vec = model.encode("passage: E5-smallは384次元のembeddingモデルで...")

参考文献

Vectara. "Chunking Strategies and Embedding Models for RAG" (NAACL 2025) — 25チャンキング手法×48 embeddingモデルの網羅的比較で、固定サイズ分割がセマンティック分割を一貫して上回ることを確認
NVIDIA. "Finding the Best Chunking Strategy for Accurate AI Responses" (2025) — 256-512トークンが最適範囲
Milvus Blog. "Best Embedding Model for RAG 2026" — E5-smallが70倍大きいモデルを上回った事例を報告
Ahogrammer. "ハイブリッド検索で必ずしも検索性能が上がるわけではない" — トークン化の違いによる性能差を指摘
Zenn fp16. "日本語RAGのEmbeddingモデル、結局どれが最強なのか？" (2026) — 6構成2000問の日本語ベンチマーク

They Routed Power Through the Back of the Chip and 30% IR Drop Vanished

plasmon — Mon, 06 Apr 2026 23:02:29 +0000

They Routed Power Through the Back of the Chip and 30% IR Drop Vanished

Every semiconductor chip has a front and a back. Transistors and wiring are built on the front side, and power delivery shares that same front surface with signal routing. It has been this way for over 60 years.

In 2026, Intel broke this convention. They implemented BSPDN (Backside Power Delivery Network) — branded PowerVia — in their 18A process node, shipping it in the Panther Lake processor as a mass-produced product.

The result: 30% IR drop reduction, 6% frequency uplift, and 5-10% standard cell utilization improvement (Intel's CES 2026 announced figures).

If the front side is congested, use the back. It sounds obvious in hindsight, but getting there required decades of process engineering. This article breaks down the physics behind BSPDN, the Intel vs TSMC vs Samsung race, and what it means for AI chips.

Why Power Delivery Became the Bottleneck

Front-Side Congestion

Look at a cross-section of any leading-edge chip: 10+ metal layers are stacked on top of the transistor layer.

Conventional chip cross-section (front-side only):

  Surface
  ─────────────────────────
  M12  Power rails (VDD/VSS) ← thick wires
  M11  Power + signal         ← shared
  M10  Signal routing
  ...
  M3   Signal routing         ← thin
  M2   Signal routing         ← thinner
  M1   Local interconnect     ← thinnest
  ─────────────────────────
  Transistor layer (FinFET/GAA)
  ─────────────────────────
  Silicon substrate
  ─────────────────────────
  Backside (nothing here)

The problem: power lines and signal lines share the same metal layers. Power rails demand thick wires, eating into the space available for signal routing.

Of the 12 metal layers, 3-4 (M10-M12) are consumed by power rails. The remaining 8-9 (M1-M9) carry signals, but power vias — vertical connections that punch through every layer — eat into signal space too.

It gets worse with every node shrink:

Node	Signal Congestion	IR Drop
5nm	Moderate	Manageable
3nm	High	Design problem
2nm	Critical	Design-limiting

At the 2nm generation, signal routing density physically runs out. Power infrastructure consumes too much space to pack in more logic.

IR Drop: A Power Quality Problem

IR drop is the voltage loss caused by resistance in the power delivery network. The longer the path from the power pin to the transistor, the more voltage you lose.

Power delivery path (conventional):
  Package → bonding pad → M12 → M11 → ... → M1 → transistor

  Each metal layer connected vertically by vias
  Horizontal runs add resistance too
  Long path → high resistance → large voltage drop

  The center of the chip suffers the most:
    Pads are at the periphery → long distance to reach center
    → Center transistors operate at lower voltage than edges
    → Must raise global voltage to maintain margin
    → Increased power consumption

The Physics of BSPDN: Power From the Back

BSPDN is built on a simple idea: move the power wiring from the front to the back of the chip.

BSPDN chip cross-section:

  Surface
  ─────────────────────────
  M12  Signal routing  ← power lines gone, space freed
  M11  Signal routing  ← all layers available for signals
  M10  Signal routing
  ...
  M1   Local interconnect
  ─────────────────────────
  Transistor layer (GAA: gate wraps all around the channel)
  ─────────────────────────
  nTSV (nano-TSV)  ← vertical power feed from backside
  ─────────────────────────
  Backside power grid (VDD/VSS)  ← thick wires, plenty of room
  ─────────────────────────
  Backside

How It's Built

1. Flip the wafer
   Build transistors + signal wiring on the front side,
   then thin the wafer and flip it over.

2. Nano-TSV (nTSV) formation
   Etch tiny vias through the silicon substrate
   reaching down to the transistor contact layer.
   Power is delivered directly to the transistors.

3. Backside metallization
   Form a VDD/VSS power grid on the backside.
   Wires can be thicker than front-side (no signal contention).
   IR drop is drastically reduced.

Quantified Gains (Intel 18A PowerVia)

Metric	Conventional (front-side power)	BSPDN	Improvement
IR drop	Baseline	-30%	Better voltage stability
Clock frequency	Baseline	+6%	Higher speed at same power
Standard cell utilization	Baseline	+5-10%	Power vias removed, more room for logic

A 6% frequency gain may look modest in isolation, but this is achieved without shrinking the process node. Normally, a 6% clock uplift is what you get from migrating to the next node. BSPDN alone delivers roughly half a node's worth of performance — for free, in architectural terms.

Intel vs TSMC vs Samsung: The BSPDN Race

Intel 18A: First Mover

Intel 18A PowerVia:
  - Mass production started early 2026 (Panther Lake)
  - Officially announced at CES 2026
  - Combines RibbonFET (GAA) + PowerVia
  - Foundry customers: Microsoft, Amazon
  - Yield: 60-65% (early 2026; recent reports suggest 65-75%). Target: 70-80%

  Technical details:
  - nTSV connects to transistor contact layer
  - Dedicated coarse-pitch power grid on backside
  - Complete separation from signal routing

TSMC: Late but Ambitious

TSMC Super Power Rail (A16, H2 2026 target):
  - TSMC's version of BSPDN
  - Claims more advanced than Intel PowerVia
  - Power connects directly to transistor source/drain
  - N2 (2nm) ships without BSPDN → A16 introduces it

  TSMC's strategy:
  - Ship N2 first (no BSPDN, but GAA)
  - Add BSPDN in A16 (improved N2)
  - Offer customers a clean N2 → A16 migration path

Samsung is also planning BSPDN introduction with their SF2Z node in 2027, but specific technical details remain undisclosed.

The Competitive Landscape

Foundry	Node	BSPDN Approach	Status	Risk
Intel	18A	PowerVia (nTSV → contact layer)	In production (early 2026)	Yield improvement is key to profitability
TSMC	A16	Super Power Rail (direct source/drain)	H2 2026 target	No shipping products yet. Yield unknown
Samsung	SF2Z	Undisclosed	2027 target	3nm yield issues still lingering

On BSPDN specifically, Intel leads TSMC by roughly a year — the first time in nearly a decade that Intel has been ahead on a major process technology. It is not overall process leadership, but BSPDN is the lifeline of Intel's foundry business.

Three Reasons BSPDN Matters for AI Chips

1. Power Efficiency: Better J/token

AI chip power consumption keeps climbing. The H100 draws 700W, the B200 hits 1000W (1200W for the full-spec variant). Data center electricity costs are starting to dominate inference economics.

BSPDN impact on AI chip power:
  30% IR drop reduction → smaller voltage margin → effective power savings

  Example: hypothetical A100 with BSPDN
    Current: 400W (~15% of which is IR drop margin)
    BSPDN: 30% reduction in IR drop margin → ~18W savings
    Inference cost: ~4.5% improvement

4.5% sounds small until you run the numbers at data center scale — it translates to millions of dollars annually.

2. Logic Density: Packing More Tensor Cores

With power vias removed from the front side, standard cell utilization improves 5-10%. That freed space can be filled with additional compute logic:

Extra tensor cores �� higher inference throughput
Larger L2 cache → on-chip KV cache for LLM serving
Dedicated PIM units → alleviate memory bandwidth bottleneck

3. Better Thermal Management

Moving power wiring to the backside reduces metal density on the front side. Dense metal creates localized hot spots.

Conventional: Power + signal on front → high metal density → heat traps
BSPDN: Power on backside → lower front-side metal density → better heat spreading
        + shorter cooling path from backside

AI chips have a specific problem:
  Tensor cores run dense computation → localized heating
  BSPDN: shorter power path → less Joule heating generated in the first place

The Cost of BSPDN: The Backside Has Walls Too

BSPDN is not free.

Additional Process Steps

Conventional process:
  Front-side lithography + etch + metallization
  → Done

BSPDN process:
  Front-side lithography + etch + signal metallization
  → Wafer thinning (backgrinding)
  → Wafer bonding (attach to carrier wafer)
  → nTSV formation (backside drilling)
  → Backside metallization
  → Additional lithography
  → Done

Additional steps: 5-8

Cost Implications

Additional Step	Est. Cost Premium
Wafer thinning (backgrind + CMP)	+5%
Carrier wafer bonding	+3%
nTSV lithography + etch	+8%
Backside metallization	+5%
Backside patterning	+4%

The total wafer cost increase is substantial, and yield loss risk compounds it. In return: +6% frequency, +5-10% cell utilization, ~4.5% power efficiency. For high-performance AI chips, the gains absorb the cost premium. For cost-sensitive mobile or IoT chips, BSPDN may be overinvestment.

When Does BSPDN Reach Consumer GPUs?

The timeline for BSPDN in consumer GPUs (GeForce, Radeon) is unclear.

NVIDIA:
  Current (Blackwell): No BSPDN
  Next gen Rubin (R100): TSMC N3P. No BSPDN
  Rubin Ultra: TSMC N2 (2027). BSPDN adoption uncertain
  Post-Rubin Ultra: A16 (with BSPDN) is a possibility

AMD:
  Current (RDNA 4): No BSPDN
  Next gen: Depends on TSMC N2 or A16

Consumer reality:
  2026-2027: No BSPDN (N2 generation)
  2028+: A16-based products may appear
  Cost premium will be passed through to GPU prices

As an RTX 4060 8GB user, the tangible benefits of BSPDN probably won't arrive until the generation after next — a 2028-2029 GPU purchase, most likely. However, data center GPUs (successors to H100/B200) may adopt BSPDN as early as 2027. If you use inference APIs, you'll benefit from BSPDN indirectly.

The Backside Was Empty All Along

The essence of BSPDN is a deceptively simple insight: the backside of the chip was unused. Use it.

For 60 years, the back of a chip was a featureless plane. By relocating power infrastructure to that space:

Front-side routing congestion is resolved
The power delivery path gets shorter, reducing IR drop
Frequency goes up, logic density increases

Intel leads, TSMC follows, Samsung waits in the wings. 2026 will be recorded as the first year BSPDN entered mass production in semiconductor history.

Of the three walls facing the 2nm generation — thermal, power, and scaling — BSPDN partially breaks through the power wall. The other walls will need their own innovations, but the paradigm shift of using the backside hints at where semiconductor design is headed.

There was space left after all. You just had to look at it from the other side.

References

Intel. "Panther Lake: Intel 18A with PowerVia and RibbonFET" (CES 2026)
fiisual. "What Is Backside Power Delivery Network (BSPDN)?" (2026) (industry analysis blog)
TrendForce. "Intel's Clearwater Forest Unveils 18A Backside Power" (2025)
"Backside Power Delivery: A Radical Shift in Chip Architecture" (2026)
Lam Research. "Transistor Channel Stress in Backside Power Delivery Networks"
TSMC. A16 Super Power Rail (H2 2026 planned)

Letting AI Control RAG Search Improved Accuracy by 79%

plasmon — Mon, 06 Apr 2026 03:25:05 +0000

Letting AI Control RAG Search Improved Accuracy by 79%

Most RAG (Retrieval-Augmented Generation) search pipelines are built like this:

Query → vector search → Top-K retrieval → dump everything into LLM

This fixed pipeline is the root cause limiting RAG accuracy.

A February 2026 ArXiv paper (arXiv:2602.03442) proposed A-RAG (Agentic RAG), replacing the fixed search pipeline with an AI agent. Result: multi-hop QA accuracy improved by 79% (50.2% → 89.7%). And retrieved tokens dropped by half.

Higher accuracy with less retrieval. Here's how this counter-intuitive result works.

Three Limits of Fixed-Pipeline RAG

Limit 1: Weak on Multi-Hop Questions

Question: "Where did the person who invented X attend university?"

Required searches:
  Round 1: "Who invented X" → identify the person
  Round 2: "That person's university" → get the answer

Fixed pipeline:
  One vector search for "inventor of X + university"
  → No chunk directly contains the answer
  → Retrieves many low-relevance chunks
  → LLM guesses → inaccurate

Multi-hop questions make up a substantial share of real queries. Fixed pipelines are structurally weak against questions that can't be answered in one search.

Limit 2: Fixed Retrieval Granularity

With Top-K=5:
  Simple question → retrieves 5 chunks → token waste
  Complex question → retrieves 5 chunks → insufficient information

Required granularity varies per question:
  "What's GPT-4's parameter count?" → 1 chunk is enough
  "How does GPT-4 vs Claude 3.5 differ on long context?" → 10 chunks needed

Limit 3: Fixed Search Strategy

Vector search only:
  Retrieves semantically similar chunks
  → Weak on exact matches (part numbers, proper nouns)

Keyword search only:
  Retrieves exact/partial matches
  → Weak on synonyms and paraphrases

Hybrid search (fixed ratio):
  70% vector + 30% keyword (or similar fixed weights)
  → Can't dynamically adjust based on question type

A-RAG Architecture: Let the Agent Search

A-RAG's core insight: replace the fixed search flow with agent decision-making.

Traditional RAG:
  Query → [fixed pipeline] → chunks → LLM → answer
  Search method, granularity, and count all pre-determined

A-RAG:
  Query → [agent] → answer
  Agent autonomously decides:
    - Which search tool to use
    - How many times to search
    - What granularity to retrieve at
    - When to stop searching

Three Search Interfaces

A-RAG gives the agent three tools:

class ARAGTools:
    def keyword_search(self, query: str) -> list[str]:
        """Keyword-based search
        Use case: proper nouns, part numbers, exact terms"""
        pass

    def semantic_search(self, query: str) -> list[str]:
        """Vector similarity search
        Use case: conceptual similarity, paraphrase handling"""
        pass

    def chunk_read(self, doc_id: str, chunk_range: str) -> str:
        """Deep read of specific chunks
        Use case: drilling into search results, getting surrounding context"""
        pass

The agent freely combines these tools based on the question.

Multi-Hop Question Example

Question: "What is the current affiliation of the first author
           of the paper that proposed Transformers?"

Agent behavior:
  Step 1: keyword_search("Transformer paper original authors")
    → "Attention Is All You Need", Vaswani et al., 2017
  Step 2: semantic_search("Ashish Vaswani current affiliation 2026")
    → Retrieves 3 chunks
  Step 3: chunk_read(doc_id="result_2", range="full")
    → Deep reads detailed info
  Step 4: Generate answer → "Essential AI (startup founded 2023)"

Fixed pipeline would:
  vector_search("first author current affiliation Transformer paper")
  → Unlikely to get direct answer in one search
  → Mostly retrieves content about "Attention Is All You Need"
  → Risk of answering with 2017-era Google affiliation

Benchmark Results: A-RAG by the Numbers

Key results from the paper (Table 1).

GPT-4o-mini Backend

Benchmark	Naive RAG	A-RAG	Improvement
MuSiQue	38.6%	46.1%	+19%
HotpotQA	74.5%	77.1%	+3.5%
2WikiMultiHopQA	42.6%	60.2%	+41%

GPT-5-mini Backend

Benchmark	Naive RAG	A-RAG	Improvement
MuSiQue	52.8%	74.1%	+40%
HotpotQA	81.2%	94.5%	+16%
2WikiMultiHopQA	50.2%	89.7%	+79%

Pattern Analysis

patterns = {
    "multi_hop_improvement": {
        "2Wiki": "+41% (4o-mini) / +79% (5-mini)",
        "MuSiQue": "+19% (4o-mini) / +40% (5-mini)",
        "insight": "Bigger gains on multi-hop questions"
    },
    "model_scaling": {
        "4o_mini_avg": "+21%",
        "5_mini_avg": "+45%",
        "insight": "Stronger models benefit more from A-RAG"
    },
    "graphrag_comparison": {
        "HotpotQA_graphrag_4o_mini": "33.2%",
        "HotpotQA_graphrag_5_mini": "82.5%",
        "HotpotQA_naive_rag": "74.5% / 81.2%",
        "insight": "GraphRAG is extremely model-dependent. Collapses with weak models"
    }
}

Three key takeaways:

Dominant improvement on multi-hop: +79% on 2WikiMultiHopQA. A-RAG is strongest where fixed pipelines are weakest
Scales with model capability: GPT-5-mini gains more than GPT-4o-mini. Agent search quality depends on model intelligence
GraphRAG is extremely model-dependent: With GPT-4o-mini, HotpotQA drops to 33.2% (less than half of Naive RAG). But with GPT-5-mini, GraphRAG hits 82.5%, beating Naive RAG's 81.2%. Weak-model GraphRAG is dangerous

Token Efficiency

HotpotQA (GPT-5-mini):
  Naive RAG: 5,358 tokens retrieved → 81.2% accuracy
  A-RAG:     2,737 tokens retrieved → 94.5% accuracy

Retrieved tokens: -49%
Accuracy: +16%

Less retrieval, higher accuracy. The agent selectively retrieves only what's needed, reducing noise and improving LLM answer quality. This directly impacts API costs.

Can Agentic RAG Run on Local LLMs?

The paper uses GPT-4o-mini and GPT-5-mini. What about local models?

Structural Challenges

agent_requirements = {
    "tool_use": "Function calling capability",
    "planning": "Multi-step planning",
    "reflection": "Evaluating search results, deciding next action",
    "context_management": "Maintaining and integrating retrieved info",
}

local_llm_capability = {
    "Qwen2.5-32B Q4_K_M": {
        "tool_use": "Supported (ChatML format)",
        "planning": "Moderate (simple 2-3 steps)",
        "speed": "~10 t/s (ngl=24)",
        "verdict": "Simple Agentic RAG works, complex multi-hop is difficult"
    },
    "Qwen3.5-9B Q4_K_M": {
        "tool_use": "Supported",
        "planning": "Moderate",
        "speed": "~33 t/s",
        "verdict": "Fast but knowledge-limited. Search judgment quality may drop"
    }
}

Minimal Implementation

# Local LLM + ChromaDB minimal Agentic RAG
import chromadb
import subprocess, json

class LocalAgenticRAG:
    def __init__(self, db_path, model_path):
        self.chroma = chromadb.PersistentClient(path=db_path)
        self.collection = self.chroma.get_collection("papers")
        self.model = model_path

    def keyword_search(self, query, k=5):
        results = self.collection.query(
            query_texts=[query], n_results=k,
            where_document={"$contains": query.split()[0]}
        )
        return results["documents"][0]

    def semantic_search(self, query, k=5):
        results = self.collection.query(
            query_texts=[query], n_results=k
        )
        return results["documents"][0]

    def agent_query(self, question):
        context = []
        for step in range(3):
            ctx_str = json.dumps(context, ensure_ascii=False)[:2000]
            prompt = f"""Question: {question}
Tools: keyword_search, semantic_search, chunk_read
Retrieved: {ctx_str}
Decide: TOOL:name(args) or ANSWER:your answer"""

            response = self._llm_call(prompt)
            if response.startswith("ANSWER:"):
                return response[7:]
            elif response.startswith("TOOL:"):
                context.append(self._execute_tool(response[5:]))

        return self._llm_call(
            f"Based on: {json.dumps(context, ensure_ascii=False)}\nAnswer: {question}"
        )

    def _execute_tool(self, tool_call):
        if tool_call.startswith("keyword_search"):
            q = tool_call.split("(", 1)[1].rstrip(")")
            return str(self.keyword_search(q.strip("'\"")))
        elif tool_call.startswith("semantic_search"):
            q = tool_call.split("(", 1)[1].rstrip(")")
            return str(self.semantic_search(q.strip("'\"")))
        elif tool_call.startswith("chunk_read"):
            doc_id = tool_call.split("(", 1)[1].rstrip(")")
            return self.chunk_read(doc_id.strip("'\""))
        return ""

This works, but don't expect the paper's +79%. Local LLM tool_use capability is the limiting factor.

Realistic Expectations

expected_improvement = {
    "32B_model": {
        "multi_hop": "+15-25% (roughly 1/3 of paper results)",
        "single_hop": "+3-5%",
    },
    "9B_model": {
        "multi_hop": "+5-10%",
        "single_hop": "+1-3%",
    },
    "recommendation": "32B+ needed for meaningful Agentic RAG benefits"
}

Before You Try Agentic RAG

A-RAG is compelling but not universally necessary.

Agentic RAG makes sense when:
  ✓ Multi-hop questions are frequent (research, investigation)
  ✓ Large knowledge base (1000+ chunks)
  ✓ Variable question complexity
  ✓ Accuracy is top priority (medical, legal)

Naive RAG is sufficient when:
  ✓ Single-hop questions dominate (FAQ, manual lookup)
  ✓ Small knowledge base (< 100 chunks)
  ✓ Uniform question patterns
  ✓ Latency is top priority

Cost Structure

cost_comparison = {
    "naive_rag": {
        "llm_calls": 1,
        "search_calls": 1,
        "avg_tokens": 5358,
        "latency": "1-2s (API) / 5-10s (local 32B)",
    },
    "agentic_rag": {
        "llm_calls": "2-4",
        "search_calls": "2-5",
        "avg_tokens": 2737,
        "latency": "3-8s (API) / 15-40s (local 32B)",
    },
}
# Price of +79% accuracy: 3-4x latency
# ROI: Worth considering if multi-hop queries exceed 30% of traffic

The Next Evolution of RAG Is Agentic

The summary:

Fixed-pipeline RAG is limited by search design: Multi-hop weakness, fixed granularity, fixed strategy
A-RAG lets the agent decide how to search: Three tools (keyword/semantic/chunk_read) selected autonomously
+79% improvement on multi-hop: Biggest gains where fixed pipelines are weakest
Less retrieval, higher accuracy: -49% tokens, +16% accuracy
Stronger models benefit more: Local LLMs see smaller gains

RAG is evolving from human-designed search optimization to model-driven search decisions. Fixed pipelines are stable and predictable but can't adapt to question diversity. Agents are less predictable but adaptive.

If your RAG system uses a fixed pipeline, measure your multi-hop question rate first. If it exceeds 30%, Agentic RAG is worth investigating.

References

"A-RAG: Scaling Agentic Retrieval-Augmented Generation via Hierarchical Retrieval Interfaces" (2026) arXiv:2602.03442
"Retrieval-Augmented Generation: A Comprehensive Survey" (2025) arXiv:2506.00054
"Ragas: Automated Evaluation of Retrieval Augmented Generation" (2023) arXiv:2309.15217

If Memory Could Compute, Would We Still Need GPUs?

plasmon — Sun, 05 Apr 2026 23:46:10 +0000

If Memory Could Compute, Would We Still Need GPUs?

The bottleneck for LLM inference isn't GPU compute. It's memory bandwidth.

A February 2026 ArXiv paper (arXiv:2601.05047) states it plainly: the primary challenges for LLM inference are memory and interconnect, not computation. GPU arithmetic units spend more than half their time idle, waiting for data to arrive.

So flip the paradigm. Compute where the data lives, and data movement disappears. This is the core idea behind Processing-in-Memory (PIM). SK Hynix's AiM is shipping as a commercial product. Samsung announced LPDDR5X-PIM in February 2026. HBM4 integrates logic dies, turning the memory stack itself into a co-processor.

Is the GPU era ending? Short answer: no. But PIM will change LLM inference architecture. How far the change goes, and where it stops — that's what the papers and product data reveal.

The Memory Wall: Why GPUs Sit Idle

LLM inference has two phases with different bottlenecks:

Prefill phase (prompt processing):
  Batch-processes many tokens at once
  Matrix-matrix multiply → Compute-bound
  GPU arithmetic units fully utilized

Decode phase (token generation):
  Generates one token at a time, autoregressively
  KV cache reads → Memory bandwidth-bound
  GPU arithmetic units idle, waiting for data

The problem is the Decode phase. Most LLM inference time is spent in Decode. And Decode is memory bandwidth-limited.

# Memory bandwidth bottleneck on RTX 4060 8GB
rtx_4060_specs = {
    "compute": "15.11 TFLOPS (FP16)",
    "memory_bandwidth": "272 GB/s",
    "required_arithmetic_intensity": "15110 / 272 = 55.6 FLOP/byte",
}

# Actual arithmetic intensity during LLM Decode
llm_decode = {
    "typical_arithmetic_intensity": "1-2 FLOP/byte",
    "bottleneck": "memory bandwidth (272 GB/s wall)",
    "gpu_utilization": "< 5% of compute capacity during decode",
}

# 95%+ of GPU compute sits idle during Decode

The A100 80GB has the same structure. HBM2e bandwidth: 2 TB/s. Compute: 312 TFLOPS. Required arithmetic intensity: 156 FLOP/byte vs. Decode's 1-2 FLOP/byte. Bandwidth falls short by 50-100x.

PIM Principle: Don't Move Data, Move Computation

Traditional architecture:

DRAM/HBM → bus → GPU compute → bus → DRAM/HBM
  Data travels round-trip. Bus bandwidth is the bottleneck.

PIM architecture:

Compute units inside DRAM/HBM → only results output
  Data doesn't move. Computation moves to the data.
  Internal bandwidth is orders of magnitude higher than bus bandwidth.

HBM's internal bandwidth (aggregate across banks) is tens of times higher than external bandwidth. Computing inside HBM without moving data out eliminates the bandwidth wall.

Shipping Products

SK Hynix AiM (Accelerator in Memory):
  - Commercial PIM processor based on GDDR6
  - Compute units per memory bank (AiMX card shipping)
  - Deployed in production environments
  - Specialized for GEMV (matrix-vector multiply)

Samsung LPDDR5X-PIM (announced Feb 2026):
  - In-memory compute in mobile LPDDR5X
  - Significant energy efficiency improvement for edge AI inference (industry estimates: several-fold)
  - Targeting smartphones and edge devices

Samsung/SK Hynix HBM4 plans:
  - Logic die integrated into HBM stack
  - Memory stack becomes a co-processor
  - Mass production from February 2026
  - Targeting NVIDIA's "Rubin" architecture

How PIM Changes LLM Inference

2025-2026 ArXiv papers propose concrete PIM × LLM architectures.

HPIM: Heterogeneous PIM Integration (arXiv:2509.12993)

HPIM (Heterogeneous PIM) Architecture:

SRAM-PIM (low latency):
  - Attention score computation
  - Small but ultra-fast
  - Equivalent to GPU L2 cache position

HBM-PIM (high bandwidth, large capacity):
  - KV cache storage and processing
  - Large capacity, medium speed
  - Equivalent to main memory position

Parallel execution of both:
  SRAM-PIM: attention score ← low latency
  HBM-PIM: KV multiplication ← high bandwidth
  → Parallelizes serial dependencies in autoregressive Decode

This envisions PIM across the entire memory hierarchy. Both cache and main memory contribute computation, each leveraging its strengths.

PAM: Processing Across Memory Hierarchy (arXiv:2602.11521)

PAM (Processing Across Memory):

HBM-PIM:  Hot data (frequent access)
DRAM-PIM: Warm data (moderate access)
SSD-PIM:  Cold data (rare access)

→ Optimize processing location by data temperature
→ Handle long-context LLMs (100K+ tokens)

When the entire model doesn't fit in HBM (an everyday reality for us RTX 4060 8GB users), cross-hierarchy PIM could become an alternative to partial offloading.

Three Reasons PIM Can't Kill GPUs

PIM is compelling, but it won't make GPUs unnecessary.

1. Training Is Off the Table

LLM training is compute-bound. Large matrix multiplications, gradient computation, parameter updates — these need GPU's high compute density. PIM's in-memory units are good for matrix-vector products (GEMV) but are vastly outperformed by GPUs on matrix-matrix products (GEMM).

# Training vs Inference workload characteristics
workload_characteristics = {
    "training": {
        "dominant_op": "GEMM (matrix-matrix multiply)",
        "arithmetic_intensity": "high (100+ FLOP/byte)",
        "bottleneck": "compute",
        "pim_advantage": "none (GEMM is GPU territory)",
    },
    "inference_prefill": {
        "dominant_op": "GEMM (batched)",
        "arithmetic_intensity": "medium-high",
        "bottleneck": "compute (batch-size dependent)",
        "pim_advantage": "limited",
    },
    "inference_decode": {
        "dominant_op": "GEMV (matrix-vector multiply)",
        "arithmetic_intensity": "low (1-2 FLOP/byte)",
        "bottleneck": "memory bandwidth",
        "pim_advantage": "★ significant",
    },
}

PIM's window of advantage is inference Decode only. Training and Prefill are GPU's domain.

2. Immature Programming Model

Leveraging PIM requires explicit programming of data placement and compute mapping. No CUDA-equivalent exists.

GPU:
  Mature software stack (CUDA: 15 years of development)
  Full framework support (PyTorch, TensorFlow, llama.cpp)
  Massive developer ecosystem

PIM:
  Vendor-specific APIs (SK Hynix AiM SDK, Samsung PIM SDK)
  No framework integration
  Small developer community
  Manual memory placement optimization required

Hardware without software is useless. CUDA made GPUs a compute platform. PIM needs its "CUDA moment." It hasn't happened yet.

3. Cost Structure

PIM costs more to manufacture than standard memory. Added die area for compute units, more complex testing, lower yields.

Standard HBM3E: ~$10-18/GB (2026 market estimates)
HBM-PIM:        ~$20-30/GB (estimated, compute unit premium)
GPU (A100):     ~$10,000 (includes 80GB HBM2e)

PIM ROI conditions:
  Inference server power cost savings > PIM premium
  → May work at datacenter scale
  → Not relevant for individual users in the near term

Will PIM Reach RTX 4060 Users?

Honestly, consumer PIM is 3-5 years away.

Currently available PIM:
  - SK Hynix AiM: Datacenter only, not consumer-purchasable
  - Samsung LPDDR5X-PIM: Mobile only, not in PCs

Expected 2027-2028:
  - Possible PIM integration in HBM4-equipped GPUs
  - NVIDIA Rubin architecture adopts HBM4
  - Whether PIM features ship in consumer GPUs: unknown

Current practical solutions for consumers:
  1. Higher-bandwidth GPUs (RTX 5090: GDDR7 at ~1.8 TB/s)
  2. MoE models to reduce active parameters (lower bandwidth demand)
  3. Speculative decoding to improve effective bandwidth utilization

Indirect benefits exist, though. As datacenter PIM adoption grows, cloud API inference costs drop. Energy efficiency improvements translate to lower per-token pricing.

Where PIM Redraws the Line

PIM won't kill GPUs. But it redraws the boundary of GPU work.

Current boundary:
  GPU = training + inference (everything)
  Memory = data storage only

Post-PIM boundary:
  GPU = training + Prefill (compute-bound)
  PIM = Decode (memory-bound)
  Memory = data storage + Decode computation

When Decode shifts to PIM, GPUs can specialize in Prefill and training. The idle-GPU problem during Decode disappears.

This also shifts semiconductor industry dynamics. Part of the inference market that NVIDIA monopolizes today moves to memory makers (Samsung, SK Hynix, Micron). TrendForce reported in March 2026 that Samsung and SK Hynix are exploring "next-gen AI memory that could challenge NVIDIA."

The Memory Wall Is Crumbling — From the Inside

Back to the opening question: "If memory could compute, would we still need GPUs?"

Yes, we need GPUs. Training and Prefill are non-negotiable. But Decode's lead actor may shift from GPU to PIM.

PIM fundamentally solves the memory bandwidth bottleneck for inference Decode
SK Hynix AiM is shipping, Samsung LPDDR5X-PIM is announced, HBM4 integrates logic dies
But: can't handle training, software stack is immature, carries a cost premium
Consumer PIM is 3-5 years out. MoE + Speculative Decoding remain the practical solution for now

The memory wall isn't being broken from the outside (more bandwidth). It's crumbling from the inside (putting computation in). That wave will take a while to reach individual GPUs, but in datacenters, it's already underway.

References

"Challenges and Research Directions for Large Language Model Inference Hardware" (2026) arXiv:2601.05047
"HPIM: Heterogeneous Processing-In-Memory-based Accelerator for LLM Inference" (2025) arXiv:2509.12993
"PAM: Processing Across Memory Hierarchy" (2026) arXiv:2602.11521
"Memory Is All You Need: Compute-in-Memory Architectures for LLM Inference" (2024) arXiv:2406.08413
TrendForce. "Beyond HBM: Samsung, SK hynix Explore Next-Gen AI Memory" (2026-03-10)
Samsung. "LPDDR5X-PIM for AI Computing" (2026-02)

I Couldn't Build a Local LLM PC for $1,300 — Budget Tiers and the VRAM Cliffs Between Them

plasmon — Sat, 04 Apr 2026 08:16:12 +0000

I Couldn't Build a Local LLM PC for $1,300 — Budget Tiers and the VRAM Cliffs Between Them

You want to run LLMs locally. But "which GPU should I buy?" has no decent answer. Gaming benchmarks are everywhere. "How many billion parameters fit in this much VRAM?" — almost nowhere.

I started at $3,500, then cut to $2,000, $1,700, $1,300. Three breaking points appeared.

Scope: New parts only, NVIDIA GPUs. Used cards (RTX 3090), AMD GPUs (RX 7900 XTX), and Apple Silicon are valid alternatives, but each introduces warranty, software compatibility, or availability trade-offs that deserve their own articles. US street pricing as of early 2026.

The Premise: VRAM Decides Everything

Local LLM inference speed is nothing like gaming fps. Whether the model fits entirely in VRAM creates a discontinuous jump in performance.

CUDA core count and clock speed are secondary. If the full model sits in VRAM, inference is fast. If it spills to system RAM, the CPU-bound layers become the bottleneck and speed drops by an order of magnitude.

Measured on an RTX 4060 8GB: a 9B model with all layers on GPU runs at 33 t/s. A 27B model loads only 24 of 58 layers onto the GPU — 3.6 t/s. Same GPU, nearly 10x difference. That's the VRAM cliff.

The "fits or doesn't fit" boundary is the cliff, and cutting budget pushes you straight into it.

$3,500: You Still Have to Choose

The RTX 5090 has 32GB GDDR7. Dream-tier VRAM for local LLM work. But the card alone is $2,000 MSRP, with street prices hitting $2,500–3,000.

[5090 Route: GPU-First]
GPU:  RTX 5090 32GB         ~$2,700  (street price)
CPU:  Ryzen 5 9600X          ~$200   ← budget consumed by GPU
RAM:  DDR5 64GB              ~$120
M/B:  B650                   ~$150
SSD:  1TB NVMe Gen4          ~$60
PSU:  1000W 80+ Gold         ~$140
Case                          ~$80
──────────────────────────
Total                         ~$3,450

At $3,500 the 5090 build barely fits — and everything around the GPU is squeezed thin. You get 32GB VRAM, but the rest of the system is entry-level.

[5080 Route: Balanced]
GPU:  RTX 5080 16GB          ~$1,000
CPU:  Ryzen 7 9800X3D        ~$400
RAM:  DDR5 128GB (32GBx4)    ~$250
M/B:  B650E                  ~$180
SSD:  2TB NVMe Gen4          ~$120
PSU:  850W 80+ Gold          ~$110
Case                          ~$90
──────────────────────────
Total                         ~$2,150  (well under budget)

The 5080 route gives 16GB VRAM with headroom everywhere else. A 27B model at Q4_K_M (~16.7GB file size) won't fit entirely, but the majority of layers land on GPU. Compared to 8GB partial offload (3.6 t/s), you're looking at an estimated 15–25 t/s. Different world.

128GB RAM matters for MoE models. Qwen3.5-35B-A3B consumes over 30GB of system RAM for inactive expert weights in real measurements. 128GB handles large MoE without breaking a sweat.

The decision at $3,500: go all-in on 5090, or take 5080 and balance everything? The 32GB VRAM advantage only matters when you're running 70B+ models seriously. For 27B-class and below, 16GB is enough to fight with. I'd pick the 5080 route.

$2,000: The Budget Where Decisions Get Interesting

Dropping from $3,500 to $2,000 makes the "GPU decides everything" structure even more visible.

GPU:  RTX 5070 Ti 16GB       ~$750
CPU:  Ryzen 7 9700X           ~$280
RAM:  DDR5 96GB (48GBx2)      ~$200
M/B:  B650                    ~$150
SSD:  1TB NVMe Gen4           ~$60
PSU:  750W 80+ Gold           ~$90
Case                           ~$60
──────────────────────────
Total                          ~$1,590

The 5070 Ti holds 16GB VRAM while keeping the CPU and RAM at respectable levels. The remaining $400 can go to a better CPU cooler, SSD expansion, or savings.

There's a fork here: "Should I stretch to the RTX 5080 at ~$1,000?" The 5080 and 5070 Ti share the same 16GB VRAM. The difference is CUDA core count and memory bandwidth. For LLM inference where VRAM is the same, the measured gap is roughly 10–20% from bandwidth.

I'd pick the 5070 Ti and put the $250 savings into RAM. MoE model inactive experts live in system RAM, so there are real scenarios where RAM capacity affects the experience more than a 10% GPU speed bump.

$2,000 is the "comfort line" for a local LLM build. 16GB VRAM + 96GB RAM + a solid CPU. Below this, you start sacrificing something important.

$1,700: The Cliff's Edge

Cutting just $300 from $2,000 changes the build's stability dramatically.

Keeping the 5070 Ti

GPU:  RTX 5070 Ti 16GB       ~$750
CPU:  Ryzen 5 7600            ~$160   ← dropped two generations
RAM:  DDR5 64GB               ~$120   ← down from 96GB
M/B:  B650                    ~$150
SSD:  1TB NVMe                ~$60
PSU:  750W                    ~$90
Case                           ~$60
──────────────────────────
Total                          ~$1,390

Everything except the GPU is compromised. The CPU drops two generations. RAM shrinks from 96GB to 64GB. Running Qwen3.5-35B-A3B MoE required 30GB+ of RAM in measurements, so at 64GB, coexistence with other applications gets tight.

Switching to the 5070 (12GB)

GPU:  RTX 5070 12GB           ~$550
CPU:  Ryzen 5 9600X            ~$200
RAM:  DDR5 64GB                ~$120
M/B + SSD + PSU + Case        ~$340
──────────────────────────
Total                          ~$1,210

The balance is better. But 12GB VRAM occupies an awkward position.

12GB VRAM: Improvement, Not Transformation

Mapping "what fits" by VRAM tier makes 12GB's position clear.

VRAM    Models (Q4_K_M)              Est. Speed      New Capability
──────────────────────────────────────────────────────────────
8GB     7B–9B all layers              25–35 t/s      ← Entry line
12GB    7B–9B comfortably, 14B full   14B: 15–25 t/s ← 14B added
16GB    27B mostly on GPU             27B: 10–20 t/s ← 27B in range ★
24GB+   27B full, 70B partial         27B: 20–30 t/s ← 70B realistic

Going from 8GB to 12GB unlocks the 14B class (Qwen3 14B, Phi-4 14B, etc.) with all layers on GPU. 14B is a genuine quality step up from 9B, especially in code generation and reasoning tasks. The improvement is real.

But the 12GB-to-16GB jump is bigger. The quality gap between 9B and 27B is significantly larger than between 9B and 14B. Code generation accuracy, long-context coherence, complex instruction following — there are many tasks where 27B is the first size that feels "usable."

Same 4GB increment, different returns:

8GB → 12GB: Incremental improvement (14B becomes available. Noticeable, not transformative)
12GB → 16GB: Step-function leap (27B class comes into range. The experience changes)

12GB isn't worthless. But if you're choosing where to invest 4GB under a budget constraint, the return on 12GB→16GB is overwhelmingly larger.

$1,300: The Budget Where I Couldn't Build

Trying to hold 16GB VRAM at $1,300:

RTX 5070 Ti 16GB ($750) → $550 left for the entire PC
  CPU: Bottom-tier
  RAM: 32GB (not even an upgrade over many laptops)
  SSD: 500GB (fills up with 2-3 models)

Everything except the GPU drops below a current mid-range laptop. That's not "building a desktop" — that's a GPU in a cardboard box.

Dropping to 12GB VRAM does allow a balanced $1,300 build. As a platform for 14B models, it's not bad. But the leap from an 8GB environment is limited, and as the payoff for building a desktop PC, it feels thin.

Spending $1,300 on a 12GB VRAM desktop versus spending $1,300 on a laptop with an RTX 4060 8GB — which gives a richer local LLM experience? Honestly, it's a close call.

Three Cliffs Revealed by Budget

Cliff	Range	What Happens
VRAM cliff	12GB → 16GB	27B class comes into range. Same 4GB, nonlinear returns
Build cliff	$2,000 → $1,700	Everything except GPU collapses. $300 changes the entire picture
Viability cliff	$1,700 → $1,300	Holding 16GB becomes impossible. Even 12GB has questionable ROI

Bottom line: $1,700 is the floor for building a local LLM PC. At this price point you can still get an RTX 5070 Ti 16GB with a reasonable CPU and RAM. Below that, you can technically build something, but the value proposition breaks down.

If you can hit $1,700, stretching to $2,000 is worth it. RAM goes from 64GB to 96GB, and MoE model operation becomes practical.

References

Running Qwen2.5-32B on RTX 4060 8GB — Beating M4 at 10.8 t/s with llama.cpp — The limits and possibilities of 8GB VRAM
MoE Beat Dense 27B by 2.4x on 8GB VRAM — The 35B-A3B Benchmark Nobody Expected — Measured RAM consumption of MoE models
Parameter Count Is the Worst Way to Pick a Model on 8GB VRAM — Measured data on the VRAM cliff