I predicted that an 8-bit KV cache would take qwen3-coder:30b from 20 to 34 tokens a second at a 100K-token prompt, and a 4-bit cache to 52. On my Surface Laptop Ultra with RTX Spark, both made it slower: 13.8 tokens a second with q8_0 and 13.1 with q4_0, against 19.9 with the default f16 cache.
The memory savings were exactly as advertised. The cache shrank from 9,600 MiB to 5,100 (q8_0) and 2,700 (q4_0), and that room let the model load a 262,144-token context on the GPU. Prompt processing and short replies barely changed, and the replies kept the same facts in different words.
This is a follow-up to my first RTX Spark post, which ended with this experiment as the next step.
The prediction
The first post showed decode on this chip tracking memory bandwidth: every token reads the active weights plus the whole attention cache, at about 235 GB/s. A smaller cache should mean a faster reply.
| Cache type | Weights per token | Cache per token at 100K | Predicted decode |
|---|---|---|---|
| f16 (default) | ~2.0 GB | ~9.9 GB | 20 tok/s |
| q8_0 | ~2.0 GB | ~5.0 GB | 34 tok/s |
| q4_0 | ~2.0 GB | ~2.5 GB | 52 tok/s |
Two things could break it. Unpacking a quantized cache adds work to every token, so q4_0 might undershoot. And prompt processing might change too, which is why I report it separately.
Setup, and checking it took effect
Ollama's cache type is two environment variables: OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=f16, q8_0 or q4_0. A quantized cache needs flash attention, and Ollama gives K and V the same type; a split like q8 keys with q4 values needs llama.cpp directly.
Setting the variable is a request, not a guarantee: on an unsupported model the cache can fall back to f16 without an error. So the script checks what llama.cpp actually started with. On this machine, all three took effect:
llama-server.exe ... -c 102400 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on
llama_kv_cache: size = 5100.00 MiB (102400 cells, 48 layers, 1/1 seqs), K (q8_0): 2550.00 MiB, V (q8_0): 2550.00 MiB
To switch types in one sitting, kv-cache-bench.ps1 stops the tray app and runs ollama serve with the variables set for that process only, then restarts the tray app as it was. Same day, same driver (617.42), same Ollama (0.40.11), on AC power with the Balanced plan.
| Per cache type | |
|---|---|
| Code benchmark | The first post's three prompts (LRU cache, rate limiter, Dijkstra), 512-token replies |
| Long prompt | The first post's 100,560-token prompt, 384-token reply, three repeats |
| Prompt cache | The model unloads before every long repeat, so all three repeats read the identical prompt in full |
| Quality check | Each long reply saved and compared word by word with the first f16 reply |
One lesson from Windows PowerShell 5.1: its Invoke-RestMethod never reached the freshly started server, and the script stalled without an error. PowerShell 7 worked at once.
Results at 100K tokens
f16 landed on its prediction; both quantized caches decoded about a third slower than f16, where I expected 1.7 and 2.6 times faster.
The runs were steady: f16 19.2–20.6, q8_0 13.6–14.1 and q4_0 13.0–13.4 tokens a second. Everything else either stayed flat or improved:
| Cache type | KV cache | Peak GPU memory | Prefill, 100K prompt | Short code prompts |
|---|---|---|---|---|
| f16 | 9,600 MiB | 30,518 MiB | 760 tok/s | 81.7 tok/s |
| q8_0 | 5,100 MiB | 26,322 MiB | 780 tok/s | 87.9 tok/s |
| q4_0 | 2,700 MiB | 23,741 MiB | 803 tok/s | 88.3 tok/s |
The cache sizes are exact: q8_0 stores 8.5 bits per value and q4_0 4.5, so 53% and 28% of f16. Prefill took 123–142 seconds in every run. Short code replies ran at 81.7 tokens a second with f16, below the first post's 90.9; f16 ran first in the session, so treat that gap as run-to-run variation rather than a cache effect.
Longer contexts: the full 262,144 tokens now fit
The real win is capacity: with q4_0, qwen3-coder loads its full 262,144-token context entirely on the GPU, the context the Lenovo in Microsoft's post used. My first post stopped at 102,400 tokens because the f16 cache would not fit.
| Run | Context | Prompt | KV cache | Peak GPU memory | Prefill | First token after | Decode |
|---|---|---|---|---|---|---|---|
| q8_0 | 163,840 | 148,051 tokens | 8,160 MiB | 28,761 MiB | 511 tok/s | 4 min 50 s | 9.9 tok/s |
| q4_0 | 262,144 | 242,898 tokens | 6,912 MiB | 28,379 MiB | 349 tok/s | 11 min 36 s | 6.5 tok/s |
At this size f16 has no chance: its cache alone would be about 24 GB next to 17 GB of weights, and q8_0 would need about 13 GB. Both runs stayed 100% on the GPU and peaked at 74–75 °C.
It fits, but it's slow. Reading a fresh 243K-token prompt took almost 12 minutes, and replies came at 6.5 tokens a second. Both replies were still correct and on topic. In an agent session the prompt cache reads most of each turn's history only once, so the 12 minutes is a one-time cost per conversation; the 6.5 tokens a second is not.
One more lesson in measuring: my first attempt at these two runs came out at 197,535 and 332,024 tokens, and Ollama silently kept only half the context. Token density varies across the source text, so the longer prompts are now built from whole copies of the 100K prompt.
Quality check: same facts, different words
All nine long replies were accurate, and each cache type produced the same reply on all three of its runs; only the wording changed between cache types.
| Cache type | Identical across its 3 runs | Word overlap with the f16 reply | Files described in 384 tokens |
|---|---|---|---|
| f16 | Yes | 100% | 7 |
| q8_0 | Yes | 50% | 6 |
| q4_0 | Yes | 62% | 7 |
Word overlap is twice the longest common run of words divided by both replies' lengths. Every reply walked through the same files in the same order, starting with Certificate.format.ps1xml, and named largely the same .NET types for each. q8_0 wrote longer descriptions, so it reached one file fewer before the 384-token limit.
At temperature 0 with a fixed seed, f16 was fully deterministic. The quantized caches change the numbers the model attends to just enough to pick different words early, and the rest of the reply follows from that. One question on 100K tokens isn't a full quality evaluation, but it found no wrong facts.
Why the prediction missed
The bandwidth model only holds while reading memory is the slowest part of a token. With a quantized cache it no longer is:
| Cache type | Bytes read per token at 100K | Measured decode | Implied bandwidth |
|---|---|---|---|
| f16 | ~11.9 GB | 19.9 tok/s | ~235 GB/s |
| q8_0 | ~7.3 GB | 13.8 tok/s | ~100 GB/s |
| q4_0 | ~4.8 GB | 13.1 tok/s | ~63 GB/s |
If memory were still the limit, all three rows would imply the same bandwidth. Instead q8_0 and q4_0 read less and still take longer per token, so the time goes into computing, not reading. Each step has to unpack about 100,000 cached keys and values per layer from 8 or 4 bits before flash attention can use them, and that costs more than the bytes it saves.
That's my reading of the numbers, not a profile. The f16 path in llama.cpp's flash attention is the most optimised one, and the quantized paths may simply be less tuned for this new Blackwell GPU yet. Prefill, which processes 512 tokens at a time, hides the unpacking cost and ran 3–6% faster with a quantized cache.
In hindsight the warning was in my own setup notes. The OpenClaw DC guide I took the configuration from sums it up as memory saved, not time, and cites q4_0 decoding about 37% slower at 110K tokens. I measured 34% slower at 100K on a different GPU, so my bandwidth prediction was the outlier, not the hardware.
What I'd use
- Up to about 100K tokens: keep f16. It's the fastest cache on this chip for long prompts, and it fits.
- Beyond 102K tokens: q4_0. It's the only way to fit qwen3-coder's full 262,144-token context on a 32 GB GPU share, and its replies stayed accurate.
- When memory, not context, is tight: q8_0 or q4_0. They free 4.4–6.7 GiB at 100K tokens for other apps or a second model, at the cost of about a third of the long-prompt decode speed.
-
Always check the log. Look for
--cache-type-kon the llama-server command line and thellama_kv_cachesize line; the variable alone proves nothing.
The prediction failed in an informative way: the bytes-per-token model is right for f16 and wrong for quantized caches, because they turn a memory-bound step into a compute-bound one. Next I want to try llama.cpp directly, where keys and values can have different types, and see whether a newer build speeds up the quantized flash-attention path on Blackwell.
Sources
- Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama, my first post, 26 Oct 2026
- Qwen3.8-27B on a Lenovo laptop: run it on RTX Spark, watch it in Azure Monitor, Microsoft Tech Community, 26 Oct 2026
- llama.cpp KV Cache Quantization: q8_0 vs q4_0 vs f16 (2026), OpenClaw DC, 8 Aug 2026, for the flash-attention requirement, the silent fallback, and its decode measurement
- My runs of
kv-cache-bench.ps1on 27 Oct 2026: one session for the three cache types and one for the long-context runs, each with server logs, GPU samples, and the reply texts:
# KV-cache quantization benchmark for Ollama: f16 vs q8_0 vs q4_0 on the same model, prompts and day.
# Runs Ollama as `ollama serve` with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE set per run (process
# environment only; no user variables change), then restarts the tray app as it was.
# Per cache type: three code prompts (512-token replies) and $Repeats long prompts (~100,589 tokens, 384-token
# replies), the model unloaded before each long repeat so the prompt cache never helps and every prompt is identical.
# Extras: q8_0 at 163,840 context (~154K prompt) and q4_0 at 262,144 (~247K prompt), one repeat each.
# Writes everything to C:\Dev\kv-bench-<COMPUTER>-<stamp>\ : results.json, server logs, GPU samples, reply texts.
param(
[string]$Model = 'qwen3-coder:30b',
[string[]]$CacheTypes = @('f16', 'q8_0', 'q4_0'),
[int]$Repeats = 3,
[switch]$SkipExtras,
[switch]$ExtrasOnly,
[double]$CharsPerToken = 6.05 # measured by speed-bench.ps1 on this corpus; fixed so every run builds the same prompt
)
$ErrorActionPreference = 'Stop'
$api = 'http://127.0.0.1:11434'
$ollamaDir = "$env:LOCALAPPDATA\Programs\Ollama"
$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$outDir = "C:\Dev\kv-bench-$env:COMPUTERNAME-$stamp"
New-Item -ItemType Directory -Force $outDir | Out-Null
$progress = "$outDir\progress.txt"
function Say($msg) { $line = "$(Get-Date -Format 'HH:mm:ss') $msg"; Write-Host $line; Add-Content $progress $line }
$codePrompts = @(
'Write a thread-safe LRU cache in Python with unit tests.',
'Write a token-bucket rate limiter in TypeScript.',
"Implement Dijkstra's shortest-path algorithm in Go."
)
$question = "`n`nQuestion: Describe what the files above define, as a numbered list covering every file."
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''
function New-LongPrompt($targetTokens) {
$chars = [int]($targetTokens * $CharsPerToken * 0.97)
$text = $corpus
while ($text.Length -lt $chars) { $text += $corpus }
"Session benchmark-1`n" + $text.Substring(0, $chars) + $question
}
# Longer prompts are whole and partial copies of the 100K prompt's text, whose token count is known (100,540 tokens
# without the first line and the question): token density varies too much across the corpus to scale by characters
function New-RepeatedPrompt($targetTokens) {
$blockChars = [int](97880 * $CharsPerToken * 0.97)
$block = $corpus.Substring(0, $blockChars)
$copies = $targetTokens / 100540
$text = ''
for ($k = 1; $k -le [math]::Floor($copies); $k++) { $text += "`n### COPY $k`n" + $block }
$rest = [int](($copies - [math]::Floor($copies)) * $blockChars)
if ($rest -gt 0) { $text += "`n### COPY PART`n" + $block.Substring(0, $rest) }
"Session benchmark-1`n" + $text + $question
}
function Post-Ollama($path, $body) {
Invoke-RestMethod -Method Post -Uri ($api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 3600
}
function Stop-Ollama {
Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.Name -in @('ollama app', 'ollama', 'llama-server') } | Stop-Process -Force
Start-Sleep 3
}
function Start-OllamaServe($cache, $log) {
$env:OLLAMA_FLASH_ATTENTION = '1'
$env:OLLAMA_KV_CACHE_TYPE = $cache
# cmd does the redirection: PowerShell's own -RedirectStandardError kept this script from continuing
Start-Process cmd.exe -ArgumentList '/c', "`"`"$ollamaDir\ollama.exe`" serve > `"$log`" 2>&1`"" -WindowStyle Hidden
for ($i = 0; $i -lt 60; $i++) {
try { Invoke-RestMethod ($api + '/api/version') -TimeoutSec 2 | Out-Null; Say ' server is up'; return } catch { Start-Sleep 1 }
}
throw "ollama serve did not answer within 60 s ($cache)"
}
function Unload-All {
foreach ($m in @((Invoke-RestMethod ($api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
for ($i = 0; $i -lt 30 -and @((Invoke-RestMethod ($api + '/api/ps')).models).Count; $i++) { Start-Sleep 1 }
Start-Sleep 2
}
function Get-GpuMemMiB { [int](& nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits).Trim() }
function Measure-Reply($r, $wall) {
[ordered]@{
prompt_tokens = $r.prompt_eval_count
prefill_tok_s = if ($r.prompt_eval_duration) { [math]::Round($r.prompt_eval_count / ($r.prompt_eval_duration / 1e9), 1) } else { $null }
prefill_s = [math]::Round($r.prompt_eval_duration / 1e9, 1)
output_tokens = $r.eval_count
decode_tok_s = if ($r.eval_duration) { [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2) } else { $null }
load_s = [math]::Round($r.load_duration / 1e9, 1)
wall_s = [math]::Round($wall, 1)
done_reason = $r.done_reason
}
}
# Word-level similarity: 2 * longest common subsequence / (words in a + words in b); 1.0 = identical
function Get-Similarity($a, $b) {
$x = @($a -split '\s+' | Where-Object { $_ }); $y = @($b -split '\s+' | Where-Object { $_ })
if (-not $x.Count -or -not $y.Count) { return 0 }
$prev = New-Object int[] ($y.Count + 1)
foreach ($wx in $x) {
$cur = New-Object int[] ($y.Count + 1)
for ($j = 1; $j -le $y.Count; $j++) {
$cur[$j] = if ($wx -ceq $y[$j - 1]) { $prev[$j - 1] + 1 } else { [math]::Max($prev[$j], $cur[$j - 1]) }
}
$prev = $cur
}
[math]::Round(2 * $prev[$y.Count] / ($x.Count + $y.Count), 3)
}
function Invoke-Config($cache, $numCtx, $longTarget, $repeats, [bool]$withCode) {
$tag = "$cache-ctx$numCtx"
$log = "$outDir\server-$tag.log"
Say "=== ${tag}: starting ollama serve ==="
Stop-Ollama
Start-OllamaServe $cache $log
$gpuCsv = "$outDir\gpu-$tag.csv"
$sampler = Start-Process nvidia-smi -ArgumentList '-i 0 --query-gpu=timestamp,power.draw,temperature.gpu,clocks.gr,memory.used --format=csv,noheader,nounits -l 1 -f', "`"$gpuCsv`"" -WindowStyle Hidden -PassThru
Say ' loading the model'
$cfg = [ordered]@{ cache_requested = $cache; flash_attention_requested = '1'; num_ctx = $numCtx }
try {
$opts = @{ num_ctx = $numCtx; num_predict = 8; temperature = 0; seed = 42 }
$r = Post-Ollama '/api/generate' @{ model = $Model; prompt = 'Reply with exactly: warm'; stream = $false; keep_alive = '30m'; options = $opts }
$loaded = @((Invoke-RestMethod ($api + '/api/ps')).models | Where-Object name -eq $Model)[0]
$cfg.load_s = [math]::Round($r.load_duration / 1e9, 1)
$cfg.loaded_context = $loaded.context_length
$cfg.size_gib = [math]::Round($loaded.size / 1GB, 2)
$cfg.gpu_percent = if ($loaded.size) { [math]::Round(100 * $loaded.size_vram / $loaded.size) } else { $null }
$cfg.gpu_mem_after_load_mib = Get-GpuMemMiB
# What llama.cpp actually applied, from the server log (the variable is a request, not a guarantee)
$applied = @(Select-String -Path $log -Pattern 'cache-type|type_k|type_v|llama_kv_cache|flash_attn|Flash Attention' -ErrorAction SilentlyContinue | ForEach-Object { $_.Line.Trim() })
$cfg.applied_log_lines = $applied
$kvLine = $applied | Where-Object { $_ -match 'K \((\w+)\).*V \((\w+)\)' } | Select-Object -Last 1
if ($kvLine -match 'K \((\w+)\).*V \((\w+)\)') { $cfg.applied_k = $Matches[1]; $cfg.applied_v = $Matches[2] }
$sizeLine = $applied | Where-Object { $_ -match 'size\s*=\s*([\d.]+)\s*MiB' } | Select-Object -Last 1
if ($sizeLine -match 'size\s*=\s*([\d.]+)\s*MiB') { $cfg.kv_cache_mib = [double]$Matches[1] }
Say (" loaded: {0} GiB, {1}% GPU, context {2}, K={3} V={4}, KV cache {5} MiB, GPU memory {6} MiB" -f $cfg.size_gib, $cfg.gpu_percent, $cfg.loaded_context, $cfg.applied_k, $cfg.applied_v, $cfg.kv_cache_mib, $cfg.gpu_mem_after_load_mib)
if ($withCode) {
$cfg.code = foreach ($cp in $codePrompts) {
$opts = @{ num_ctx = $numCtx; num_predict = 512; temperature = 0; seed = 42 }
$sw = [Diagnostics.Stopwatch]::StartNew()
$r = Post-Ollama '/api/generate' @{ model = $Model; prompt = $cp; stream = $false; keep_alive = '30m'; options = $opts }
$sw.Stop()
$m = Measure-Reply $r $sw.Elapsed.TotalSeconds; $m.prompt = $cp
Say (" code: {0} tok/s" -f $m.decode_tok_s)
[pscustomobject]$m
}
$cfg.code_mean_decode_tok_s = [math]::Round(($cfg.code | Measure-Object decode_tok_s -Average).Average, 2)
}
$prompt = if ($longTarget -le 97880) { New-LongPrompt $longTarget } else { New-RepeatedPrompt $longTarget }
$cfg.long = for ($i = 1; $i -le $repeats; $i++) {
Unload-All # a fresh load empties the prompt cache, so the identical prompt is read in full every time
$opts = @{ num_ctx = $numCtx; num_predict = 384; temperature = 0; seed = 42 }
$sw = [Diagnostics.Stopwatch]::StartNew()
$r = Post-Ollama '/api/generate' @{ model = $Model; prompt = $prompt; stream = $false; keep_alive = '30m'; options = $opts }
$sw.Stop()
$m = Measure-Reply $r $sw.Elapsed.TotalSeconds
$m.repeat = $i
$m.truncated = $r.prompt_eval_count -lt 0.8 * $longTarget
$m.gpu_mem_mib = Get-GpuMemMiB
$replyFile = "reply-$tag-r$i.txt"
[IO.File]::WriteAllText("$outDir\$replyFile", $r.response, (New-Object Text.UTF8Encoding $false))
$m.reply_file = $replyFile
Say (" long {0}: {1} prompt tokens, prefill {2} tok/s ({3} s), decode {4} tok/s, truncated={5}" -f $i, $m.prompt_tokens, $m.prefill_tok_s, $m.prefill_s, $m.decode_tok_s, $m.truncated)
[pscustomobject]$m
}
$cfg.long_mean_decode_tok_s = [math]::Round(($cfg.long | Measure-Object decode_tok_s -Average).Average, 2)
$cfg.long_mean_prefill_tok_s = [math]::Round(($cfg.long | Measure-Object prefill_tok_s -Average).Average, 1)
}
catch {
$cfg.error = $_.Exception.Message
Say " ERROR: $($cfg.error)"
}
finally {
Stop-Process -Id $sampler.Id -ErrorAction SilentlyContinue
$samples = @(Get-Content $gpuCsv -ErrorAction SilentlyContinue | Where-Object { $_ -match '^\d{4}/' } | ForEach-Object { $f = $_.Split(','); [pscustomobject]@{ w = [double]$f[1]; c = [int]$f[2]; m = [int]$f[3]; mem = [int]$f[4] } })
if ($samples.Count) {
$cfg.gpu_peak_mem_mib = ($samples | Measure-Object mem -Maximum).Maximum
$cfg.gpu_peak_power_w = ($samples | Measure-Object w -Maximum).Maximum
$cfg.gpu_peak_temp_c = ($samples | Measure-Object c -Maximum).Maximum
}
}
[pscustomobject]$cfg
}
$results = @()
try {
if (-not $ExtrasOnly) { foreach ($ct in $CacheTypes) { $results += Invoke-Config $ct 102400 97880 $Repeats $true } }
if (-not $SkipExtras) {
$results += Invoke-Config 'q8_0' 163840 150000 1 $false
$results += Invoke-Config 'q4_0' 262144 245000 1 $false
}
}
finally {
Stop-Ollama
Remove-Item Env:OLLAMA_KV_CACHE_TYPE, Env:OLLAMA_FLASH_ATTENTION -ErrorAction SilentlyContinue
Start-Process "$ollamaDir\ollama app.exe" # back to the tray app with the user's own settings
Say 'Restarted the Ollama tray app'
}
# Quality check: every long reply against the first f16 reply (f16's own repeats show the noise floor)
$ref = $results | Where-Object { $_.cache_requested -eq 'f16' -and $_.long } | Select-Object -First 1
$quality = @()
if ($ref) {
$refText = [IO.File]::ReadAllText("$outDir\$($ref.long[0].reply_file)")
foreach ($cfg in $results) {
foreach ($l in @($cfg.long)) {
if (-not $l) { continue }
$t = [IO.File]::ReadAllText("$outDir\$($l.reply_file)")
$quality += [pscustomobject]@{ reply = $l.reply_file; identical_to_f16_r1 = ($t -ceq $refText); similarity_to_f16_r1 = Get-Similarity $refText $t }
}
}
}
$battery = Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue | Select-Object -First 1
$out = [ordered]@{
computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s'); model = $Model
ollama_version = (& "$ollamaDir\ollama.exe" --version 2>&1 | Out-String).Trim()
gpu = (& nvidia-smi -i 0 --query-gpu=name,driver_version,memory.total --format=csv,noheader).Trim()
power_scheme = (powercfg /getactivescheme) -replace '^.*\((.*)\)\s*$', '$1'
on_ac_power = if ($battery) { $battery.BatteryStatus -eq 2 } else { $null }
chars_per_token = $CharsPerToken
configs = $results
quality = $quality
}
[IO.File]::WriteAllText("$outDir\results.json", ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$results | Select-Object cache_requested, applied_k, num_ctx, kv_cache_mib, gpu_peak_mem_mib, code_mean_decode_tok_s, long_mean_prefill_tok_s, long_mean_decode_tok_s, error | Format-Table -AutoSize | Out-String | Write-Host
$quality | Format-Table -AutoSize | Out-String | Write-Host
Say "DONE: $outDir\results.json"
Set-Content "$outDir\DONE" 'done'
Top comments (0)