Every 1,000 tokens of context costs 0.39 ms: decode speed from 1K to 100K on RTX Spark

On my Surface Laptop Ultra, qwen3-coder:30b's time per token grows in a straight line with the context it has to read: 11.8 ms plus 0.39 ms for every 1,000 tokens of f16 cache. That's within 7% of what memory bandwidth predicts, so the bytes-per-token model from my first RTX Spark post holds for the cache. Decode falls from 80 tokens a second at 1K tokens to 20 at 97K.

The 8-bit cache follows a straight line too, but a steeper one: 0.62 ms per 1,000 tokens, 1.6 times f16, for half the bytes. q8_0 is never faster than f16 at any length I tested, and from 8K tokens on it is clearly slower. This turns my second post's single 100K measurement into a curve.

Method and the prediction

One question after prompts of seven sizes, three runs each, with the context window fixed at 102,400 tokens so only the amount of filled cache changes.

Model	qwen3-coder:30b, Q4_K_M, Ollama 0.40.11, 100% on the GPU
Prompt sizes	935, 3,651, 7,159, 13,974, 30,009, 57,365 and 96,926 tokens
Reply	256 tokens, temperature 0, thinking off
Prompt cache	A random first line on every prompt, so each one is read in full
Cache types	f16, then q8_0, each in its own ollama serve with flash attention on; the server log confirmed 9,600 and 5,100 MiB caches
Machine	Surface Laptop Ultra, RTX Spark N1X, 64 GB; AC power, Balanced plan, 27 Oct 2026

The prediction is the first post's bandwidth model, now per token of context:

time per token ≈ (2.0 GB + context tokens × 96 KiB × cache factor) ÷ 235 GB/s

The cache factor is 1 for f16 and 0.53 for q8_0. That gives a straight line: 8.5 ms of weights plus 0.42 ms per 1,000 cached tokens for f16, and 0.22 ms for q8_0.

The curve

Plotted as milliseconds per token, both cache types are straight lines; the difference between them is the slope.

A least-squares line through the seven means fits both series almost exactly (R² 0.999):

	Fixed cost per token	Added per 1,000 cached tokens	Implied bandwidth on the cache
Predicted, f16	8.5 ms	0.42 ms	235 GB/s
Measured, f16	11.8 ms	0.39 ms	~250 GB/s
Predicted, q8_0	8.5 ms	0.22 ms	235 GB/s
Measured, q8_0	12.4 ms	0.62 ms	~84 GB/s

The runs were consistent: at 57K tokens the three f16 runs spanned 29.5–30.0 tokens a second and q8_0 21.5–21.8. None of the 42 prompts was truncated.

Where the measurement leaves the prediction

The measured lines peel away from the prediction in two different places, for two different reasons.

f16 peels away at short context, not long. The slope matches: each cached token costs what reading 96 KiB at about 250 GB/s takes. The gap is the starting point, 11.8 ms instead of 8.5. Those extra 3.3 ms per token aren't memory reads; my best guess is the mixture-of-experts routing, kernel launches and sampling that happen once per token whatever the context. At 1K tokens that overhead is a quarter of every token, so decode lands at 80 tokens a second instead of the predicted 112; at 97K it is 7%, and the prediction is within half a token a second.

q8_0 peels away more with every token. Its fixed cost matches f16's (12.4 ms), but each 1,000 cached tokens adds 0.62 ms where 0.22 was predicted, 2.8 times the bandwidth estimate and 1.6 times f16. So the extra cost of the 8-bit cache isn't a one-off; it scales with the cache, which points at unpacking every cached value before attention can use it. That's my reading of the slope, not a profile.

The two lines start together: at 1K and 4K tokens the difference between f16 and q8_0 is inside the run-to-run spread. By 14K tokens q8_0 is 25% slower, and from 30K to 97K it is 26–32% slower.

One check from the notes: the 1K decode of 80 tokens a second sits below the first post's 90.9. It matches the 81.7 that f16 managed on today's 512-token code prompts, so the 256-token reply isn't too short to average out startup; this session simply ran about 10% below the first one.

Prompt processing, first token and clocks

Prompt processing slows as the prompt grows, from about 2,400 tokens a second up to 8K tokens to 795 at 97K, and the cache type barely matters there.

Prompt	Prefill, f16	First token, f16	Prefill, q8_0	First token, q8_0	Clock reading the prompt, f16	Clock replying, f16
935 tokens	2,398 tok/s	0.4 s	2,311 tok/s	0.4 s	too short to sample	2,246 MHz
3,651	2,255	1.6 s	2,169	1.7 s	too short to sample	2,307 MHz
7,159	2,377	3.0 s	2,144	3.4 s	1,596 MHz	2,153 MHz
13,974	2,072	6.8 s	1,809	7.8 s	1,861 MHz	2,159 MHz
30,009	1,562	19.2 s	1,466	20.5 s	1,846 MHz	2,229 MHz
57,365	1,153	49.8 s	1,109	51.8 s	1,775 MHz	2,332 MHz
96,926	795	122 s	779	125 s	1,702 MHz	2,266 MHz

The power-limit pattern from the first post holds at every size I could sample: the GPU reads a prompt at 1.6–1.9 GHz and replies at 2.15–2.33 GHz. Reading is compute-bound and hits the power limit; replying is memory-bound and runs at full clock. During the f16 run the GPU peaked at 73 °C.

q8_0 read prompts 2–13% slower than f16 here. In the previous post it was slightly faster at 100K; both differences are small next to the 3× drop in prefill speed between 8K and 97K tokens.

What it means

• Estimate decode speed from the context length. For qwen3-coder:30b on RTX Spark with f16: time per token ≈ 11.8 ms + 0.39 ms per 1,000 tokens in the conversation. At 50K tokens that is 31 ms, or about 32 tokens a second.
• Long conversations cost linearly, not suddenly. There is no cliff; each 10K tokens of history adds about 4 ms to every reply token.
• Keep f16 unless you need the memory. q8_0 saves 4.4 GiB at 102,400 tokens but adds 0.62 ms per 1,000 tokens instead of 0.39, and never wins at any length.
• Short prompts are limited by fixed work, not memory. Below 8K tokens, a faster cache or more bandwidth would change little; the 11.8 ms per token is the floor.

The bandwidth model from the first post now has a clear scope: it predicts the slope of the f16 line to within 7%, it underestimates the fixed cost by 3.3 ms, and it does not apply to quantized caches.

Script notes

The script is context-curve.ps1, run once per cache type by a small wrapper that starts ollama serve with that type and checks the llama_kv_cache line in its log. Two fixes went in along the way:

• Calibration. Characters per token are measured over the same text the largest prompt uses. This corpus runs at about 6.2 characters per token at the start and 5.7 over the first 570K characters; calibrating on a shorter slice would have pushed the 100K prompt past the context, and Ollama would have cut it in half without an error.
• GPU sample parsing. Stopping nvidia-smi can leave a half-written last line in its CSV, which crashed the q8_0 run's final step after all measurements were taken. Those numbers come from the run's console output; the parser now skips any line it can't read.

Sources

• Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama, the first post, 26 Oct 2026
• Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower, the second post, 27 Oct 2026
• My runs of context-curve.ps1 on 27 Oct 2026: 42 prompts, with per-run JSON, CSV and nvidia-smi samples for f16, and the console output for q8_0:

# Decode speed vs context length on Ollama.
# Sends one question after prompts of increasing size (default 1K, 8K, 32K, 64K, 100K tokens),
# each with a random first line so the prefix cache cannot help, and records Ollama's own timings.
# Prints measured decode next to the bandwidth prediction (bandwidth / bytes read per token).
# Writes context-curve-<COMPUTER>-<kv>-<stamp>.json and .csv in the current directory.
param(
    [string]$Model = 'qwen3-coder:30b',
    [int[]]$Sizes = @(1000, 8000, 32000, 64000, 100000),
    [int]$Repeats = 2,
    [int]$NumCtx = 102400,          # kept constant so only the prompt length varies
    [int]$Reply = 256,
    [string]$Api = 'http://127.0.0.1:11434',
    # Prediction inputs for qwen3-coder:30b at Q4_K_M; change for another model
    [double]$BandwidthGBs = 235,
    [double]$WeightsGB = 2.0,       # active weights read per token
    [double]$CacheKiBPerToken = 96  # 48 layers x 4 KV heads x 128 dims x 2 (K,V) x 2 bytes, f16
)
$ErrorActionPreference = 'Stop'

$kvType = if ($env:OLLAMA_KV_CACHE_TYPE) { $env:OLLAMA_KV_CACHE_TYPE } else { 'f16' }
$cacheFactor = switch ($kvType) { 'q8_0' { 0.53 } 'q4_0' { 0.28 } default { 1.0 } }   # bytes relative to f16 (q8_0 = 8.5 bits, q4_0 = 4.5 bits per value)

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($Api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 1800
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($Api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    Start-Sleep 3
}
function Invoke-Timed($body) {
    $sw = [Diagnostics.Stopwatch]::StartNew(); $r = Post-Ollama '/api/generate' $body; $sw.Stop()
    return $r, $sw.Elapsed.TotalSeconds
}
function Predict-Decode($tokens) {
    $bytesGB = $WeightsGB + ($tokens * $CacheKiBPerToken * 1024 * $cacheFactor / 1e9)
    [math]::Round($BandwidthGBs / $bytesGB, 1)
}

$caps = @((Post-Ollama '/api/show' @{ model = $Model }).capabilities)
function New-Body($prompt, $n) {
    $b = @{ model = $Model; prompt = $prompt; stream = $false; keep_alive = '15m'; options = @{ num_ctx = $NumCtx; num_predict = $n; temperature = 0; seed = 42 } }
    if ($caps -contains 'thinking') { $b.think = $false }
    $b
}

# Prompt text: PowerShell's own format/type files and module sources, present on every Windows machine
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
    Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''
while ($corpus.Length -lt $NumCtx * 5) { $corpus += $corpus }

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$base = Join-Path (Get-Location) "context-curve-$env:COMPUTERNAME-$kvType-$stamp"
$gpuCsv = "$base.gpu.csv"
$sampler = Start-Process nvidia-smi -ArgumentList '-i 0 --query-gpu=timestamp,power.draw,temperature.gpu,clocks.gr,memory.used --format=csv,noheader,nounits -l 1 -f', "`"$gpuCsv`"" -WindowStyle Hidden -PassThru

$runs = @()
try {
    Write-Host "=== $Model, num_ctx $NumCtx, KV cache $kvType ==="
    Unload-All
    Invoke-Timed (New-Body 'Reply with exactly: warm' 8) | Out-Null

    # Calibrate characters per token on the slice the largest prompt will use: token density varies a lot across
    # this corpus (about 6.2 chars/token in the first 300K characters, 5.7 over the first 570K, 3 beyond that)
    $maxTarget = ($Sizes | Measure-Object -Maximum).Maximum
    $sample = $corpus.Substring(0, [math]::Min([int]($maxTarget * 5.6), $corpus.Length))
    $r, $w = Invoke-Timed (New-Body ("Calibration $(Get-Random)`n" + $sample) 1)
    $charsPerToken = $sample.Length / $r.prompt_eval_count
    Write-Host ("calibration: {0:N2} chars/token over {1} tokens" -f $charsPerToken, $r.prompt_eval_count)

    $question = "`n`nQuestion: Describe what the files above define, as a numbered list covering every file."
    foreach ($target in $Sizes) {
        if ($target + $Reply + 64 -gt $NumCtx) { Write-Warning "$target tokens does not fit num_ctx $NumCtx with a $Reply-token reply; skipped"; continue }
        $chars = [int]($target * $charsPerToken * 0.97)
        for ($i = 1; $i -le $Repeats; $i++) {
            $prompt = "Session $([guid]::NewGuid())`n" + $corpus.Substring(0, $chars) + $question
            Start-Sleep 3
            $t0 = Get-Date
            $r, $w = Invoke-Timed (New-Body $prompt $Reply)
            $run = [pscustomobject][ordered]@{
                target_tokens  = $target
                run            = $i
                prompt_tokens  = $r.prompt_eval_count
                prefill_tok_s  = [math]::Round($r.prompt_eval_count / ($r.prompt_eval_duration / 1e9), 1)
                first_token_s  = [math]::Round(($r.load_duration + $r.prompt_eval_duration) / 1e9, 2)
                output_tokens  = $r.eval_count
                decode_tok_s   = [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2)
                predicted_tok_s = Predict-Decode $r.prompt_eval_count
                wall_s         = [math]::Round($w, 1)
                truncated      = $r.prompt_eval_count -lt 0.8 * $target
                start          = $t0; end = Get-Date
                reply          = $r.response
            }
            if ($run.truncated) { Write-Warning "prompt was cut to $($run.prompt_tokens) tokens" }
            Write-Host ("  {0,7} tokens  run {1}: prefill {2,7} tok/s, first token {3,6} s, decode {4,6} tok/s (predicted {5})" -f $run.prompt_tokens, $i, $run.prefill_tok_s, $run.first_token_s, $run.decode_tok_s, $run.predicted_tok_s)
            $runs += $run
        }
    }
}
finally {
    Stop-Process -Id $sampler.Id -ErrorAction SilentlyContinue
    Unload-All
}

# GPU samples per run (clock during prefill vs decode is the interesting bit)
# nvidia-smi can leave a half-written line when it is stopped, so skip any line that doesn't parse
$samples = Get-Content $gpuCsv | ForEach-Object {
    $f = $_.Split(',') | ForEach-Object { $_.Trim() }
    $t = [datetime]::MinValue
    if ($f.Count -ge 5 -and [datetime]::TryParseExact($f[0], 'yyyy/MM/dd HH:mm:ss.fff', [Globalization.CultureInfo]::InvariantCulture, 'None', [ref]$t)) {
        [pscustomobject]@{ time = $t; power_w = [double]$f[1]; temp_c = [int]$f[2]; clock_mhz = [int]$f[3]; mem_mib = [int]$f[4] }
    }
}
foreach ($run in $runs) {
    $s = @($samples | Where-Object { $_.time -ge $run.start -and $_.time -le $run.end })
    $run | Add-Member avg_power_w ($(if ($s.Count) { [math]::Round(($s | Measure-Object power_w -Average).Average, 1) }))
    $run | Add-Member max_temp_c  ($(if ($s.Count) { ($s | Measure-Object temp_c -Maximum).Maximum }))
    $run | Add-Member max_mem_mib ($(if ($s.Count) { ($s | Measure-Object mem_mib -Maximum).Maximum }))
}

# Summary per prompt size: mean of the repeats
$summary = $runs | Group-Object target_tokens | ForEach-Object {
    $g = $_.Group
    [pscustomobject][ordered]@{
        prompt_tokens   = [int](($g | Measure-Object prompt_tokens -Average).Average)
        prefill_tok_s   = [math]::Round(($g | Measure-Object prefill_tok_s -Average).Average, 0)
        first_token_s   = [math]::Round(($g | Measure-Object first_token_s -Average).Average, 1)
        decode_tok_s    = [math]::Round(($g | Measure-Object decode_tok_s -Average).Average, 1)
        predicted_tok_s = $g[0].predicted_tok_s
        max_mem_mib     = ($g | Measure-Object max_mem_mib -Maximum).Maximum
    }
}

$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s')
    model = $Model; num_ctx = $NumCtx; reply_tokens = $Reply
    ollama_version = (Invoke-RestMethod ($Api + '/api/version')).version
    kv_cache_type = $kvType; flash_attention = $env:OLLAMA_FLASH_ATTENTION
    gpu = (& nvidia-smi -i 0 --query-gpu=name,driver_version,memory.total --format=csv,noheader).Trim()
    power_scheme = (powercfg /getactivescheme) -replace '^.*\((.*)\)\s*$', '$1'
    prediction = @{ bandwidth_gb_s = $BandwidthGBs; weights_gb = $WeightsGB; cache_kib_per_token = $CacheKiBPerToken; cache_factor = $cacheFactor }
    summary = $summary
    runs = $runs
}
[IO.File]::WriteAllText("$base.json", ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$summary | Export-Csv "$base.csv" -NoTypeInformation
$summary | Format-Table -AutoSize | Out-String | Write-Host
Write-Host "Results: $base.json / $base.csv"
Write-Host "GPU samples: $gpuCsv"

Comments

[Mihai Perdum]

The q8_0 slope being 1.6x f16 while moving half the bytes is the part I'd poke at. If the cost really is unpacking every cached value before attention, then it should be flat in the number of bytes and tied to the number of elements, so a q4_0 cache would be a cheap extra data point: same element count, fewer bytes, and if the slope lands at or above 0.62 ms per 1,000 tokens it's dequant overhead rather than bandwidth. It would also be worth checking whether flash attention actually ran the quantized path fused or fell back to a slower kernel for q8_0 on this build, since the server log line you used for the cache size probably shows that too. Either way the fixed 11.8 ms floor is a useful number to have written down.