DEV Community: Steef-Jan Wiggers

Turn 10 of a 50K-token chat waits 5 seconds, not 49: the prefix cache on RTX Spark, and whether the code compiles

Steef-Jan Wiggers — Tue, 03 Nov 2026 08:00:13 +0000

The 146-second wait in my first RTX Spark post was the cost of a brand-new 100K-token prompt. A conversation doesn't work like that: each turn adds a few thousand tokens to a history Ollama has already read. So I measured the thing people actually do. Ten turns of about 5,000 tokens each, and the tenth turn, with 50,375 tokens of history, produced its first token in 5.3 seconds. The same conversation sent cold took 49.4 seconds.

The second test is the one every tokens-per-second post skips: does the code run? I took the three code prompts from the first post, gave the model room for a complete answer, and put the replies through the real tools. All three passed: ten pytest tests including two threading tests for the Python LRU cache, tsc --strict plus a timing check for the TypeScript token bucket, and go vet, go build and a run with hand-checked distances for Dijkstra in Go.

The growing chat

The script holds one conversation open against /api/chat. Every turn appends a fresh ~5K-token slice of text (PowerShell's own module files, as in the earlier posts) and asks a two-sentence question about it; the reply is kept in the history. After ten turns it unloads the model, changes the first line so the prefix cache can't match, and sends the whole conversation again.


Model	qwen3-coder:30b, Q4_K_M, Ollama 0.40.11, f16 cache, 102,400-token context, all on the GPU
Turns	10, each adding 4,400–5,700 tokens of new text plus a question; 128-token replies, temperature 0
Cold run	The same 10-turn history with a new session line and one more question, after `keep_alive: 0`
Machine	Surface Laptop Ultra, RTX Spark N1X, 64 GB; AC power, Balanced plan, 27 Oct 2026

Turn	Prompt	New tokens	First token	New tokens read per second	Decode
1	4,772	4,772	2.1 s	2,262	71.7 tok/s
2	9,218	4,446	2.4 s	1,884	61.2
3	14,163	4,945	2.7 s	1,805	60.4
4	19,878	5,715	3.4 s	1,686	50.0
5	25,531	5,653	4.9 s	1,161	45.5
6	30,537	5,006	4.1 s	1,215	45.0
7	35,810	5,273	4.4 s	1,201	41.1
8	40,617	4,807	4.5 s	1,068	38.3
9	45,609	4,992	5.0 s	992	36.0
10	50,375	4,766	5.3 s	904	33.2
Cold	50,473	50,473	49.4 s	1,022	29.7

Three things in that table:

The cache pays for the history, not the new text. The ten cached first-token waits add up to 38.8 seconds, less than the single cold resend of the final conversation. Each turn only reads what was added since the last one.

New tokens get slower to read as the history grows. Turn 1 read its 4,772 tokens at 2,262 a second; turn 10 read 4,766 at 904. The new tokens attend over everything before them, so reading 5K tokens at the end of a 45K history costs 2.5 times what it did at the start. That is the same slowdown the cold prefill shows between 1K and 57K tokens (2,433 to 1,149 tokens a second in the curve post), just arriving a turn at a time.

Decode doesn't care about the cache. Turn 10 replied at 33.2 tokens a second with a warm cache; the cold run of the same 50K tokens replied at 29.7. The curve post's line, 11.8 ms + 0.39 ms per 1,000 tokens, predicts 31.8 at 50K. Every reply token reads the whole cache whether the prefill was cached or not.

One thing to know about Ollama's numbers: on a cached turn, prompt_eval_count still reports the whole prompt, 50,375 tokens at turn 10, while prompt_eval_duration covers only the new part. Divide one by the other and you get a flattering 9,573 tokens a second. Time to first token is the number that means something; the "new tokens" column is the difference between consecutive turns.

Does the code compile?

The first post timed three code prompts with a 512-token cap, enough to measure decode speed and not enough for a complete answer. This time the cap was 3,072 tokens and the replies were checked.

Prompt	Reply	Code checked	Check	Result
A thread-safe LRU cache in Python with unit tests	2,634 tokens, 87.5 tok/s	348 lines	`pytest`	10 passed, including `test_thread_safety` (five threads, ten put/get pairs each) and `test_concurrent_operations` (five tasks of 100 mixed operations over 50 keys)
A token-bucket rate limiter in TypeScript	1,119 tokens, 88.3 tok/s	143 lines	`tsc --noEmit --strict`, then a timing test	Compiles. Burst of 5 allows exactly 5 of 8 requests; about 3 tokens back after 300 ms at 10/s; the bucket caps at capacity after 1.5 s idle
Dijkstra's shortest-path algorithm in Go	2,142 tokens, 83.2 tok/s	250 lines	`go vet`, `go build`, run	Builds and runs. Distances 0, 4, 2, 9, 11, 14 from vertex 0 on its six-node example; I checked them by hand

The checks ran on a Linux box with Python 3.13, pytest 9, TypeScript 7.0 on Node 22 and Go 1.24, because the laptop has Python but no pytest, Node or Go. The generation ran on the laptop; only the compile and test steps moved.

Three notes from reading the code, since passing proves less than it sounds:

The Python cache uses an OrderedDict behind an RLock, which is the textbook answer. The two threading tests check that nothing crashes and the size ends at, or stays under, the capacity of 3 after concurrent puts and gets; they don't try to catch a lost update, which is the bug a lock is there for.
The Go implementation is the O(V²) array scan, not the heap version. Correct, and what the prompt asked for; it would be the wrong choice for a large sparse graph, and the reply says so in its own complexity note.
The TypeScript limiter takes its fill rate per millisecond, and one of its own usage comments gets that wrong (1 / 1000 is described as one token per millisecond; it is one per second). The code is right, the comment isn't.

One lesson for anyone wiring a local model into a harness: the replies don't stop at the code. The Go answer ends with its sample output in an untagged fence; my first extractor accepted untagged blocks, glued that output onto main.go, and the build failed on line 252 with a syntax error that had nothing to do with the model. The TypeScript answer ends with a nine-line usage snippet in a second typescript fence, which the fixed extractor still appends; it happens to compile, but it is not part of the module. Take the first block tagged with the language, or ask for one file and nothing else.

What it means

Count turns, not tokens, for an interactive session. With the history cached, each turn costs about a second per 1,000 new tokens at the start and about five seconds per 5K by the time the conversation is 50K long. The 146-second wait only happens when the whole context is new: a restart, a different model, or a tool that rewrites the start of the prompt.
Keep the start of the prompt stable. The cache matches a prefix. A system prompt with a timestamp, or an agent that reorders earlier messages, turns every turn into the cold case.
Decode speed follows the context, warm or cold. Plan on 30 tokens a second at 50K and 20 at 100K with an f16 cache on this chip, as the curve post measured.
Check the code, not the speed. At Q4_K_M, qwen3-coder:30b produced working code in three languages on the first try. Whether that holds for harder prompts is a different test; this one says the quantization isn't the thing breaking your build.

Script notes

Both scripts are at the end of the post. Two bugs cost me a run each, and both are in the fixed versions:

Measure is a built-in alias for Measure-Object, and PowerShell resolves aliases before functions, so my Measure function was never called. The function is now Measure-Turn.
Ollama's prompt_eval_count includes the cached prefix, so a running total of it over ten turns adds up to a 277K-token conversation that never existed. The script now reports the prompt size per turn and the new tokens as the difference.

The code check needs python with pytest, node and go on the PATH; it skips a language whose tools are missing and says so, rather than reporting a failure that belongs to the toolchain.

Sources

Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama, the first post, 26 Oct 2026
Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower, the second post, 27 Oct 2026
Every 1,000 tokens of context costs 0.39 ms: decode speed from 1K to 100K on RTX Spark, the third post, 27 Oct 2026
My runs of growing-chat.ps1 and code-check.ps1 on 27 Oct 2026, with the per-turn JSON, the three replies and the tool output

growing-chat.ps1

# The growing chat: does Ollama's prefix cache make turn 10 of a 50K-token conversation cheap?
# Each turn appends a ~5K-token slice of text plus a question to the same conversation, via /api/chat.
# Per turn it records the prompt size, the time to first token and the decode speed. At the end it sends the
# whole conversation cold (new session line, model reloaded) so cached turn N sits next to the uncached cost.
# Note: Ollama's prompt_eval_count is the whole prompt, cached or not, so prefill tok/s from it is not meaningful
# on cached turns; time to first token is the honest measure. new_tokens is the difference between turns.
# Writes growing-chat-<COMPUTER>-<kv>-<stamp>.json and .csv in the current directory.
param(
    [string]$Model = 'qwen3-coder:30b',
    [int]$Turns = 10,
    [int]$TokensPerTurn = 5000,
    [int]$NumCtx = 102400,
    [int]$Reply = 128,
    [string]$Api = 'http://localhost:11434'
)
$ErrorActionPreference = 'Stop'
$kvType = if ($env:OLLAMA_KV_CACHE_TYPE) { $env:OLLAMA_KV_CACHE_TYPE } else { 'f16' }

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($Api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 10) -TimeoutSec 1800
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($Api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    Start-Sleep 3
}
$caps = @((Post-Ollama '/api/show' @{ model = $Model }).capabilities)
function Chat($messages, $n) {
    $b = @{ model = $Model; messages = $messages; stream = $false; keep_alive = '15m'; options = @{ num_ctx = $NumCtx; num_predict = $n; temperature = 0; seed = 42 } }
    if ($caps -contains 'thinking') { $b.think = $false }
    $sw = [Diagnostics.Stopwatch]::StartNew(); $r = Post-Ollama '/api/chat' $b; $sw.Stop()
    return $r, $sw.Elapsed.TotalSeconds
}
function Measure-Turn($r, $wall, $turn, $label, $contextTokens) {
    [pscustomobject][ordered]@{
        turn               = $turn
        label              = $label
        prompt_tokens      = $r.prompt_eval_count      # the whole prompt, cached part included
        new_tokens         = $r.prompt_eval_count - $contextTokens
        first_token_s      = [math]::Round(($r.load_duration + $r.prompt_eval_duration) / 1e9, 2)
        new_tok_per_s      = if ($r.prompt_eval_duration) { [math]::Round(($r.prompt_eval_count - $contextTokens) / ($r.prompt_eval_duration / 1e9), 0) } else { $null }
        decode_tok_s       = [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 1)
        output_tokens      = $r.eval_count
        wall_s             = [math]::Round($wall, 1)
    }
}

# Text slices from PowerShell's own files; each turn gets a different slice so nothing repeats
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
    Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''
while ($corpus.Length -lt $NumCtx * 5) { $corpus += $corpus }

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$base = Join-Path (Get-Location) "growing-chat-$env:COMPUTERNAME-$kvType-$stamp"
$rows = @()
try {
    Write-Host "=== $Model, $Turns turns x ~$TokensPerTurn tokens, num_ctx $NumCtx, KV cache $kvType ==="
    Unload-All
    Chat @(@{ role = 'user'; content = 'Reply with exactly: warm' }) 8 | Out-Null

    # Calibrate characters per token
    $sample = $corpus.Substring(0, 200000)
    $r, $w = Chat @(@{ role = 'user'; content = "Calibration $(Get-Random)`n$sample" }) 1
    $charsPerToken = $sample.Length / $r.prompt_eval_count
    $chars = [int]($TokensPerTurn * $charsPerToken)
    Write-Host ("calibration: {0:N2} chars/token" -f $charsPerToken)
    Unload-All; Chat @(@{ role = 'user'; content = 'Reply with exactly: warm' }) 8 | Out-Null

    $messages = @(@{ role = 'system'; content = "Session $([guid]::NewGuid()). You are reviewing PowerShell source files one batch at a time." })
    $contextTokens = 0   # prompt size of the previous turn, so new_tokens can be derived
    for ($t = 1; $t -le $Turns; $t++) {
        $slice = $corpus.Substring(($t - 1) * $chars, $chars)
        $messages += @{ role = 'user'; content = "Batch ${t}:`n$slice`n`nIn two sentences, what does batch $t define?" }
        $r, $w = Chat $messages $Reply
        $row = Measure-Turn $r $w $t 'cached' $contextTokens
        $messages += @{ role = 'assistant'; content = $r.message.content }
        $contextTokens = $r.prompt_eval_count
        $rows += $row
        Write-Host ("  turn {0,2}: prompt {1,6} tokens ({2,5} new), first token {3,6} s, decode {4,5} tok/s" -f $t, $row.prompt_tokens, $row.new_tokens, $row.first_token_s, $row.decode_tok_s)
        Start-Sleep 2
    }
    $totalContext = $contextTokens

    # Cold comparison: the same final conversation, new session line, after unloading the model
    Write-Host "  cold: resending the full $totalContext-token conversation with a fresh session line"
    Unload-All
    Chat @(@{ role = 'user'; content = 'Reply with exactly: warm' }) 8 | Out-Null
    $cold = @($messages | ForEach-Object { $_ })
    $cold[0] = @{ role = 'system'; content = "Session $([guid]::NewGuid()). You are reviewing PowerShell source files one batch at a time." }
    $cold += @{ role = 'user'; content = 'In two sentences, what did the last batch define?' }
    $r, $w = Chat $cold $Reply
    $coldRow = Measure-Turn $r $w ($Turns + 1) 'cold' 0
    $rows += $coldRow
    Write-Host ("  cold:     prompt {0,6} tokens, first token {1,6} s, decode {2,5} tok/s" -f $coldRow.prompt_tokens, $coldRow.first_token_s, $coldRow.decode_tok_s)
}
finally { Unload-All }

$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s')
    model = $Model; num_ctx = $NumCtx; turns = $Turns; tokens_per_turn = $TokensPerTurn; reply_tokens = $Reply
    ollama_version = (Invoke-RestMethod ($Api + '/api/version')).version
    kv_cache_type = $kvType; flash_attention = $env:OLLAMA_FLASH_ATTENTION
    rows = $rows
}
[IO.File]::WriteAllText("$base.json", ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$rows | Export-Csv "$base.csv" -NoTypeInformation
$rows | Format-Table turn, label, prompt_tokens, new_tokens, first_token_s, new_tok_per_s, decode_tok_s -AutoSize | Out-String | Write-Host
Write-Host "Results: $base.json / $base.csv"

code-check.ps1

# Does the code the laptop writes actually run?
# Re-sends the three code prompts from speed-bench.ps1 to each model, extracts the code blocks from the reply,
# and runs them through the real toolchain: pytest for Python, tsc --strict for TypeScript, go vet + go build for Go.
# Needs on PATH: python (with pytest installed), node/npx, go.
# Writes code-check-<COMPUTER>-<stamp>.json plus a folder per model with the generated files and tool output.
param(
    [string[]]$Models = @('qwen3-coder:30b'),   # add a Q8 tag, e.g. 'qwen3-coder:30b-a3b-q8_0', to compare quantizations
    [int]$Reply = 3072,                          # 512 was enough for speed; a complete answer with tests needs more room
    [int]$NumCtx = 16384,
    [string]$Api = 'http://localhost:11434'
)
$ErrorActionPreference = 'Stop'

$tasks = @(
    @{ name = 'python-lru'; lang = 'python'; file = 'test_lru_cache.py'
       prompt = 'Write a thread-safe LRU cache in Python with unit tests.' }
    @{ name = 'ts-token-bucket'; lang = 'typescript'; file = 'rate-limiter.ts'
       prompt = 'Write a token-bucket rate limiter in TypeScript.' }
    @{ name = 'go-dijkstra'; lang = 'go'; file = 'main.go'
       prompt = "Implement Dijkstra's shortest-path algorithm in Go." }
)

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($Api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 1800
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($Api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    Start-Sleep 3
}
# Pull the fenced code blocks tagged with the language, in order, joined with blank lines.
# Untagged blocks are used only when no tagged block exists: models put sample output and usage snippets in
# untagged fences after the code, and gluing those onto a Go file breaks the build.
function Extract-Code($text, $lang) {
    $tags = switch ($lang) { 'python' { 'python|py' } 'typescript' { 'typescript|ts' } 'go' { 'go|golang' } }
    $all = [regex]::Matches($text, '(?ms)^```

[ \t]*(\w*)[^\n]*\n(.*?)(?:^

```|\z)')
    $tagged = @($all | Where-Object { $_.Groups[1].Value -match "^($tags)$" })
    if (-not $tagged) { $tagged = @($all | Where-Object { $_.Groups[1].Value -eq '' }) }
    if (-not $tagged) { return $null }
    (($tagged | ForEach-Object { $_.Groups[2].Value.TrimEnd() }) -join "`n`n")
}
function Run-Tool($dir, $cmd, $args) {
    Push-Location $dir
    try {
        $output = & $cmd @args 2>&1 | Out-String
        [pscustomobject]@{ command = "$cmd $($args -join ' ')"; exit_code = $LASTEXITCODE; output = $output.Trim() }
    } finally { Pop-Location }
}

$have = @{}
foreach ($c in 'python', 'npx', 'go') { $have[$c] = [bool](Get-Command $c -ErrorAction SilentlyContinue) }
Write-Host ("toolchain: python={0} npx={1} go={2}" -f $have.python, $have.npx, $have.go)

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$root = Join-Path (Get-Location) "code-check-$env:COMPUTERNAME-$stamp"
New-Item -ItemType Directory -Path $root | Out-Null
$results = @()

try {
    foreach ($model in $Models) {
        Write-Host "=== $model ==="
        Unload-All
        $caps = @((Post-Ollama '/api/show' @{ model = $model }).capabilities)
        $modelDir = Join-Path $root ($model -replace '[:/]', '-')
        New-Item -ItemType Directory -Path $modelDir | Out-Null

        foreach ($t in $tasks) {
            $dir = Join-Path $modelDir $t.name
            New-Item -ItemType Directory -Path $dir | Out-Null
            $b = @{ model = $model; prompt = $t.prompt; stream = $false; keep_alive = '15m'; options = @{ num_ctx = $NumCtx; num_predict = $Reply; temperature = 0; seed = 42 } }
            if ($caps -contains 'thinking') { $b.think = $false }
            $sw = [Diagnostics.Stopwatch]::StartNew(); $r = Post-Ollama '/api/generate' $b; $sw.Stop()
            [IO.File]::WriteAllText((Join-Path $dir 'reply.md'), $r.response, (New-Object Text.UTF8Encoding $false))

            $code = Extract-Code $r.response $t.lang
            $res = [ordered]@{
                model = $model; task = $t.name; language = $t.lang
                output_tokens = $r.eval_count; done_reason = $r.done_reason; decode_tok_s = [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 1)
                code_blocks_found = [bool]$code; code_lines = if ($code) { ($code -split "`n").Count } else { 0 }
                checks = @(); passed = $false; skipped = $null
            }
            if ($r.done_reason -eq 'length') { Write-Warning "  $($t.name): reply hit the $Reply-token limit; the code may be cut off" }
            if (-not $code) { Write-Warning "  $($t.name): no code block found"; $results += [pscustomobject]$res; continue }
            [IO.File]::WriteAllText((Join-Path $dir $t.file), $code + "`n", (New-Object Text.UTF8Encoding $false))

            try {
                switch ($t.lang) {
                    'python' {
                        if (-not $have.python) { $res.skipped = 'python not on PATH'; break }
                        $res.checks += Run-Tool $dir 'python' @('-m', 'pytest', '-q', $t.file)
                        $res.passed = $res.checks[-1].exit_code -eq 0
                    }
                    'typescript' {
                        if (-not $have.npx) { $res.skipped = 'node/npx not on PATH'; break }
                        $res.checks += Run-Tool $dir 'npx' @('-y', '-p', 'typescript', 'tsc', '--noEmit', '--strict', '--target', 'es2020', $t.file)
                        $res.passed = $res.checks[-1].exit_code -eq 0
                    }
                    'go' {
                        if (-not $have.go) { $res.skipped = 'go not on PATH'; break }
                        Run-Tool $dir 'go' @('mod', 'init', 'codecheck') | Out-Null
                        $res.checks += Run-Tool $dir 'go' @('vet', './...')
                        $res.checks += Run-Tool $dir 'go' @('build', './...')
                        $res.passed = ($res.checks | Where-Object exit_code -ne 0).Count -eq 0
                    }
                }
            } catch { $res.checks += [pscustomobject]@{ command = 'toolchain'; exit_code = -1; output = "$_" } }
            # Keep the tool output next to the code for the post
            ($res.checks | ForEach-Object { "> $($_.command)`nexit $($_.exit_code)`n$($_.output)`n" }) -join "`n" | Set-Content (Join-Path $dir 'check.log')
            $verdict = if ($res.skipped) { "SKIP ($($res.skipped))" } elseif ($res.passed) { 'PASS' } else { 'FAIL' }
            Write-Host ("  {0,-16} {1}  ({2} tokens, {3} lines, {4} tok/s)" -f $t.name, $verdict, $res.output_tokens, $res.code_lines, $res.decode_tok_s)
            $results += [pscustomobject]$res
        }
    }
}
finally { Unload-All }

$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s')
    ollama_version = (Invoke-RestMethod ($Api + '/api/version')).version
    reply_tokens = $Reply
    toolchain = [ordered]@{
        python = if ($have.python) { (& python --version 2>&1 | Out-String).Trim() } else { 'missing' }
        pytest = if ($have.python) { (& python -m pytest --version 2>&1 | Out-String).Trim() } else { 'missing' }
        node   = if ($have.npx) { (& node --version 2>&1 | Out-String).Trim() } else { 'missing' }
        go     = if ($have.go) { (& go version 2>&1 | Out-String).Trim() } else { 'missing' }
    }
    results = $results
}
[IO.File]::WriteAllText((Join-Path $root 'code-check.json'), ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$results | Format-Table model, task, passed, skipped, done_reason, output_tokens, code_lines -AutoSize | Out-String | Write-Host
Write-Host "Results and generated code: $root"

Every 1,000 tokens of context costs 0.39 ms: decode speed from 1K to 100K on RTX Spark

Steef-Jan Wiggers — Thu, 29 Oct 2026 15:22:59 +0000

On my Surface Laptop Ultra, qwen3-coder:30b's time per token grows in a straight line with the context it has to read: 11.8 ms plus 0.39 ms for every 1,000 tokens of f16 cache. That's within 7% of what memory bandwidth predicts, so the bytes-per-token model from my first RTX Spark post holds for the cache. Decode falls from 80 tokens a second at 1K tokens to 20 at 97K.

The 8-bit cache follows a straight line too, but a steeper one: 0.62 ms per 1,000 tokens, 1.6 times f16, for half the bytes. q8_0 is never faster than f16 at any length I tested, and from 8K tokens on it is clearly slower. This turns my second post's single 100K measurement into a curve.

Method and the prediction

One question after prompts of seven sizes, three runs each, with the context window fixed at 102,400 tokens so only the amount of filled cache changes.


Model	qwen3-coder:30b, Q4_K_M, Ollama 0.40.11, 100% on the GPU
Prompt sizes	935, 3,651, 7,159, 13,974, 30,009, 57,365 and 96,926 tokens
Reply	256 tokens, temperature 0, thinking off
Prompt cache	A random first line on every prompt, so each one is read in full
Cache types	f16, then q8_0, each in its own `ollama serve` with flash attention on; the server log confirmed 9,600 and 5,100 MiB caches
Machine	Surface Laptop Ultra, RTX Spark N1X, 64 GB; AC power, Balanced plan, 27 Oct 2026

The prediction is the first post's bandwidth model, now per token of context:

time per token ≈ (2.0 GB + context tokens × 96 KiB × cache factor) ÷ 235 GB/s

The cache factor is 1 for f16 and 0.53 for q8_0. That gives a straight line: 8.5 ms of weights plus 0.42 ms per 1,000 cached tokens for f16, and 0.22 ms for q8_0.

The curve

Plotted as milliseconds per token, both cache types are straight lines; the difference between them is the slope.

A least-squares line through the seven means fits both series almost exactly (R² 0.999):

	Fixed cost per token	Added per 1,000 cached tokens	Implied bandwidth on the cache
Predicted, f16	8.5 ms	0.42 ms	235 GB/s
Measured, f16	11.8 ms	0.39 ms	~250 GB/s
Predicted, q8_0	8.5 ms	0.22 ms	235 GB/s
Measured, q8_0	12.4 ms	0.62 ms	~84 GB/s

The runs were consistent: at 57K tokens the three f16 runs spanned 29.5–30.0 tokens a second and q8_0 21.5–21.8. None of the 42 prompts was truncated.

Where the measurement leaves the prediction

The measured lines peel away from the prediction in two different places, for two different reasons.

f16 peels away at short context, not long. The slope matches: each cached token costs what reading 96 KiB at about 250 GB/s takes. The gap is the starting point, 11.8 ms instead of 8.5. Those extra 3.3 ms per token aren't memory reads; my best guess is the mixture-of-experts routing, kernel launches and sampling that happen once per token whatever the context. At 1K tokens that overhead is a quarter of every token, so decode lands at 80 tokens a second instead of the predicted 112; at 97K it is 7%, and the prediction is within half a token a second.

q8_0 peels away more with every token. Its fixed cost matches f16's (12.4 ms), but each 1,000 cached tokens adds 0.62 ms where 0.22 was predicted, 2.8 times the bandwidth estimate and 1.6 times f16. So the extra cost of the 8-bit cache isn't a one-off; it scales with the cache, which points at unpacking every cached value before attention can use it. That's my reading of the slope, not a profile.

The two lines start together: at 1K and 4K tokens the difference between f16 and q8_0 is inside the run-to-run spread. By 14K tokens q8_0 is 25% slower, and from 30K to 97K it is 26–32% slower.

One check from the notes: the 1K decode of 80 tokens a second sits below the first post's 90.9. It matches the 81.7 that f16 managed on today's 512-token code prompts, so the 256-token reply isn't too short to average out startup; this session simply ran about 10% below the first one.

Prompt processing, first token and clocks

Prompt processing slows as the prompt grows, from about 2,400 tokens a second up to 8K tokens to 795 at 97K, and the cache type barely matters there.

Prompt	Prefill, f16	First token, f16	Prefill, q8_0	First token, q8_0	Clock reading the prompt, f16	Clock replying, f16
935 tokens	2,398 tok/s	0.4 s	2,311 tok/s	0.4 s	too short to sample	2,246 MHz
3,651	2,255	1.6 s	2,169	1.7 s	too short to sample	2,307 MHz
7,159	2,377	3.0 s	2,144	3.4 s	1,596 MHz	2,153 MHz
13,974	2,072	6.8 s	1,809	7.8 s	1,861 MHz	2,159 MHz
30,009	1,562	19.2 s	1,466	20.5 s	1,846 MHz	2,229 MHz
57,365	1,153	49.8 s	1,109	51.8 s	1,775 MHz	2,332 MHz
96,926	795	122 s	779	125 s	1,702 MHz	2,266 MHz

The power-limit pattern from the first post holds at every size I could sample: the GPU reads a prompt at 1.6–1.9 GHz and replies at 2.15–2.33 GHz. Reading is compute-bound and hits the power limit; replying is memory-bound and runs at full clock. During the f16 run the GPU peaked at 73 °C.

q8_0 read prompts 2–13% slower than f16 here. In the previous post it was slightly faster at 100K; both differences are small next to the 3× drop in prefill speed between 8K and 97K tokens.

What it means

Estimate decode speed from the context length. For qwen3-coder:30b on RTX Spark with f16: time per token ≈ 11.8 ms + 0.39 ms per 1,000 tokens in the conversation. At 50K tokens that is 31 ms, or about 32 tokens a second.
Long conversations cost linearly, not suddenly. There is no cliff; each 10K tokens of history adds about 4 ms to every reply token.
Keep f16 unless you need the memory. q8_0 saves 4.4 GiB at 102,400 tokens but adds 0.62 ms per 1,000 tokens instead of 0.39, and never wins at any length.
Short prompts are limited by fixed work, not memory. Below 8K tokens, a faster cache or more bandwidth would change little; the 11.8 ms per token is the floor.

The bandwidth model from the first post now has a clear scope: it predicts the slope of the f16 line to within 7%, it underestimates the fixed cost by 3.3 ms, and it does not apply to quantized caches.

Script notes

The script is context-curve.ps1, run once per cache type by a small wrapper that starts ollama serve with that type and checks the llama_kv_cache line in its log. Two fixes went in along the way:

Calibration. Characters per token are measured over the same text the largest prompt uses. This corpus runs at about 6.2 characters per token at the start and 5.7 over the first 570K characters; calibrating on a shorter slice would have pushed the 100K prompt past the context, and Ollama would have cut it in half without an error.
GPU sample parsing. Stopping nvidia-smi can leave a half-written last line in its CSV, which crashed the q8_0 run's final step after all measurements were taken. Those numbers come from the run's console output; the parser now skips any line it can't read.

Sources

Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama, the first post, 26 Oct 2026
Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower, the second post, 27 Oct 2026
My runs of context-curve.ps1 on 27 Oct 2026: 42 prompts, with per-run JSON, CSV and nvidia-smi samples for f16, and the console output for q8_0:

# Decode speed vs context length on Ollama.
# Sends one question after prompts of increasing size (default 1K, 8K, 32K, 64K, 100K tokens),
# each with a random first line so the prefix cache cannot help, and records Ollama's own timings.
# Prints measured decode next to the bandwidth prediction (bandwidth / bytes read per token).
# Writes context-curve-<COMPUTER>-<kv>-<stamp>.json and .csv in the current directory.
param(
    [string]$Model = 'qwen3-coder:30b',
    [int[]]$Sizes = @(1000, 8000, 32000, 64000, 100000),
    [int]$Repeats = 2,
    [int]$NumCtx = 102400,          # kept constant so only the prompt length varies
    [int]$Reply = 256,
    [string]$Api = 'http://127.0.0.1:11434',
    # Prediction inputs for qwen3-coder:30b at Q4_K_M; change for another model
    [double]$BandwidthGBs = 235,
    [double]$WeightsGB = 2.0,       # active weights read per token
    [double]$CacheKiBPerToken = 96  # 48 layers x 4 KV heads x 128 dims x 2 (K,V) x 2 bytes, f16
)
$ErrorActionPreference = 'Stop'

$kvType = if ($env:OLLAMA_KV_CACHE_TYPE) { $env:OLLAMA_KV_CACHE_TYPE } else { 'f16' }
$cacheFactor = switch ($kvType) { 'q8_0' { 0.53 } 'q4_0' { 0.28 } default { 1.0 } }   # bytes relative to f16 (q8_0 = 8.5 bits, q4_0 = 4.5 bits per value)

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($Api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 1800
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($Api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    Start-Sleep 3
}
function Invoke-Timed($body) {
    $sw = [Diagnostics.Stopwatch]::StartNew(); $r = Post-Ollama '/api/generate' $body; $sw.Stop()
    return $r, $sw.Elapsed.TotalSeconds
}
function Predict-Decode($tokens) {
    $bytesGB = $WeightsGB + ($tokens * $CacheKiBPerToken * 1024 * $cacheFactor / 1e9)
    [math]::Round($BandwidthGBs / $bytesGB, 1)
}

$caps = @((Post-Ollama '/api/show' @{ model = $Model }).capabilities)
function New-Body($prompt, $n) {
    $b = @{ model = $Model; prompt = $prompt; stream = $false; keep_alive = '15m'; options = @{ num_ctx = $NumCtx; num_predict = $n; temperature = 0; seed = 42 } }
    if ($caps -contains 'thinking') { $b.think = $false }
    $b
}

# Prompt text: PowerShell's own format/type files and module sources, present on every Windows machine
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
    Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''
while ($corpus.Length -lt $NumCtx * 5) { $corpus += $corpus }

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$base = Join-Path (Get-Location) "context-curve-$env:COMPUTERNAME-$kvType-$stamp"
$gpuCsv = "$base.gpu.csv"
$sampler = Start-Process nvidia-smi -ArgumentList '-i 0 --query-gpu=timestamp,power.draw,temperature.gpu,clocks.gr,memory.used --format=csv,noheader,nounits -l 1 -f', "`"$gpuCsv`"" -WindowStyle Hidden -PassThru

$runs = @()
try {
    Write-Host "=== $Model, num_ctx $NumCtx, KV cache $kvType ==="
    Unload-All
    Invoke-Timed (New-Body 'Reply with exactly: warm' 8) | Out-Null

    # Calibrate characters per token on the slice the largest prompt will use: token density varies a lot across
    # this corpus (about 6.2 chars/token in the first 300K characters, 5.7 over the first 570K, 3 beyond that)
    $maxTarget = ($Sizes | Measure-Object -Maximum).Maximum
    $sample = $corpus.Substring(0, [math]::Min([int]($maxTarget * 5.6), $corpus.Length))
    $r, $w = Invoke-Timed (New-Body ("Calibration $(Get-Random)`n" + $sample) 1)
    $charsPerToken = $sample.Length / $r.prompt_eval_count
    Write-Host ("calibration: {0:N2} chars/token over {1} tokens" -f $charsPerToken, $r.prompt_eval_count)

    $question = "`n`nQuestion: Describe what the files above define, as a numbered list covering every file."
    foreach ($target in $Sizes) {
        if ($target + $Reply + 64 -gt $NumCtx) { Write-Warning "$target tokens does not fit num_ctx $NumCtx with a $Reply-token reply; skipped"; continue }
        $chars = [int]($target * $charsPerToken * 0.97)
        for ($i = 1; $i -le $Repeats; $i++) {
            $prompt = "Session $([guid]::NewGuid())`n" + $corpus.Substring(0, $chars) + $question
            Start-Sleep 3
            $t0 = Get-Date
            $r, $w = Invoke-Timed (New-Body $prompt $Reply)
            $run = [pscustomobject][ordered]@{
                target_tokens  = $target
                run            = $i
                prompt_tokens  = $r.prompt_eval_count
                prefill_tok_s  = [math]::Round($r.prompt_eval_count / ($r.prompt_eval_duration / 1e9), 1)
                first_token_s  = [math]::Round(($r.load_duration + $r.prompt_eval_duration) / 1e9, 2)
                output_tokens  = $r.eval_count
                decode_tok_s   = [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2)
                predicted_tok_s = Predict-Decode $r.prompt_eval_count
                wall_s         = [math]::Round($w, 1)
                truncated      = $r.prompt_eval_count -lt 0.8 * $target
                start          = $t0; end = Get-Date
                reply          = $r.response
            }
            if ($run.truncated) { Write-Warning "prompt was cut to $($run.prompt_tokens) tokens" }
            Write-Host ("  {0,7} tokens  run {1}: prefill {2,7} tok/s, first token {3,6} s, decode {4,6} tok/s (predicted {5})" -f $run.prompt_tokens, $i, $run.prefill_tok_s, $run.first_token_s, $run.decode_tok_s, $run.predicted_tok_s)
            $runs += $run
        }
    }
}
finally {
    Stop-Process -Id $sampler.Id -ErrorAction SilentlyContinue
    Unload-All
}

# GPU samples per run (clock during prefill vs decode is the interesting bit)
# nvidia-smi can leave a half-written line when it is stopped, so skip any line that doesn't parse
$samples = Get-Content $gpuCsv | ForEach-Object {
    $f = $_.Split(',') | ForEach-Object { $_.Trim() }
    $t = [datetime]::MinValue
    if ($f.Count -ge 5 -and [datetime]::TryParseExact($f[0], 'yyyy/MM/dd HH:mm:ss.fff', [Globalization.CultureInfo]::InvariantCulture, 'None', [ref]$t)) {
        [pscustomobject]@{ time = $t; power_w = [double]$f[1]; temp_c = [int]$f[2]; clock_mhz = [int]$f[3]; mem_mib = [int]$f[4] }
    }
}
foreach ($run in $runs) {
    $s = @($samples | Where-Object { $_.time -ge $run.start -and $_.time -le $run.end })
    $run | Add-Member avg_power_w ($(if ($s.Count) { [math]::Round(($s | Measure-Object power_w -Average).Average, 1) }))
    $run | Add-Member max_temp_c  ($(if ($s.Count) { ($s | Measure-Object temp_c -Maximum).Maximum }))
    $run | Add-Member max_mem_mib ($(if ($s.Count) { ($s | Measure-Object mem_mib -Maximum).Maximum }))
}

# Summary per prompt size: mean of the repeats
$summary = $runs | Group-Object target_tokens | ForEach-Object {
    $g = $_.Group
    [pscustomobject][ordered]@{
        prompt_tokens   = [int](($g | Measure-Object prompt_tokens -Average).Average)
        prefill_tok_s   = [math]::Round(($g | Measure-Object prefill_tok_s -Average).Average, 0)
        first_token_s   = [math]::Round(($g | Measure-Object first_token_s -Average).Average, 1)
        decode_tok_s    = [math]::Round(($g | Measure-Object decode_tok_s -Average).Average, 1)
        predicted_tok_s = $g[0].predicted_tok_s
        max_mem_mib     = ($g | Measure-Object max_mem_mib -Maximum).Maximum
    }
}

$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s')
    model = $Model; num_ctx = $NumCtx; reply_tokens = $Reply
    ollama_version = (Invoke-RestMethod ($Api + '/api/version')).version
    kv_cache_type = $kvType; flash_attention = $env:OLLAMA_FLASH_ATTENTION
    gpu = (& nvidia-smi -i 0 --query-gpu=name,driver_version,memory.total --format=csv,noheader).Trim()
    power_scheme = (powercfg /getactivescheme) -replace '^.*\((.*)\)\s*$', '$1'
    prediction = @{ bandwidth_gb_s = $BandwidthGBs; weights_gb = $WeightsGB; cache_kib_per_token = $CacheKiBPerToken; cache_factor = $cacheFactor }
    summary = $summary
    runs = $runs
}
[IO.File]::WriteAllText("$base.json", ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$summary | Export-Csv "$base.csv" -NoTypeInformation
$summary | Format-Table -AutoSize | Out-String | Write-Host
Write-Host "Results: $base.json / $base.csv"
Write-Host "GPU samples: $gpuCsv"

Quantizing the KV cache in Ollama on RTX Spark: half the memory, a third slower

Steef-Jan Wiggers — Tue, 27 Oct 2026 16:09:00 +0000

I predicted that an 8-bit KV cache would take qwen3-coder:30b from 20 to 34 tokens a second at a 100K-token prompt, and a 4-bit cache to 52. On my Surface Laptop Ultra with RTX Spark, both made it slower: 13.8 tokens a second with q8_0 and 13.1 with q4_0, against 19.9 with the default f16 cache.

The memory savings were exactly as advertised. The cache shrank from 9,600 MiB to 5,100 (q8_0) and 2,700 (q4_0), and that room let the model load a 262,144-token context on the GPU. Prompt processing and short replies barely changed, and the replies kept the same facts in different words.

This is a follow-up to my first RTX Spark post, which ended with this experiment as the next step.

The prediction

The first post showed decode on this chip tracking memory bandwidth: every token reads the active weights plus the whole attention cache, at about 235 GB/s. A smaller cache should mean a faster reply.

Cache type	Weights per token	Cache per token at 100K	Predicted decode
f16 (default)	~2.0 GB	~9.9 GB	20 tok/s
q8_0	~2.0 GB	~5.0 GB	34 tok/s
q4_0	~2.0 GB	~2.5 GB	52 tok/s

Two things could break it. Unpacking a quantized cache adds work to every token, so q4_0 might undershoot. And prompt processing might change too, which is why I report it separately.

Setup, and checking it took effect

Ollama's cache type is two environment variables: OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE=f16, q8_0 or q4_0. A quantized cache needs flash attention, and Ollama gives K and V the same type; a split like q8 keys with q4 values needs llama.cpp directly.

Setting the variable is a request, not a guarantee: on an unsupported model the cache can fall back to f16 without an error. So the script checks what llama.cpp actually started with. On this machine, all three took effect:

llama-server.exe ... -c 102400 --cache-type-k q8_0 --cache-type-v q8_0 --flash-attn on
llama_kv_cache: size = 5100.00 MiB (102400 cells, 48 layers, 1/1 seqs), K (q8_0): 2550.00 MiB, V (q8_0): 2550.00 MiB

To switch types in one sitting, kv-cache-bench.ps1 stops the tray app and runs ollama serve with the variables set for that process only, then restarts the tray app as it was. Same day, same driver (617.42), same Ollama (0.40.11), on AC power with the Balanced plan.

	Per cache type
Code benchmark	The first post's three prompts (LRU cache, rate limiter, Dijkstra), 512-token replies
Long prompt	The first post's 100,560-token prompt, 384-token reply, three repeats
Prompt cache	The model unloads before every long repeat, so all three repeats read the identical prompt in full
Quality check	Each long reply saved and compared word by word with the first f16 reply

One lesson from Windows PowerShell 5.1: its Invoke-RestMethod never reached the freshly started server, and the script stalled without an error. PowerShell 7 worked at once.

Results at 100K tokens

f16 landed on its prediction; both quantized caches decoded about a third slower than f16, where I expected 1.7 and 2.6 times faster.

The runs were steady: f16 19.2–20.6, q8_0 13.6–14.1 and q4_0 13.0–13.4 tokens a second. Everything else either stayed flat or improved:

Cache type	KV cache	Peak GPU memory	Prefill, 100K prompt	Short code prompts
f16	9,600 MiB	30,518 MiB	760 tok/s	81.7 tok/s
q8_0	5,100 MiB	26,322 MiB	780 tok/s	87.9 tok/s
q4_0	2,700 MiB	23,741 MiB	803 tok/s	88.3 tok/s

The cache sizes are exact: q8_0 stores 8.5 bits per value and q4_0 4.5, so 53% and 28% of f16. Prefill took 123–142 seconds in every run. Short code replies ran at 81.7 tokens a second with f16, below the first post's 90.9; f16 ran first in the session, so treat that gap as run-to-run variation rather than a cache effect.

Longer contexts: the full 262,144 tokens now fit

The real win is capacity: with q4_0, qwen3-coder loads its full 262,144-token context entirely on the GPU, the context the Lenovo in Microsoft's post used. My first post stopped at 102,400 tokens because the f16 cache would not fit.

Run	Context	Prompt	KV cache	Peak GPU memory	Prefill	First token after	Decode
q8_0	163,840	148,051 tokens	8,160 MiB	28,761 MiB	511 tok/s	4 min 50 s	9.9 tok/s
q4_0	262,144	242,898 tokens	6,912 MiB	28,379 MiB	349 tok/s	11 min 36 s	6.5 tok/s

At this size f16 has no chance: its cache alone would be about 24 GB next to 17 GB of weights, and q8_0 would need about 13 GB. Both runs stayed 100% on the GPU and peaked at 74–75 °C.

It fits, but it's slow. Reading a fresh 243K-token prompt took almost 12 minutes, and replies came at 6.5 tokens a second. Both replies were still correct and on topic. In an agent session the prompt cache reads most of each turn's history only once, so the 12 minutes is a one-time cost per conversation; the 6.5 tokens a second is not.

One more lesson in measuring: my first attempt at these two runs came out at 197,535 and 332,024 tokens, and Ollama silently kept only half the context. Token density varies across the source text, so the longer prompts are now built from whole copies of the 100K prompt.

Quality check: same facts, different words

All nine long replies were accurate, and each cache type produced the same reply on all three of its runs; only the wording changed between cache types.

Cache type	Identical across its 3 runs	Word overlap with the f16 reply	Files described in 384 tokens
f16	Yes	100%	7
q8_0	Yes	50%	6
q4_0	Yes	62%	7

Word overlap is twice the longest common run of words divided by both replies' lengths. Every reply walked through the same files in the same order, starting with Certificate.format.ps1xml, and named largely the same .NET types for each. q8_0 wrote longer descriptions, so it reached one file fewer before the 384-token limit.

At temperature 0 with a fixed seed, f16 was fully deterministic. The quantized caches change the numbers the model attends to just enough to pick different words early, and the rest of the reply follows from that. One question on 100K tokens isn't a full quality evaluation, but it found no wrong facts.

Why the prediction missed

The bandwidth model only holds while reading memory is the slowest part of a token. With a quantized cache it no longer is:

Cache type	Bytes read per token at 100K	Measured decode	Implied bandwidth
f16	~11.9 GB	19.9 tok/s	~235 GB/s
q8_0	~7.3 GB	13.8 tok/s	~100 GB/s
q4_0	~4.8 GB	13.1 tok/s	~63 GB/s

If memory were still the limit, all three rows would imply the same bandwidth. Instead q8_0 and q4_0 read less and still take longer per token, so the time goes into computing, not reading. Each step has to unpack about 100,000 cached keys and values per layer from 8 or 4 bits before flash attention can use them, and that costs more than the bytes it saves.

That's my reading of the numbers, not a profile. The f16 path in llama.cpp's flash attention is the most optimised one, and the quantized paths may simply be less tuned for this new Blackwell GPU yet. Prefill, which processes 512 tokens at a time, hides the unpacking cost and ran 3–6% faster with a quantized cache.

In hindsight the warning was in my own setup notes. The OpenClaw DC guide I took the configuration from sums it up as memory saved, not time, and cites q4_0 decoding about 37% slower at 110K tokens. I measured 34% slower at 100K on a different GPU, so my bandwidth prediction was the outlier, not the hardware.

What I'd use

Up to about 100K tokens: keep f16. It's the fastest cache on this chip for long prompts, and it fits.
Beyond 102K tokens: q4_0. It's the only way to fit qwen3-coder's full 262,144-token context on a 32 GB GPU share, and its replies stayed accurate.
When memory, not context, is tight: q8_0 or q4_0. They free 4.4–6.7 GiB at 100K tokens for other apps or a second model, at the cost of about a third of the long-prompt decode speed.
Always check the log. Look for --cache-type-k on the llama-server command line and the llama_kv_cache size line; the variable alone proves nothing.

The prediction failed in an informative way: the bytes-per-token model is right for f16 and wrong for quantized caches, because they turn a memory-bound step into a compute-bound one. Next I want to try llama.cpp directly, where keys and values can have different types, and see whether a newer build speeds up the quantized flash-attention path on Blackwell.

Sources

Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama, my first post, 26 Oct 2026
Qwen3.8-27B on a Lenovo laptop: run it on RTX Spark, watch it in Azure Monitor, Microsoft Tech Community, 26 Oct 2026
llama.cpp KV Cache Quantization: q8_0 vs q4_0 vs f16 (2026), OpenClaw DC, 8 Aug 2026, for the flash-attention requirement, the silent fallback, and its decode measurement
My runs of kv-cache-bench.ps1 on 27 Oct 2026: one session for the three cache types and one for the long-context runs, each with server logs, GPU samples, and the reply texts:

# KV-cache quantization benchmark for Ollama: f16 vs q8_0 vs q4_0 on the same model, prompts and day.
# Runs Ollama as `ollama serve` with OLLAMA_FLASH_ATTENTION=1 and OLLAMA_KV_CACHE_TYPE set per run (process
# environment only; no user variables change), then restarts the tray app as it was.
# Per cache type: three code prompts (512-token replies) and $Repeats long prompts (~100,589 tokens, 384-token
# replies), the model unloaded before each long repeat so the prompt cache never helps and every prompt is identical.
# Extras: q8_0 at 163,840 context (~154K prompt) and q4_0 at 262,144 (~247K prompt), one repeat each.
# Writes everything to C:\Dev\kv-bench-<COMPUTER>-<stamp>\ : results.json, server logs, GPU samples, reply texts.
param(
    [string]$Model = 'qwen3-coder:30b',
    [string[]]$CacheTypes = @('f16', 'q8_0', 'q4_0'),
    [int]$Repeats = 3,
    [switch]$SkipExtras,
    [switch]$ExtrasOnly,
    [double]$CharsPerToken = 6.05   # measured by speed-bench.ps1 on this corpus; fixed so every run builds the same prompt
)
$ErrorActionPreference = 'Stop'
$api = 'http://127.0.0.1:11434'
$ollamaDir = "$env:LOCALAPPDATA\Programs\Ollama"
$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$outDir = "C:\Dev\kv-bench-$env:COMPUTERNAME-$stamp"
New-Item -ItemType Directory -Force $outDir | Out-Null
$progress = "$outDir\progress.txt"
function Say($msg) { $line = "$(Get-Date -Format 'HH:mm:ss') $msg"; Write-Host $line; Add-Content $progress $line }

$codePrompts = @(
    'Write a thread-safe LRU cache in Python with unit tests.',
    'Write a token-bucket rate limiter in TypeScript.',
    "Implement Dijkstra's shortest-path algorithm in Go."
)
$question = "`n`nQuestion: Describe what the files above define, as a numbered list covering every file."
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
    Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''
function New-LongPrompt($targetTokens) {
    $chars = [int]($targetTokens * $CharsPerToken * 0.97)
    $text = $corpus
    while ($text.Length -lt $chars) { $text += $corpus }
    "Session benchmark-1`n" + $text.Substring(0, $chars) + $question
}

# Longer prompts are whole and partial copies of the 100K prompt's text, whose token count is known (100,540 tokens
# without the first line and the question): token density varies too much across the corpus to scale by characters
function New-RepeatedPrompt($targetTokens) {
    $blockChars = [int](97880 * $CharsPerToken * 0.97)
    $block = $corpus.Substring(0, $blockChars)
    $copies = $targetTokens / 100540
    $text = ''
    for ($k = 1; $k -le [math]::Floor($copies); $k++) { $text += "`n### COPY $k`n" + $block }
    $rest = [int](($copies - [math]::Floor($copies)) * $blockChars)
    if ($rest -gt 0) { $text += "`n### COPY PART`n" + $block.Substring(0, $rest) }
    "Session benchmark-1`n" + $text + $question
}

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 3600
}
function Stop-Ollama {
    Get-Process -ErrorAction SilentlyContinue | Where-Object { $_.Name -in @('ollama app', 'ollama', 'llama-server') } | Stop-Process -Force
    Start-Sleep 3
}
function Start-OllamaServe($cache, $log) {
    $env:OLLAMA_FLASH_ATTENTION = '1'
    $env:OLLAMA_KV_CACHE_TYPE = $cache
    # cmd does the redirection: PowerShell's own -RedirectStandardError kept this script from continuing
    Start-Process cmd.exe -ArgumentList '/c', "`"`"$ollamaDir\ollama.exe`" serve > `"$log`" 2>&1`"" -WindowStyle Hidden
    for ($i = 0; $i -lt 60; $i++) {
        try { Invoke-RestMethod ($api + '/api/version') -TimeoutSec 2 | Out-Null; Say '  server is up'; return } catch { Start-Sleep 1 }
    }
    throw "ollama serve did not answer within 60 s ($cache)"
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    for ($i = 0; $i -lt 30 -and @((Invoke-RestMethod ($api + '/api/ps')).models).Count; $i++) { Start-Sleep 1 }
    Start-Sleep 2
}
function Get-GpuMemMiB { [int](& nvidia-smi -i 0 --query-gpu=memory.used --format=csv,noheader,nounits).Trim() }
function Measure-Reply($r, $wall) {
    [ordered]@{
        prompt_tokens = $r.prompt_eval_count
        prefill_tok_s = if ($r.prompt_eval_duration) { [math]::Round($r.prompt_eval_count / ($r.prompt_eval_duration / 1e9), 1) } else { $null }
        prefill_s     = [math]::Round($r.prompt_eval_duration / 1e9, 1)
        output_tokens = $r.eval_count
        decode_tok_s  = if ($r.eval_duration) { [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2) } else { $null }
        load_s        = [math]::Round($r.load_duration / 1e9, 1)
        wall_s        = [math]::Round($wall, 1)
        done_reason   = $r.done_reason
    }
}
# Word-level similarity: 2 * longest common subsequence / (words in a + words in b); 1.0 = identical
function Get-Similarity($a, $b) {
    $x = @($a -split '\s+' | Where-Object { $_ }); $y = @($b -split '\s+' | Where-Object { $_ })
    if (-not $x.Count -or -not $y.Count) { return 0 }
    $prev = New-Object int[] ($y.Count + 1)
    foreach ($wx in $x) {
        $cur = New-Object int[] ($y.Count + 1)
        for ($j = 1; $j -le $y.Count; $j++) {
            $cur[$j] = if ($wx -ceq $y[$j - 1]) { $prev[$j - 1] + 1 } else { [math]::Max($prev[$j], $cur[$j - 1]) }
        }
        $prev = $cur
    }
    [math]::Round(2 * $prev[$y.Count] / ($x.Count + $y.Count), 3)
}

function Invoke-Config($cache, $numCtx, $longTarget, $repeats, [bool]$withCode) {
    $tag = "$cache-ctx$numCtx"
    $log = "$outDir\server-$tag.log"
    Say "=== ${tag}: starting ollama serve ==="
    Stop-Ollama
    Start-OllamaServe $cache $log
    $gpuCsv = "$outDir\gpu-$tag.csv"
    $sampler = Start-Process nvidia-smi -ArgumentList '-i 0 --query-gpu=timestamp,power.draw,temperature.gpu,clocks.gr,memory.used --format=csv,noheader,nounits -l 1 -f', "`"$gpuCsv`"" -WindowStyle Hidden -PassThru
    Say '  loading the model'
    $cfg = [ordered]@{ cache_requested = $cache; flash_attention_requested = '1'; num_ctx = $numCtx }
    try {
        $opts = @{ num_ctx = $numCtx; num_predict = 8; temperature = 0; seed = 42 }
        $r = Post-Ollama '/api/generate' @{ model = $Model; prompt = 'Reply with exactly: warm'; stream = $false; keep_alive = '30m'; options = $opts }
        $loaded = @((Invoke-RestMethod ($api + '/api/ps')).models | Where-Object name -eq $Model)[0]
        $cfg.load_s = [math]::Round($r.load_duration / 1e9, 1)
        $cfg.loaded_context = $loaded.context_length
        $cfg.size_gib = [math]::Round($loaded.size / 1GB, 2)
        $cfg.gpu_percent = if ($loaded.size) { [math]::Round(100 * $loaded.size_vram / $loaded.size) } else { $null }
        $cfg.gpu_mem_after_load_mib = Get-GpuMemMiB
        # What llama.cpp actually applied, from the server log (the variable is a request, not a guarantee)
        $applied = @(Select-String -Path $log -Pattern 'cache-type|type_k|type_v|llama_kv_cache|flash_attn|Flash Attention' -ErrorAction SilentlyContinue | ForEach-Object { $_.Line.Trim() })
        $cfg.applied_log_lines = $applied
        $kvLine = $applied | Where-Object { $_ -match 'K \((\w+)\).*V \((\w+)\)' } | Select-Object -Last 1
        if ($kvLine -match 'K \((\w+)\).*V \((\w+)\)') { $cfg.applied_k = $Matches[1]; $cfg.applied_v = $Matches[2] }
        $sizeLine = $applied | Where-Object { $_ -match 'size\s*=\s*([\d.]+)\s*MiB' } | Select-Object -Last 1
        if ($sizeLine -match 'size\s*=\s*([\d.]+)\s*MiB') { $cfg.kv_cache_mib = [double]$Matches[1] }
        Say ("  loaded: {0} GiB, {1}% GPU, context {2}, K={3} V={4}, KV cache {5} MiB, GPU memory {6} MiB" -f $cfg.size_gib, $cfg.gpu_percent, $cfg.loaded_context, $cfg.applied_k, $cfg.applied_v, $cfg.kv_cache_mib, $cfg.gpu_mem_after_load_mib)

        if ($withCode) {
            $cfg.code = foreach ($cp in $codePrompts) {
                $opts = @{ num_ctx = $numCtx; num_predict = 512; temperature = 0; seed = 42 }
                $sw = [Diagnostics.Stopwatch]::StartNew()
                $r = Post-Ollama '/api/generate' @{ model = $Model; prompt = $cp; stream = $false; keep_alive = '30m'; options = $opts }
                $sw.Stop()
                $m = Measure-Reply $r $sw.Elapsed.TotalSeconds; $m.prompt = $cp
                Say ("  code: {0} tok/s" -f $m.decode_tok_s)
                [pscustomobject]$m
            }
            $cfg.code_mean_decode_tok_s = [math]::Round(($cfg.code | Measure-Object decode_tok_s -Average).Average, 2)
        }

        $prompt = if ($longTarget -le 97880) { New-LongPrompt $longTarget } else { New-RepeatedPrompt $longTarget }
        $cfg.long = for ($i = 1; $i -le $repeats; $i++) {
            Unload-All   # a fresh load empties the prompt cache, so the identical prompt is read in full every time
            $opts = @{ num_ctx = $numCtx; num_predict = 384; temperature = 0; seed = 42 }
            $sw = [Diagnostics.Stopwatch]::StartNew()
            $r = Post-Ollama '/api/generate' @{ model = $Model; prompt = $prompt; stream = $false; keep_alive = '30m'; options = $opts }
            $sw.Stop()
            $m = Measure-Reply $r $sw.Elapsed.TotalSeconds
            $m.repeat = $i
            $m.truncated = $r.prompt_eval_count -lt 0.8 * $longTarget
            $m.gpu_mem_mib = Get-GpuMemMiB
            $replyFile = "reply-$tag-r$i.txt"
            [IO.File]::WriteAllText("$outDir\$replyFile", $r.response, (New-Object Text.UTF8Encoding $false))
            $m.reply_file = $replyFile
            Say ("  long {0}: {1} prompt tokens, prefill {2} tok/s ({3} s), decode {4} tok/s, truncated={5}" -f $i, $m.prompt_tokens, $m.prefill_tok_s, $m.prefill_s, $m.decode_tok_s, $m.truncated)
            [pscustomobject]$m
        }
        $cfg.long_mean_decode_tok_s = [math]::Round(($cfg.long | Measure-Object decode_tok_s -Average).Average, 2)
        $cfg.long_mean_prefill_tok_s = [math]::Round(($cfg.long | Measure-Object prefill_tok_s -Average).Average, 1)
    }
    catch {
        $cfg.error = $_.Exception.Message
        Say "  ERROR: $($cfg.error)"
    }
    finally {
        Stop-Process -Id $sampler.Id -ErrorAction SilentlyContinue
        $samples = @(Get-Content $gpuCsv -ErrorAction SilentlyContinue | Where-Object { $_ -match '^\d{4}/' } | ForEach-Object { $f = $_.Split(','); [pscustomobject]@{ w = [double]$f[1]; c = [int]$f[2]; m = [int]$f[3]; mem = [int]$f[4] } })
        if ($samples.Count) {
            $cfg.gpu_peak_mem_mib = ($samples | Measure-Object mem -Maximum).Maximum
            $cfg.gpu_peak_power_w = ($samples | Measure-Object w -Maximum).Maximum
            $cfg.gpu_peak_temp_c = ($samples | Measure-Object c -Maximum).Maximum
        }
    }
    [pscustomobject]$cfg
}

$results = @()
try {
    if (-not $ExtrasOnly) { foreach ($ct in $CacheTypes) { $results += Invoke-Config $ct 102400 97880 $Repeats $true } }
    if (-not $SkipExtras) {
        $results += Invoke-Config 'q8_0' 163840 150000 1 $false
        $results += Invoke-Config 'q4_0' 262144 245000 1 $false
    }
}
finally {
    Stop-Ollama
    Remove-Item Env:OLLAMA_KV_CACHE_TYPE, Env:OLLAMA_FLASH_ATTENTION -ErrorAction SilentlyContinue
    Start-Process "$ollamaDir\ollama app.exe"   # back to the tray app with the user's own settings
    Say 'Restarted the Ollama tray app'
}

# Quality check: every long reply against the first f16 reply (f16's own repeats show the noise floor)
$ref = $results | Where-Object { $_.cache_requested -eq 'f16' -and $_.long } | Select-Object -First 1
$quality = @()
if ($ref) {
    $refText = [IO.File]::ReadAllText("$outDir\$($ref.long[0].reply_file)")
    foreach ($cfg in $results) {
        foreach ($l in @($cfg.long)) {
            if (-not $l) { continue }
            $t = [IO.File]::ReadAllText("$outDir\$($l.reply_file)")
            $quality += [pscustomobject]@{ reply = $l.reply_file; identical_to_f16_r1 = ($t -ceq $refText); similarity_to_f16_r1 = Get-Similarity $refText $t }
        }
    }
}

$battery = Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue | Select-Object -First 1
$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s'); model = $Model
    ollama_version = (& "$ollamaDir\ollama.exe" --version 2>&1 | Out-String).Trim()
    gpu = (& nvidia-smi -i 0 --query-gpu=name,driver_version,memory.total --format=csv,noheader).Trim()
    power_scheme = (powercfg /getactivescheme) -replace '^.*\((.*)\)\s*$', '$1'
    on_ac_power = if ($battery) { $battery.BatteryStatus -eq 2 } else { $null }
    chars_per_token = $CharsPerToken
    configs = $results
    quality = $quality
}
[IO.File]::WriteAllText("$outDir\results.json", ($out | ConvertTo-Json -Depth 8), (New-Object Text.UTF8Encoding $false))
$results | Select-Object cache_requested, applied_k, num_ctx, kv_cache_mib, gpu_peak_mem_mib, code_mean_decode_tok_s, long_mean_prefill_tok_s, long_mean_decode_tok_s, error | Format-Table -AutoSize | Out-String | Write-Host
$quality | Format-Table -AutoSize | Out-String | Write-Host
Say "DONE: $outDir\results.json"
Set-Content "$outDir\DONE" 'done'

Qwen3-Coder 30B on a Surface Laptop Ultra: run it on RTX Spark with Ollama

Steef-Jan Wiggers — Mon, 26 Oct 2026 20:13:28 +0000

My Surface Laptop Ultra decodes short code replies from qwen3-coder:30b at 90.9 tokens a second, twice the 44.7 that Microsoft's Lenovo Yoga 9n review measured for Qwen3.8-27B on the same chip. Both laptops use NVIDIA's RTX Spark N1X with 64 GB of LPDDR5X-9400. I repeated that post's two benchmarks with Ollama on Windows: no Docker, no vLLM, no speculative decoding.

The long prompt tells a different story. With a 100,589-token prompt, qwen3-coder took 146 seconds to its first token and then decoded at 19.7 tokens a second, where the Lenovo setup held 44.3. Over the whole run the GPU averaged 33 W and peaked at 75 °C.

At a glance

Verdict: with Ollama alone, the Surface Laptop Ultra is a fast machine for short, interactive coding requests with a local model. For a coding agent that resends a 100K-token conversation on every turn, the Lenovo post's vLLM setup is the better pattern.

What I liked:

Setup is one signed installer and one ollama pull. Nothing runs in WSL or Docker.
qwen3-coder:30b (30.5B parameters, 8 of 128 experts active per token) and a 102,400-token context fit entirely on the GPU: 26.7 GiB.
90.9 tokens a second on short code prompts, steady across three runs (88–92).
qwen3:8b, the smaller model, decodes at 44–50 tokens a second in every run I made.

Good to know:

Decode speed falls about 4.6 times between a 20-token and a 100K-token prompt, because the attention cache grows with every token.
Prompt processing ran at 687 tokens a second at 100K tokens, two-thirds of the Lenovo setup's 1,033.
Ollama silently cuts a prompt longer than its context window to about half. Only its server log says so.

Who it's for: developers who want a local coding model for chat, edits and short agent turns, with no per-token bill and no container stack.

The machine

The Surface Laptop Ultra has the same chip and memory as the Lenovo in Microsoft's post, so the hardware comparison is like for like.

	Microsoft Surface Laptop Ultra, as tested
Chip	NVIDIA RTX Spark N1X: 20-core Arm CPU, 6,144-core Blackwell GPU, plus an NVIDIA NPU
Memory	64 GB of Samsung LPDDR5X-9400, shared by the CPU and the GPU
GPU's share	31,232 MiB dedicated to the GPU; Windows sees the other 32.2 GB
Software	Windows 11 Pro on Arm (build 28000), NVIDIA driver 617.42, Ollama 0.40.11 (native Windows ARM64, per-user install)
Power	On AC power, Windows Balanced power plan
Models	qwen3-coder:30b and qwen3:8b, both Q4_K_M, stored on the internal SSD

The GPU's share is the same 32 GB split the Lenovo post used. With qwen3-coder and its 102,400-token cache loaded, GPU memory peaked at 29,952 of 31,232 MiB.

How I measured

I copied the Lenovo post's two single-request benchmarks as closely as Ollama allows; the agent session and Azure Monitor parts I left out.

	Code benchmark	Long-prompt benchmark
Task	A thread-safe LRU cache in Python with tests, a token-bucket rate limiter in TypeScript, Dijkstra's algorithm in Go	One question after a long prompt cut from Windows PowerShell's own format and module files
Prompt	18–21 tokens	100,589 tokens (qwen3-coder) or 37,151 (qwen3:8b, whose limit is 40,960), with a random first line so nothing came from the prompt cache
Reply	512 tokens each, temperature 0, seed 42, thinking off	384 tokens, temperature 0, thinking off
Context	102,400 tokens (qwen3-coder), 40,960 (qwen3:8b)	Same
How it was sent	One request at a time to Ollama's `/api/generate`, after a warm-up load	One request, after a calibration prompt that measured characters per token

The script, speed-bench.ps1, unloads every model first, records Ollama's own token timings and samples GPU power, temperature, clock and memory with nvidia-smi once a second. Decode speed is Ollama's eval_count / eval_duration; prompt speed is prompt_eval_count / prompt_eval_duration.

One lesson from the first attempt: my first long prompt for qwen3-coder came out at 105,530 tokens, over its 102,400-token window, and Ollama kept only the last 51,202 without returning an error. The script now calibrates on a large slice of the text and flags any prompt that comes back short.

Results

On short prompts, qwen3-coder:30b is the fastest setup on either laptop; at 100K tokens it is the slowest.

The three code prompts landed close together: 92.1, 90.2 and 90.5 tokens a second for qwen3-coder, and 46.5, 45.1 and 43.8 for qwen3:8b. The first token of a short reply arrived in 0.08–0.19 seconds once the model was loaded.

Reading the long prompt is where the wait comes from:

Setup	Prompt	Prompt processing	First token
qwen3-coder:30b, Ollama	100,589 tokens	687 tok/s	146 s
qwen3:8b, Ollama	37,151 tokens	1,492 tok/s	25 s
Qwen3.8-27B, vLLM on the Lenovo	97,880 tokens	1,033 tok/s	about 95 s, worked out from its prompt speed

For a chat that grows turn by turn, Ollama reuses the cached start of the conversation, so only the new part is read. The 146 seconds is the cost of a brand-new 100K-token prompt.

Power, heat and clocks

The GPU slows its clock while it reads a long prompt and speeds back up to reply, so the lower clock tracks the work, not heat building up.

Reading a prompt is compute-bound: after a short burst to 60 W, the GPU settled near 34 W and dropped its clock to about 1.65 GHz for the whole 146 seconds. Replying is memory-bound: the clock went back up to about 2.25 GHz at 30 W, and the temperature fell from 71 to 64 °C. That pattern looks like a power limit, not thermal throttling.

The 75 °C peak is 8 °C above the Lenovo's three-hour peak, reached in a run of about four minutes. I haven't tested sustained load yet, so treat this as a first look at the Surface's cooling, not a verdict.

Why long context costs so much here

Decode on this chip is a question of bytes, as the Lenovo post showed: every token reads the active weights plus the whole attention cache, at roughly 235 GB/s.

tokens per second ≈ memory bandwidth ÷ bytes read per token

qwen3-coder's mixture of experts makes the weights cheap: only about 3.3B of its 30.5B parameters run per token, about 2 GB at Q4_K_M. Its cache is the expensive part. Each token keeps keys and values for 48 layers × 4 heads × 128 dimensions in 16-bit floats, 96 KiB a token, so 100,000 tokens of cache is about 9.9 GB.

Run	Weights read per token	Cache read per token	Measured decode	Implied bandwidth
qwen3-coder:30b, short prompt	~2.0 GB	~0.05 GB	90.9 tok/s	~185 GB/s
qwen3-coder:30b, 100,589-token prompt	~2.0 GB	~9.9 GB	19.7 tok/s	~235 GB/s
qwen3:8b, short prompt	~5.1 GB	~0.08 GB	45.1 tok/s	~235 GB/s
qwen3:8b, 37,151-token prompt	~5.1 GB	~5.5 GB	22.3 tok/s	~235 GB/s

Three of the four runs imply the same 235 GB/s, close to the 229 GB/s the Lenovo post measured with a read test. The short qwen3-coder run falls short of it because routing tokens between experts adds work that isn't memory reads.

The weights and cache sizes are my estimates from the model files and their architecture, not measurements. They explain the gap with the Lenovo setup: Qwen3.8-27B keeps a fixed-size state in 48 of its 64 layers, so its cache barely grows, and speculative decoding produces about five tokens per read of the weights.

Laptop vs laptop

The hardware matches, so the differences come from the software stack and the model. Read this as two working setups, not a controlled benchmark.

	Surface Laptop Ultra (this post)	Lenovo Yoga 9n 2-in-1 (Microsoft's post)
Chip and memory	RTX Spark N1X, 64 GB LPDDR5X-9400, 32 GB to the GPU	Same
Model	qwen3-coder:30b, mixture of experts, ~3.3B active	Qwen3.8-27B, dense, hybrid attention
Weights	Q4_K_M, 17.3 GiB	NVFP4, 19.1 GB
Server	Ollama 0.40.11, native on Windows	vLLM 0.31 in Docker Desktop on WSL 2
Speculative decoding	None	DFlash 2 drafter, 7 tokens
Context used	102,400 tokens (the model supports 262,144)	262,144 tokens
Setup effort	One installer, one model pull	Docker, a custom image, WSL memory caps, LiteLLM
Monitoring	nvidia-smi samples to CSV	Azure Monitor, Managed Grafana, alert rules
Best for	Chat, edits and short agent turns	Long agent sessions with 100K+ token conversations

The Lenovo post also ran three hours of OpenCode traffic at about 27 tokens a second per request. I haven't run an agent session on the Surface yet, so there is no matching number here.

What runs where, and how to set it up

The whole stack is one Windows service in front of the GPU; there is no container, gateway, or collector to run.

Ollama starts a llama.cpp server per loaded model and offloads every layer to the GPU. The GPU already had a 32 GB share of the unified memory; Microsoft's post says the NVIDIA app's Dedicated GPU Memory setting controls it.

Check the GPU: nvidia-smi should list the RTX Spark N1X and its memory.
Install Ollama with its signed script: irm https://ollama.com/install.ps1 | iex. It installs per user, with no admin rights needed.
Pull the models: ollama pull qwen3-coder:30b (17.3 GiB) and ollama pull qwen3:8b (4.9 GiB). Keep them on the internal SSD.
Set a default context for editors: the user variable OLLAMA_CONTEXT_LENGTH=65536, then restart Ollama. The benchmark sets its own context per request.
Run the benchmark from the folder that holds it: powershell -ExecutionPolicy Bypass -File .\speed-bench.ps1 (see PowerShell at bottom of the post), or add -Models qwen3-coder:30b for one model. Running both models takes about seven minutes.
Point VS Code at it: Continue's config.yaml with provider: ollama, or the Ollama provider in Copilot Chat's language models.

Rules of thumb

Count bytes per token. On RTX Spark, decode speed is about 235 GB/s divided by the weights and cache each token reads.
Pick a mixture-of-experts model for short turns. Reading 2 GB of weights per token instead of 5 GB is what doubles qwen3-coder's speed over qwen3:8b.
Budget for the cache at long context. At 100K tokens, qwen3-coder's 16-bit cache is five times the size of its active weights.
Check the prompt count. If prompt_eval_count comes back far below what you sent, Ollama cut your prompt to fit the context window.
Watch the clock, not just the temperature. During prompt processing, the GPU held about 1.65 GHz at 34 W; during replies, it ran at 2.25 GHz.

The bottom line

The Surface Laptop Ultra has the same chip and memory as the Lenovo in Microsoft's post, so the same models and limits apply. With nothing more than Ollama, qwen3-coder:30b answers short coding requests at 91 tokens a second, with the first token in under 0.2 seconds.

For 100K-token agent sessions, Ollama on this machine is the weak spot: a fresh prompt waits more than two minutes and replies drop to 20 tokens a second. Next, I want to try an 8-bit KV cache (OLLAMA_KV_CACHE_TYPE=q8_0), which should roughly halve the cache's bytes, and a three-hour agent session to compare with the Lenovo's 27 tokens a second.

Powershell:

# Ollama version of the speed benchmark from Microsoft's "Qwen3.8-27B on a Lenovo laptop" post:
#  - code benchmark: three code-generation prompts, 512-token replies, temperature 0, thinking off, one at a time
#  - long-prompt benchmark: one question after a long prompt whose random first line defeats the prefix cache, 384-token reply
# GPU power, temperature, clock and memory are sampled every second with nvidia-smi during the whole run.
# Writes speed-bench-<COMPUTER>-<stamp>.json and .gpu.csv in the current directory.
param(
    [string[]]$Models = @('qwen3:8b', 'qwen3-coder:30b'),
    [string]$Api = 'http://localhost:11434'
)
$ErrorActionPreference = 'Stop'

# Context per model: big enough for the long prompt plus reply, within the model's maximum and the GPU's memory
$plan = @{
    'qwen3:8b'        = @{ num_ctx = 40960;  long_tokens = 36000 }   # qwen3:8b's trained maximum is 40,960
    'qwen3-coder:30b' = @{ num_ctx = 102400; long_tokens = 97880 }   # the article's long prompt was 97,880 tokens
}
$codePrompts = @(
    'Write a thread-safe LRU cache in Python with unit tests.',
    'Write a token-bucket rate limiter in TypeScript.',
    "Implement Dijkstra's shortest-path algorithm in Go."
)

function Post-Ollama($path, $body) {
    Invoke-RestMethod -Method Post -Uri ($Api + $path) -ContentType 'application/json' -Body ($body | ConvertTo-Json -Depth 8) -TimeoutSec 1800
}
function Unload-All {
    foreach ($m in @((Invoke-RestMethod ($Api + '/api/ps')).models)) { Post-Ollama '/api/generate' @{ model = $m.name; keep_alive = 0 } | Out-Null }
    Start-Sleep 3
}
function Measure-Reply($r, $wall) {
    [ordered]@{
        prompt_tokens       = $r.prompt_eval_count
        prefill_tok_s       = if ($r.prompt_eval_duration) { [math]::Round($r.prompt_eval_count / ($r.prompt_eval_duration / 1e9), 1) } else { $null }
        output_tokens       = $r.eval_count
        decode_tok_s        = if ($r.eval_duration) { [math]::Round($r.eval_count / ($r.eval_duration / 1e9), 2) } else { $null }
        first_token_s       = [math]::Round(($r.load_duration + $r.prompt_eval_duration) / 1e9, 2)
        wall_s              = [math]::Round($wall, 2)
        done_reason         = $r.done_reason
    }
}
function Invoke-Timed($body) {
    $sw = [Diagnostics.Stopwatch]::StartNew(); $r = Post-Ollama '/api/generate' $body; $sw.Stop()
    return $r, $sw.Elapsed.TotalSeconds
}

# Text for the long prompt: PowerShell's own format/type files and module sources, present on every Windows machine
$corpus = (Get-ChildItem "$env:windir\System32\WindowsPowerShell\v1.0" -Recurse -File -Include *.ps1xml, *.psm1, *.psd1 -ErrorAction SilentlyContinue |
    Sort-Object FullName | ForEach-Object { "`n### FILE: $($_.Name)`n" + [IO.File]::ReadAllText($_.FullName) }) -join ''

$stamp = Get-Date -Format 'yyyyMMdd-HHmmss'
$base = Join-Path (Get-Location) "speed-bench-$env:COMPUTERNAME-$stamp"
$gpuCsv = "$base.gpu.csv"
$gpuInfo = (& nvidia-smi -i 0 --query-gpu=name,driver_version,memory.total --format=csv,noheader).Trim()
$sampler = Start-Process nvidia-smi -ArgumentList '-i 0 --query-gpu=timestamp,power.draw,temperature.gpu,clocks.gr,memory.used --format=csv,noheader,nounits -l 1 -f', "`"$gpuCsv`"" -WindowStyle Hidden -PassThru
$phases = @()
$results = @()

try {
    foreach ($model in $Models) {
        $p = $plan[$model]
        if (-not $p) { $p = @{ num_ctx = 32768; long_tokens = 30000 } }
        $caps = @((Post-Ollama '/api/show' @{ model = $model }).capabilities)
        $opts = { param($n) @{ num_ctx = $p.num_ctx; num_predict = $n; temperature = 0; seed = 42 } }
        function New-Body($prompt, $n) {
            $b = @{ model = $model; prompt = $prompt; stream = $false; keep_alive = '15m'; options = (& $opts $n) }
            if ($caps -contains 'thinking') { $b.think = $false }   # thinking off, as in the article; non-thinking models reject the flag
            $b
        }

        Write-Host "=== $model (num_ctx $($p.num_ctx)) ==="
        Unload-All
        $t0 = Get-Date
        $r, $w = Invoke-Timed (New-Body 'Reply with exactly: warm' 8)
        $loaded = @((Invoke-RestMethod ($Api + '/api/ps')).models | Where-Object name -eq $model)[0]
        $phases += [pscustomobject]@{ model = $model; phase = 'load'; start = $t0; end = Get-Date }
        $modelInfo = [ordered]@{
            model = $model; num_ctx = $p.num_ctx; loaded_context = $loaded.context_length
            load_s = [math]::Round($r.load_duration / 1e9, 1)
            size_gib = [math]::Round($loaded.size / 1GB, 1)
            gpu_percent = if ($loaded.size) { [math]::Round(100 * $loaded.size_vram / $loaded.size) } else { $null }
        }

        # Code benchmark
        $t0 = Get-Date
        $code = foreach ($cp in $codePrompts) {
            $r, $w = Invoke-Timed (New-Body $cp 512)
            $m = Measure-Reply $r $w; $m.prompt = $cp
            Write-Host ("  code: {0} tok/s decode, {1} tokens" -f $m.decode_tok_s, $m.output_tokens)
            [pscustomobject]$m
        }
        $phases += [pscustomobject]@{ model = $model; phase = 'code'; start = $t0; end = Get-Date }

        # Long-prompt benchmark: calibrate characters per token, then build the prompt just under the target
        # A large slice: token density varies a lot between files, and Ollama silently cuts prompts longer than num_ctx
        $sample = $corpus.Substring(0, [math]::Min($p.num_ctx * 3, $corpus.Length))
        $r, $w = Invoke-Timed (New-Body ("Calibration $(Get-Random)`n" + $sample) 1)
        $charsPerToken = $sample.Length / $r.prompt_eval_count
        $chars = [int]($p.long_tokens * $charsPerToken * 0.97)
        $text = $corpus
        while ($text.Length -lt $chars) { $text += $corpus }
        $question = "`n`nQuestion: Describe what the files above define, as a numbered list covering every file."
        $longPrompt = "Session $([guid]::NewGuid())`n" + $text.Substring(0, $chars) + $question
        Start-Sleep 5
        $t0 = Get-Date
        $r, $w = Invoke-Timed (New-Body $longPrompt 384)
        $long = Measure-Reply $r $w
        $long.chars_per_token = [math]::Round($charsPerToken, 2)
        # Ollama keeps only the end of an over-long prompt (about half the context); a short count means that happened
        $long.truncated = $r.prompt_eval_count -lt 0.8 * $p.long_tokens
        if ($long.truncated) { Write-Warning "$model long prompt was truncated to $($r.prompt_eval_count) tokens" }
        $phases += [pscustomobject]@{ model = $model; phase = 'long'; start = $t0; end = Get-Date }
        Write-Host ("  long: {0} prompt tokens, {1} tok/s prefill, {2} tok/s decode, {3} s to first token" -f $long.prompt_tokens, $long.prefill_tok_s, $long.decode_tok_s, $long.first_token_s)

        $modelInfo.code = $code
        $modelInfo.code_mean_decode_tok_s = [math]::Round(($code | Measure-Object decode_tok_s -Average).Average, 2)
        $modelInfo.long = [pscustomobject]$long
        $results += [pscustomobject]$modelInfo
        Start-Sleep 5
    }
}
finally {
    Stop-Process -Id $sampler.Id -ErrorAction SilentlyContinue
    Unload-All
}

# GPU summary per phase from the nvidia-smi samples
$samples = Get-Content $gpuCsv | Where-Object { $_ -match '^\d{4}/' } | ForEach-Object {
    $f = $_.Split(',') | ForEach-Object { $_.Trim() }
    [pscustomobject]@{
        time = [datetime]::ParseExact($f[0], 'yyyy/MM/dd HH:mm:ss.fff', $null)
        power_w = [double]$f[1]; temp_c = [int]$f[2]; clock_mhz = [int]$f[3]; mem_mib = [int]$f[4]
    }
}
$gpuPhases = foreach ($ph in $phases) {
    $s = @($samples | Where-Object { $_.time -ge $ph.start -and $_.time -le $ph.end })
    if (-not $s.Count) { continue }
    [pscustomobject][ordered]@{
        model = $ph.model; phase = $ph.phase; seconds = [math]::Round(($ph.end - $ph.start).TotalSeconds, 1)
        avg_power_w = [math]::Round(($s | Measure-Object power_w -Average).Average, 1)
        max_power_w = ($s | Measure-Object power_w -Maximum).Maximum
        max_temp_c = ($s | Measure-Object temp_c -Maximum).Maximum
        avg_clock_mhz = [math]::Round(($s | Measure-Object clock_mhz -Average).Average)
        max_mem_mib = ($s | Measure-Object mem_mib -Maximum).Maximum
    }
}

$battery = Get-CimInstance Win32_Battery -ErrorAction SilentlyContinue | Select-Object -First 1
$out = [ordered]@{
    computer = $env:COMPUTERNAME; date = (Get-Date).ToString('s')
    ollama_version = (Invoke-RestMethod ($Api + '/api/version')).version
    gpu = $gpuInfo
    power_scheme = (powercfg /getactivescheme) -replace '^.*\((.*)\)\s*$', '$1'
    on_ac_power = if ($battery) { $battery.BatteryStatus -eq 2 } else { $null }
    models = $results
    gpu_phases = $gpuPhases
}
$json = $out | ConvertTo-Json -Depth 8
[IO.File]::WriteAllText("$base.json", $json, (New-Object Text.UTF8Encoding $false))
$gpuPhases | Format-Table -AutoSize | Out-String | Write-Host
Write-Host "Results: $base.json"
Write-Host "GPU samples: $gpuCsv"

Intent-Sized vs Resource-Sized, Measured

Steef-Jan Wiggers — Thu, 22 Oct 2026 06:16:38 +0000

Every MCP design discussion eventually reaches the granularity debate. Should a server expose intent-sized tools, one per thing a user wants done, or resource-sized tools, one per operation on each resource, the shape an OpenAPI-to-MCP generator produces from an existing REST API? The arguments are usually aesthetic. This series settles things by measuring, so the companion repo now has measure/eval_granularity.py: the same restaurant kitchen rendered both ways, and a live model asked, for twelve user requests, which tool it would call first on each full surface.

The intent surface is the sample's three tools: search_restaurants, get_menu, place_order. The resource surface is the seven tools a generator would derive from the REST door: list_restaurants, get_restaurant, list_menu_items, get_menu_item, create_order, get_order, list_orders, with the neutral CRUD descriptions generated surfaces actually have. Same kitchen, same capability, two vocabularies.

The fixed costs

The tokenizer from What the Agent Pays for Discovery prices the two contracts:

surface	tools	tokens per model turn
intent	3	302
resource	7	394

A 30 percent toll on every single model turn, paid before anything is called, forever. That is the budget argument for intent-sized tools, and it compounds: agents make several turns per task, and fleets carry many servers.

The selection results

Twelve requests, three repeats each, gpt-5.4-mini choosing its first call from the full surface:

surface	kind	selected	args ok
intent	direct	66.7%	66.7%
intent	chained	100.0%	100.0%
resource	direct	94.4%	94.4%
resource	chained	75.0%	75.0%

Read naively, that table says each surface wins half the board. Read the actual misses and it says something much more interesting.

The intent surface's misses are obedience

Every single intent-surface miss, six out of six, is the same event: given "Order two Margherita at restaurant r1", with every id it needs in hand, the model chose get_menu instead of place_order. All three repeats, both order tasks.

That is not confusion. That is the model doing what the contract told it to do. place_order's description, the one that went through the CI lint gate, says "Use it only after get_menu has confirmed the exact item name", and the model obeyed it over the shortcut the benchmark scored as correct. The harness's strict scoring calls that a 33 percent failure; a production incident review would call it the safety sentence working. The sentence exists because the confusion telemetry showed agents inventing item names; the model reading it and checking the menu first is the intended behavior, deliberately purchased, in production the difference between an order and an order_rejected sentinel.

The resource surface's misses are the dangerous kind

The resource misses are a different species. "Get the menu for restaurant r1" once went to get_restaurant instead of list_menu_items, ordinary sibling confusion in a CRUD family where four tools differ by one noun. But "Order a Stamppot from De Molen Bistro in Nijmegen" missed all three repeats, and twice the miss was create_order called directly, with a restaurant name where an identifier belongs. That is the exact name-as-id failure that Watching the Contract Fail in Production counted in live telemetry, and the resource surface invites it because generated CRUD descriptions carry no provenance sentences. Nothing in "Creates a new order resource" says where a restaurantId comes from, so the model improvises one. The intent surface went 100 percent on these same natural-language requests, steered by the sentences "as returned by search_restaurants" and "call it after search_restaurants".

Notice the symmetry. On direct order tasks the resource surface was happy to create_order immediately, which the scoring rewards, because no sentence told it to verify anything first. The intent surface hesitated, because a sentence did. Same model, same requests; the behavior difference is entirely the prose.

Where resource-sized honestly wins

The gap probes give the resource surface its real victory. "Show me the details of restaurant r2" and "Did my order f774f41d go through?" have no correct tool on the intent surface, and the model did not say so: all six repeats it called search_restaurants, a plausible-looking wrong call that would send an agent looping. The resource surface answered both perfectly, get_restaurant and get_order, three for three each. Resource-shaped surfaces inherit the coverage of the API they were generated from, and coverage gaps on an intent surface do not fail loudly, they fail as confident wrong calls. If your users ask about order status, no amount of description quality rescues a surface that has no tool for it.

The takeaway

The granularity debate turns out to be a proxy war, and the real fight is the one this series has been measuring all along: who writes the prose, and what does the surface cover. Intent-sized tools won the chained tasks not because three is a better number than seven, but because their descriptions carry provenance and sequencing sentences that a generator does not write. Resource-sized tools won the gap tasks not because CRUD is a better shape, but because they cover more of the backend. Both results are about contract quality, not tool count.

So the practical guidance is not "pick a granularity". It is: start from coverage, because a missing tool fails as a confident wrong call; pay the token toll only for operations agents actually need, because every tool costs every turn; and whatever shape you end with, write the provenance and when-to-call sentences into it, because they are what moved behavior in every measurement this series has made. A generated resource surface with linted prose would likely beat both of the surfaces measured here. That is a falsifiable claim, the harness takes arbitrary surfaces, and I would genuinely like someone to run it against mine.

The harness, both surfaces, and the full per-row results are in the companion repo: measure/eval_granularity.py and measure/eval_granularity_results.json.

The Embedding Rematch

Steef-Jan Wiggers — Wed, 14 Oct 2026 02:15:28 +0000

The Shortlist Decides First ended with a caveat I promised to resolve: the retriever in that experiment was BM25, a lexical matcher, and its catastrophic 5 percent recall on paraphrased requests was exactly the failure lexical matching should produce. Would a semantic retriever change the verdict? That is a measurable question, so here is the rematch: the same 100 tools, the same 200 tasks, the same four description conditions and k-sweep, with one change. Instead of BM25, the retriever embeds every tool document and every request with text-embedding-3-small on Azure OpenAI and ranks by cosine similarity. The harness is measure/eval_recall_embed.py in the companion repo; nothing else moved.

The results

Recall of the correct tool into the shortlist, by description condition and task kind:

condition	vocab @3	vocab @5	vocab @10	para @3	para @5	para @10
terse	95%	99%	100%	59%	73%	87%
realistic	91%	98%	100%	59%	76%	94%
verbose	97%	99%	100%	65%	82%	98%
realistic + aliases	98%	99%	100%	99%	100%	100%

Three findings, one win, one surprise, one standing verdict.

The win: embeddings rescue paraphrase

BM25 scored 5 percent on paraphrased requests at k=5. Embeddings score 76 percent on the same tasks with the same descriptions. That is the rematch headline, and it lands exactly where the theory says it should: a user who says "I want to cancel my booking" shares almost no tokens with a tool described as "deletes a reservation", but the two sentences live close together in embedding space.

The k-sweep behaves differently now too, and this is worth noticing. With BM25, widening the shortlist from 3 to 10 bought paraphrase recall from 3 to 10 percent, because the correct tool usually was not even near the top of the ranking. With embeddings, the same widening buys 59 to 94 percent. Semantic misses are near misses; lexical misses were not misses at all, they were absences. When your retriever is at least semantically in the neighborhood, paying for a longer shortlist actually purchases something.

The surprise: verbosity finally pays, and exactness costs

The shortlist post concluded that verbosity is not coverage, because doubling description tokens moved BM25's paraphrase recall not at all. The rematch forces me to refine that: it is true lexically and less true semantically. Under embeddings, verbose descriptions beat terse ones on paraphrase at every k, 82 against 73 at k=5, because every extra sentence adds semantic surface for the query vector to land on. Verbosity is still a poor deal, you pay roughly triple the tokens per tool for nine points, but it is no longer a zero.

The reverse surprise sits in the vocabulary columns. On requests that echo the tool's own vocabulary, BM25 scored 97 to 100 percent at k=3; embeddings score 91 to 97. Embeddings blur exact matches. When the user says precisely the words in the description, nothing beats matching those words literally, and a semantic retriever occasionally ranks a similar-sounding sibling above the exact hit. The two retrievers fail on opposite task populations, which is the textbook argument for hybrid retrieval, and now the series has the numbers for it.

The standing verdict: the alias sentence still wins

Look at the last row. With the 13-token alias sentence per tool, recall is 99 to 100 percent everywhere, at every k, and it was also 100 percent under BM25. The cheapest intervention in the entire series makes the retriever choice irrelevant.

That reframes what an embedding index buys you. If you control the tool descriptions, the alias sentence gets you to perfect recall for about 13 tokens per tool and zero infrastructure. The embedding pipeline, an embedding model, an index, a lookup on every request, buys you from 5 to 76 percent on paraphrase when you cannot touch the descriptions, which is precisely the situation of a gateway or catalog fronting other teams' MCP servers. Both answers are right; they answer different ownership questions.

The end-to-end multiplication from The Model Was Never the Problem updates accordingly. Recall times selection on paraphrased requests with realistic descriptions: BM25 gave 5 percent times 93 percent, roughly 5 percent of tasks ending in the right tool call. Embeddings give 76 times 93, roughly 71 percent. Aliases give 100 times 97, roughly 97, with either retriever underneath.

Honest limits

One embedding model, text-embedding-3-small; a larger model or a hybrid lexical-plus-vector ranker would likely narrow both gaps, and the harness takes any Azure OpenAI embedding deployment if you want to try. The paraphrases are template-generated, which makes them systematic but tamer than real user phrasing. The catalog is 100 tools; at 1,000 the embedding index gets more attractive because the alias discipline gets harder to enforce across teams. And this measures retrieval in isolation; the selection stage was measured separately and multiplies on top.

The takeaway

The rematch confirms the retriever was the right suspect and the description was still the right fix. Embeddings turn paraphrase recall from a catastrophe into a tax, 5 to 76 percent, and they are the honest recommendation when the descriptions belong to someone else. But when the descriptions are yours, one alias sentence per tool beats the entire vector pipeline, at one ten-thousandth of the operational surface. Fix the words first. Add the index when the words stop being yours to fix.

The harness, both retrievers, and the full results are in the companion repo: measure/eval_recall.py for BM25, measure/eval_recall_embed.py for the rematch.

The Third Leg of the Parity Test

Steef-Jan Wiggers — Wed, 07 Oct 2026 12:31:29 +0000

Months ago, a commenter on this series proposed a test: if the same backend really stands behind both doors, then auth, error behavior, and idempotency must be preserved across them. Authorization Stays in the Kitchen settled the first leg. The confusion telemetry and recovery messages settled the second. This post settles the third, and it turns out to be the leg that matters most for agents, because agents retry.

Why agents make this urgent

Code retries when a developer writes a retry policy. Agents retry on their own judgment: a timeout, an ambiguous error, a truncated response, and the model decides the call probably failed and tries again. Nothing in the MCP protocol stops the second call from being a second order. The new driver in the companion repo proves the failure mode as a control case before proving the fix:

3. retry without clientOrderId    PASS: orderId 9714f45b then 99523aed (two orders exist, as feared)

Same restaurant, same item, same quantity, two calls, two orders. Every production incident involving an agent and a non-idempotent write looks like this line.

One decision, one place, again

The series rule holds: neither door contains business logic, and deduplication is business logic. The kitchen keeps a store keyed on the caller's idempotency key, remembers a fingerprint of the request alongside the original confirmation, and answers one of four ways: placed, replayed, conflict, or invalid.

if (idempotencyKey is not null
    && _byIdempotencyKey.TryGetValue(idempotencyKey, out var seen))
{
    return seen.Fingerprint == Fingerprint(request)
        ? new OrderResult(OrderOutcome.Replayed, seen.Confirmation)
        : new OrderResult(OrderOutcome.Conflict, null);
}

A replayed key with the same request returns the original confirmation and creates nothing. The same key with a different request is a conflict, which is always a caller bug and deserves to be refused loudly. The doors only translate those outcomes into their caller's dialect, exactly as they did for authorization.

The REST door speaks header

HTTP already has a convention for this: the Idempotency-Key request header, specified in an IETF httpapi draft that Stripe, Adyen, and most payment APIs shaped in practice. The draft says a completed operation replays its original result and a key reused with a different payload SHOULD get a 422, so that is what the door does.

Verified live, the same curl twice:

HTTP/1.1 200 OK
{"orderId":"97e92454","restaurantName":"Trattoria Valkhof","itemName":"Margherita","quantity":2,"total":23.00,"status":"confirmed"}

HTTP/1.1 200 OK
{"orderId":"97e92454","restaurantName":"Trattoria Valkhof","itemName":"Margherita","quantity":2,"total":23.00,"status":"confirmed"}

Same orderId, one order, and the client cannot tell the replay from the original, which is the whole point. Then the same key with quantity 5:

HTTP/1.1 422 Unprocessable Entity
{"error":"idempotency_conflict","detail":"This Idempotency-Key was already used for a different request. Send a new key for a new order."}
The MCP door has no headers

Here the two doors genuinely differ, and the difference is instructive. A tool call carries a name and arguments, nothing else, so the idempotency key cannot ride in a header. It has to be an argument, which means it has to be in the contract, which means a sentence has to teach the agent to use it:

[McpToolProperty("clientOrderId", "An order reference that comes from you, the caller: invent a unique value, and reuse the same value if you retry this order after an error or timeout.", isRequired: false)]
    string? clientOrderId

That sentence is the mechanism. On the REST door, idempotency works because a developer read the docs and wrote a header into their client. On the MCP door, it works only if the description convinces the model to invent a reference and hold onto it across a retry. There is no SDK to do it for the agent; the prose is the SDK.

The series tooling priced and policed the sentence immediately. The contract linter flagged clientOrderId as an identifier parameter the moment it existed, so the description had to say where the value comes from, and this parameter gave the provenance rule its first unusual answer: it comes from you. The token budget rule priced the addition at 42 tokens, taking the contract from 302 to 344 against a budget of 350, so the next contract sentence will have to pay for itself by shortening another. And because the deployed description hash is computed from these same attributes, the descHash changed, and the confusion telemetry can attribute any behavior shift to this exact wording.

The driver verified both halves end to end against the deployed server:

1. retry with same clientOrderId  PASS: orderId d120204a then d120204a (one order exists)
2. same key, different request    PASS: The clientOrderId 'order-2ccc9bd3' was already used for a different order.

The conflict answer is recovery prose, as this door's denials always are, and it goes out through the telemetry helper as a new order_conflict sentinel, so key misuse is observable next to every other contract failure. Replays log their own trace, which gives you a retry rate for free: how often agents actually double-fire is now a query, not a guess.

Honest limits

The store is in-memory, like everything in this sample, so the replay guarantee spans one instance and one process lifetime. A production version needs a durable store with a TTL, and on a plan that scales out, a shared one. The guarantee is also only as good as the agent's discipline: an agent that invents a fresh clientOrderId on every retry gets no protection, and nothing in the protocol forces it to comply. The description can only persuade. That makes idempotency on the MCP door a probabilistic defense in a way it never was for REST clients, and a good candidate for the eval harness to measure someday: given a simulated timeout, does the model actually reuse the key the description told it to reuse?

The takeaway

The parity test now passes on all three legs, and each leg landed the same way: one decision in the kitchen, one dialect per door. For idempotency the dialects are a standard header on one side and a contract sentence on the other, and the asymmetry is the lesson. REST made retries safe with infrastructure; MCP has to make them safe with language. If your tools write anything that costs money, the deduplication store is the easy half. The sentence that gets the agent to use it is the half this series keeps measuring.

The kitchen policy, both doors, and the double-fire driver are in the companion repo; run measure/drive_idempotency.py against your own server and see whether check 3 scares you as much as it should.

Authorization Stays in the Kitchen

Steef-Jan Wiggers — Fri, 02 Oct 2026 09:50:06 +0000

What a 401 Means to an MCP Client ended on a deliberately unfinished note: the token asserts who the caller is, it does not decide what they may do once inside. That sentence has been an IOU ever since. This post pays it, with claims-based authorization implemented in the sample, the same denial rendered as a 403 JSON body on one door and as recovery prose on the other, and every state verified live, including two detours that taught me more than the happy path did.

One decision, one place

The design rule from the first post was that neither door contains business logic, and authorization is business logic. So the policy lives in the kitchen:

public static class OrderPolicy
{
    public const string RequiredRole = "Orders.Place";

    public static bool MayPlaceOrders(CallerContext caller) =>
        caller.Roles.Any(r => string.Equals(r, RequiredRole, StringComparison.OrdinalIgnoreCase));
}

Anyone may search restaurants and read menus; placing an order requires the Orders.Place role. The doors do exactly two things: gather identity however their protocol allows, and render the kitchen's decision in their caller's dialect. The REST door answers code, so it returns a machine-shaped refusal:

{"error":"forbidden","detail":"Placing orders requires the Orders.Place role.","requiredRole":"Orders.Place"}

The MCP door answers an agent, so it returns an instruction it can act on: "This caller is not allowed to place orders; that requires the Orders.Place role. Searching restaurants and reading menus remain available." And because that sentence goes out through the telemetry helper from the confusion-telemetry work, every denial is also an attributable order_forbidden row in Application Insights.

The REST door: a sentence in a token

App Service authentication validates the bearer token and hands the function the caller's claims in the X-MS-CLIENT-PRINCIPAL header; the door parses out the roles and asks the kitchen. Verifying it took one Entra app role (value Orders.Place on the restaurant-mcp-auth registration), one role assignment to my own account, and produced the cleanest before-and-after in this series: the same curl, the same body, the same endpoint, a 403 with the role named in it, and then an order confirmation for 23 euros. The only thing that changed between the two was one sentence inside the token.

Getting that sentence took two detours worth telling. First, the role assignment needs an enterprise application to attach to, and the portal's one-click MCP auth button had created the app registration without a service principal; the "Managed application in local directory: Create Service Principal" link on the registration's overview was the missing step, and until it was clicked, the app simply did not exist in the Enterprise applications list to assign anyone to.

Second, and more operationally important: after the assignment, the token stubbornly kept returning no roles. Not propagation, not misconfiguration, but two layers of caching, the Azure CLI reusing its still-valid cached token, and, more fundamentally, the fact that claims are minted at token issuance, not looked up live. A role you grant right now reaches a caller only when they next acquire a token. For human users that is an inconvenience; for agents, which hold tokens and run long sessions, it is an architectural fact: revoking or granting a role changes agent behavior at token-expiry speed, not immediately. Decode the token when in doubt; the roles claim either is or is not in there, and no amount of retrying the API changes it.

The MCP door cannot see you

Here is the honest half. The Functions MCP extension authenticates the request, Entra and Easy Auth both did their jobs in the previous posts, and then hands the tool code a ToolInvocationContext containing the tool name, the arguments, a session id, and transport info. Nothing about the caller. The platform knows exactly who is calling; the tool does not get told. Per-user authorization on the MCP door is therefore not possible at this layer today, the same category of gap as the clientInfo one from the telemetry post, and the feature request writes itself: forward the authenticated principal to the tool invocation.

What remains honest in the meantime is door-level authorization: an MCP_DOOR_ROLES app setting grants roles to the door as a whole, an explicit operator decision about what the MCP channel may do, which is a real and common enterprise posture ("agents may read, agents may not transact") even though it is coarser than per-user claims.

And the telemetry proved the gate works, in the most satisfying way this series has produced. The traffic driver's deliberately wrong order calls hit the door before the grant and landed in Application Insights as order_forbidden. One app setting later, the same calls landed as order_rejected, because authorization now passed and the invalid arguments failed business validation instead. Same requests, different sentinel, and the shift between the two is the authorization gate becoming visible in a Log Analytics table. (One footnote for your own dashboards: the isolated worker emits each log line twice through the host, so absolute counts double; dedupe on operation id or read ratios, not totals.)

Door parity, completed

The commenter's test from months ago asked whether both doors preserve the same auth, error, and idempotency behavior. The answer the sample now gives in full: the decision is identical because it is taken once, in the kitchen, from the same policy. The rendering is deliberately different per door, a status code for code, a sentence for an agent, and the denial is observable in both channels, as a 4xx in the request logs and as a sentinel in the confusion telemetry. Parity of decisions, dialect per door.

The takeaway

Authentication moved to the platform over the course of this series; authorization never left your code, and this post is the argument that it should not. Put the policy in the kitchen where both doors share it, feed it claims where the platform forwards them and explicit door grants where it does not yet, render denials in each caller's dialect, and log every denial with the same attributable telemetry as any other contract failure. And remember the quiet operational lesson from the token detour: claims are minted, not looked up, so authorization changes reach your agents exactly as fast as their tokens expire, and no faster.

The policy, both doors, and the traffic driver are in the companion repo; the whole implementation is one small policy class and a dozen lines per door.

Linting Prose in CI

Steef-Jan Wiggers — Thu, 01 Oct 2026 06:25:36 +0000

A commenter on Watching the Contract Fail in Production put the conclusion of this whole series into one sentence: once agents consume tool descriptions as part of their decision-making, changing a sentence is no longer just documentation, it can be an interface change, so important wording changes should be treated like code changes. This post makes the build system agree. The companion repo now has a contract linter and a GitHub Action, and the first thing the linter did was fail its own repo.

Four rules, each one earned

The linter is about a hundred lines of dependency-free Python (measure/lint_contract.py), and every rule in it was paid for by a measurement earlier in the series:

Provenance. Every identifier parameter must say where its value comes from ("as returned by search_restaurants"). This is the sentence a JSON schema cannot express, and the difference between an agent that chains tools and one that invents ids; the confusion telemetry showed exactly that failure as the menu_unknown_restaurant sentinel.

When-to-call. Every tool description must contain a usage cue, not only a what-it-does sentence. Selection within a shortlist works because descriptions disambiguate between siblings, and when-to-call is where that disambiguation lives.

Token budget. The serialized tools array must stay under a budget, counted with the same cl100k tokenizer the discovery post used. The budget here is 350 tokens; every description edit now has a visible price.

Drift. The C# attribute descriptions and the APIM Bicep copies of the same tools must be character-for-character identical, because the contract post demonstrated that two copies of a contract fork within a week even with one author.

A fifth rule, the alias sentence that took paraphrase recall from 5 to 100 percent, is a warning rather than an error: proven valuable in the retrieval experiments, advisory at three tools.

The linter parses the contract straight out of the McpToolTrigger and McpToolProperty attributes, the authoritative copy, the same source the deployed description hash is computed from. One pleasant cross-check fell out of that: the parser's token count of the extracted contract was 277, exactly the number the discovery post measured from the live server's tools/list response. Two extraction paths, one contract, same count.

The first run: red, deservedly

I pushed the linter with the honest commit message "current contract fails its own lint," and the Action delivered:

Nothing here was staged. The drift errors are the exact fork the contract post documented by eye weeks ago, still sitting in the repo, now caught mechanically. The when-to-call errors are real too: get_menu's description said what the tool does and nothing about when to reach for it, and I wrote that description believing descriptions were the contract. The linter's first catch was its own author, which is the strongest argument I can offer for having one.

The fix: 25 tokens, priced

The fix commit added when-to-call clauses to two tools ("Call it after search_restaurants and before placing an order", "Use it only after get_menu has confirmed the exact item name") and aligned the three Bicep descriptions with the C#. The build went green, and the budget rule priced the edit: the contract grew from 277 to 302 tokens, 25 tokens for two sentences of disambiguation, well under budget and exactly the kind of spend the measurement posts said earns its keep.

The commit pair is the point of the whole exercise. One push failed CI over sentences; the next push fixed sentences and passed. Wording changes went through the same gate code goes through: a diff, a red build, a fix, a green build. And because the deployed description hash is computed from these same attributes, this fix also changed the descHash, so the confusion telemetry from the previous post can compare agent behavior before and after the new wording. The lint gate and the telemetry loop are two ends of one pipeline: CI checks what can be checked statically, the dashboard watches what only production can reveal.

What the gate can and cannot hold

Worth being precise about the boundary. These rules check the mechanical half of description quality: presence of provenance, presence of a usage cue, total cost, copy consistency. None of them can tell you whether the sentence is good, whether "Use it only after get_menu has confirmed the exact item name" actually improves selection, that is what the eval harness measures and what the telemetry observes. The lint is the cheap gate that runs in six seconds on every push; the eval is the expensive gate you run when the cheap one says the shape is right. Most teams have neither; the order to add them in is lint first, because it catches the failures that need no model at all.

The takeaway

The contract is part of the system behavior, so it gets the treatment behavior gets: versioned by hash, observed in telemetry, and now gated in CI. A one-word description edit can fail your build, and that is not overengineering, that is the build system catching up with where your interface actually lives. The linter, the workflow, and both commits, the red one and the green one, are in the companion repo; point measure/lint_contract.py at your own attributes and find out what your contract has been getting away with.

Foundry Model Ledger: Which Model, at What Price, Until When, and Which of Mine

Steef-Jan Wiggers — Tue, 29 Sep 2026 07:16:38 +0000

Every model conversation with a team ends with the same four questions. Is the model available in our region? What can it do? What does it cost per million tokens? And when does Microsoft retire it? Then the platform team asks a fifth: which of our own deployments are affected? Microsoft’s Foundry Model Explorer answers the first question well. However, it does not answer the other four. It cannot, because the answers live in your subscription and in a price list that was never meant to be read by software. So I built the Foundry Model Ledger to read them together. The interesting part is what the data looks like once you do.

What the Foundry Model Explorer already does

Microsoft’s explorer is a region availability reference. It shows one row per model and version, with lifecycle status, deployment SKUs, retirement date, and the list of regions that carry it, exportable to CSV. If your question is “where can I deploy gpt-5.6-terra”, it is the fastest answer there is. However, it is a periodic snapshot rather than a live read. It shows no prices in any currency. And it knows nothing about what you have deployed. Those three gaps are the Foundry Model Ledger.

Three sources, one Foundry Model Ledger

The model catalog lives in Azure Resource Manager. A single call, GET /subscriptions/{id}/providers/Microsoft.CognitiveServices/locations/{region}/models, returns every model and version a region carries. Each entry has its capabilities (chat completion, tool calling, embeddings, context window), its lifecycle status, its deployment SKUs, and its retirement date under deprecation.inference. This is the same data the Foundry portal shows. Moreover, it needs nothing more than Reader on the subscription.

Prices live in the Azure Retail Prices API. It is public, needs no authentication, and returns every meter Azure bills. Filter on serviceName eq 'Foundry Models' and a region, and you get the list prices for tokens, images, and hours, in any currency the API knows.

Your own deployments live under each Cognitive Services or Foundry account. At first I reached for Azure Resource Graph, which is the natural place to list resources across a subscription. It returned zero rows against a subscription with twelve accounts. Resource Graph does not index the accounts/deployments child type. So the Ledger lists the accounts through ARM and then calls each account’s /deployments endpoint, six at a time.

How the Foundry Model Ledger is built

The Foundry Model Ledger is a .NET 8 isolated Azure Function on Flex Consumption with a single-page UI. It deploys with azd up, in the same shape as the [INTERNAL LINK: RAG in 8 Steps on Azure] sample. A user-assigned managed identity with Reader on the subscription reads the first and third source. The second source needs no identity at all.

Pick a region and a currency, and the table shows model, version, lifecycle, capabilities, and retirement date with a days-left bar. Next to those sit the input and output prices per million tokens. Click a row for every SKU with its capacity range, every capability ARM reports, and every price meter the tool matched. The same panel has a button that checks every other region for the same model and version.

Foundry Model Ledger detail panel for gpt-5.6-terra with lifecycle, deployment SKUs, matched retail price meters and region availability
The detail panel for gpt-5.6-terra: lifecycle, retirement date, every SKU with its capacity range, and the retail meters the matcher tied to this model. The area: EUR line under capabilities is the catalog’s own geography tag, not the currency switch.

Finally, a second tab joins your deployments to the catalog of their own region and sorts them by soonest retirement.

Every model deployment in the subscription, listed through ARM per account and joined to the catalog of its own region, sorted by soonest retirement. Resource Graph returned zero rows for the same subscription.

What the data says

Sweden Central, on the day I took these screenshots, carried more than 300 catalog entries from 11 publishers. 216 were generally available, 66 in preview, and 16 marked as deprecating. 70 versions retire within 90 days. Furthermore, 30 are already past their retirement date but still listed. That last group matters. The catalog endpoint tells you what the region knows about, not what you can still deploy. For example, gpt-4o-mini has been closed to new deployments for a long time and still appears. The SKU list and the retirement date are the better signals.

The same model and version often appears twice, once for account kind OpenAI and once for AIServices, each with its own SKU list. The Foundry Model Ledger merges those into one row. If you script against the endpoint yourself, expect the duplicates.

The Retail Prices API returned 1,727 meters for Foundry Models in that one region. None of them carries a model identifier.

Prices are written for invoices

A meter name is a billing label, not a key. The same model, gpt-5.6-sol, is spread across meters such as 5.6 sol ShortCo Inp Std Gl 1M Tokens, 5.6 sol LongCo Cd Wr PP DZ 1M Tokens, and 56sol ShCo Cd Wr Fl Gl 1M Tokens. Older meters read gpt 4.1 nano cached Inp glbl Tokens and bill per 1K tokens; newer ones bill per 1M. Similarly, grok-4.6 appears as 4.6 Inp DZ Tokens under the product Azure Grok Models, with no “grok” in the meter name at all.

The abbreviations are their own dialect. Input is Inp, inpt, or in. Output is Outp, opt, outpt, or out. Cached input is Cd. Global, data zone, and regional are Gl, DZ, and regnl. Batch, priority, and flex tiers have their own tokens.

How the matcher works

So the Foundry Model Ledger has a matcher. First, it narrows meters to the model’s publisher. Then it tokenizes both sides the same way and requires every token of the model name (minus the family prefix) to appear in the meter, in order. It also rejects meters that carry a sibling variant such as mini or pro the model does not have. When a meter carries a date token, o3 0416 or chat-latest 08062026, it pins the version. Next, it classifies direction, deployment type, tier, and context length, and normalizes the unit to a price per million tokens. Finally, it picks a headline: global standard, short context, uncached. Every price in the table carries a confidence label, exact, name, or loose. In addition, the detail panel shows every matched meter, so the headline number is never the only evidence.

What the first live run got wrong

On the first live run it priced 204 of the entries in Sweden Central. The misses were instructive. FLUX image models came out at “40,000 per million” because their meters bill per 1K images, and I had treated every 1K unit as tokens. Qwen sits under product Qwen models, which the publisher map did not know. Moreover, most of its meters are fine-tuning meters that must never become a headline price. And the Anthropic models, plus Cohere rerank and parse, are in the catalog with no Foundry Models meter in the region at all. The model exists; the public price does not.

Foundry Model Ledger price meters tab showing raw Azure Retail Prices rows for gpt-5.6-sol in Sweden Central
The raw Azure Retail Prices rows for gpt-5.6-sol in Sweden Central. Thirty-odd meters, no model identifier, and the abbreviations the matcher has to decode: ShortCo, Cd, PP, Gl, DZ.

Two tabs that came from using it

The first version stopped at the tabs above. Two more followed within a day, because the first questions people asked were not the ones I had built for.

Where else is this model available

The region check in the detail panel gives a list of names. That is correct and hard to read. So the Foundry Model Ledger now has an availability map. Every Azure region is plotted from the subscription’s own location metadata, on a world outline from Natural Earth. Type a model name, pick a version, and the regions that carry it light up in the brand blue. Regions without Azure AI services show as a dashed ring, and regions that host the service but not the model stay grey. Zoom presets for Europe, North America, Asia Pacific, the Middle East and Africa, and South America keep the labels readable where regions cluster. In Europe alone there are twenty. The data behind the map is one lightweight catalog call per region, cached, so a check across every region takes a few seconds the first time and is instant after that.

Foundry Model Ledger availability map showing the Azure regions that carry a selected model version
Where claude-haiku-4-5:2 is available. Blue is carried, grey is hosted but absent, dashed is no Azure AI services. The Europe preset separates the twenty European regions.

What else does the same job for less

The second question is the one that decides budgets. If we use this model for that workload, what else in the region can do the same job, and what would it cost instead? The Alternatives tab answers it with the data the Ledger already has. Pick the model you use or consider. Its capability flags from the catalog, such as chat completion, tool calling, JSON schema output, and image input, become chips, and every chip is required by default. Click one off if you do not need it and the list widens. Every other model in the region that reports all remaining capabilities is listed with its list price, what your monthly workload would cost on it, and the delta against your model, cheapest first. Enter the workload as millions of input and output tokens per month. Filters restrict the list to the same publisher or to generally available models, and retired versions are excluded. Other versions of the same model are listed last, so they do not pose as alternatives.

The comparison is on capabilities, not on quality. A nano model will always look like a 99 percent saving next to a frontier model, and the table does not know whether it would pass your evaluation set. What it does give you is the shortlist and the price gap in one view, which is the part that used to take an afternoon with the pricing page open in three tabs.

Foundry Model Ledger alternatives tab listing models with the same capabilities and their monthly cost delta
Alternatives to claude-fable-5:1 with the same capability flags, priced for 10M input and 2M output tokens a month. Green is cheaper, orange is dearer. The match is on capabilities, not quality.

Where the Foundry Model Ledger is the wrong answer

Do not budget on it. These are list prices, matched heuristically, in whatever currency you pick. The matcher will be wrong somewhere the day Microsoft renames a meter. It also cannot price provisioned throughput, which is billed per hour per unit rather than per model. Use it to see the shape of a decision, then confirm on the pricing page.

Do not put it on the internet as is. The Function has no authentication, and it exposes your deployment list to anyone with the URL. Keep it internal, or put Easy Auth with Entra ID in front of it. If you already run model endpoints behind API Management, the same gateway can front the Ledger, with the policies from the APIM for AI Workloads series.

And do not read the catalog as an availability promise. Listed does not mean deployable, and a retirement date is a floor, not a schedule. For a plain “where is it available” question, Microsoft’s explorer remains the quicker tool.

What I would build next

The tab I did not plan is the one people ask about: my deployments, joined to the catalog, sorted by retirement. That is a governance question, not a browsing question. It sits next to the routing questions from Don’t Build Around Today’s Model. Build for the AI Control Plane. Consequently, the natural next steps are all on that side. All subscriptions in a tenant instead of one. An alert when a deployment sits on a version retiring within 90 days. A history, so you can see what appeared and what closed in a region since last month, which is a changelog Microsoft does not publish. And a cost delta for moving each deployment to its successor, which the Alternatives tab now does for one model at a time and should do for the whole deployment list.

The Foundry Model Ledger repo is at github.com/steefjan1/foundry-model-ledger. azd up deploys it; scripts/run-local.ps1 runs it against your az login; node tools/mock-server.mjs runs the UI on a sample snapshot without .NET or Azure. If the matcher misprices a model in your region, the detail panel shows why. An issue with that screenshot is the fastest way to get it fixed.

Watching the Contract Fail in Production

Steef-Jan Wiggers — Mon, 28 Sep 2026 06:00:40 +0000

Tool Descriptions Are the Contract made a claim I had not tested: your MCP server's recovery messages are free confusion telemetry, because every time one is returned, you know exactly which misunderstanding just happened. The comment thread then designed the implementation better than I had: track each string alongside the tool version and a description hash, add the schema version as its own dimension, and treat a missing value as a first-class result, otherwise the cleanest dashboard is the one hiding the deployment that broke observability. This post builds exactly that, drives real traffic at it through the Entra-protected MCP door, and checks the dashboard against ground truth.

The one-line change, and the hash that cannot drift

Each of the sample's three recovery returns now routes through one helper:

private string Confusion(string sentinel, string tool, string message)
{
    logger.LogWarning(
        "Contract confusion: {Sentinel} on {Tool} (descHash {DescHash}, schemaVersion {SchemaVersion})",
        sentinel, tool, ToolContract.DescriptionHash, ToolContract.SchemaVersion);
    return message;
}

The sentinels are stable slugs, one per recovery sentence: search_no_match, menu_unknown_restaurant, order_rejected. The structured parameters become customDimensions in Application Insights, so the log line is a queryable row, not prose.

The description hash is the part worth stealing. The contract post showed that prose copies drift, so a hash maintained by hand would just be one more copy to forget. Instead, ToolContract computes it at startup by reflecting over the same McpToolTrigger and McpToolProperty attributes the MCP extension serves to clients, hashing every name and description in the deployed binary. The hash cannot disagree with what agents actually see, because it is derived from it. Edit one word of one description, redeploy, and every subsequent confusion row carries a new hash: the before and after of a contract change, joinable in KQL.

Two doors before the traffic

Driving traffic at the server turned into its own small verification of the 401 post. The traffic driver is a Python script that speaks streamable-HTTP MCP directly, and since the server has built-in Entra authentication, the driver needs a bearer token. Getting one stopped at two separate gates.

First, Entra refused to mint the token at all: AADSTS65001, because Azure CLI, the tool requesting it, was not on the app registration's preauthorized client list. Then, with that fixed, the function app answered 403, because App Service authentication keeps its own allowed-client-applications list, and the CLI was not on that one either. The known-clients control from the 401 post exists twice, independently: Entra decides who may start the flow, Easy Auth decides whose tokens the app accepts, and a client must pass both. My own traffic generator had to be admitted as a known client, twice, before it was allowed to confuse the server. For a regulated organization, that is not friction; that is the audit trail existing before you asked for it.

The traffic, with ground truth attached

The driver then made 60 calls: a rotation of requests that are wrong the way agents are actually wrong. An invented restaurant id, a restaurant name where an id belongs, a menu item from memory instead of from get_menu, a quantity past the cap, filters that match nothing, and valid calls as control. Because the rotation is deterministic, the console output is ground truth: exactly 16 menu_unknown_restaurant, 16 order_rejected, 14 search_no_match, and 14 clean results.

Which makes the dashboard check a real test instead of a demo. In Application Insights:

traces
| where message startswith "Contract confusion"
| extend sentinel = tostring(customDimensions.Sentinel),
         tool = tostring(customDimensions.Tool),
         descHash = tostring(customDimensions.DescHash),
         schemaVersion = tostring(customDimensions.SchemaVersion)
| summarize confusions = count() by sentinel, tool, descHash, schemaVersion
| order by confusions desc

The result: order_rejected 16, menu_unknown_restaurant 16, search_no_match 14, every row carrying the same descHash and schemaVersion 1. The dashboard matches the driver's console exactly, so the pipeline is verified end to end, from a wrong argument in a JSON-RPC call to an attributable row in a Log Analytics table.

Reading the dashboard as a contract report

Each sentinel names a failing sentence. A rising menu_unknown_restaurant count says the provenance sentence in get_menu's description, "as returned by search_restaurants", is not landing: agents are inventing ids. A rising search_no_match says callers expect a wider catalog than the descriptions promise. And because every row carries the descHash, the response to a bad trend is an experiment, not an argument: rewrite the sentence, redeploy, and compare the same sentinel across the two hashes. The description edit gets the same before-and-after treatment a performance fix gets.

The commenters' missing-value rule matters here too. The healthy state of this dashboard is not empty, it is present-with-baseline: some confusion is normal while valid traffic flows. A sentinel that silently vanishes, or rows arriving without a descHash, means a deployment broke the telemetry, not that agents stopped being confused. The alert to write is on absence and on unknown dimension values, not only on spikes.

What is still missing, honestly

One dimension from the design is not in these rows: the client. The MCP initialize handshake carries clientInfo with a name and version, but the Functions MCP extension does not currently hand that to the tool method through ToolInvocationContext, so per-client confusion rates are not yet possible at this layer. That is a real gap; per-client breakdowns are how you distinguish one badly prompted agent from a contract problem affecting everyone. Until the extension exposes it, the client mix has to come from the platform's request telemetry instead of the tool's own rows.

The takeaway

This is the cheapest contract test I know of: one logging helper, three stable sentinels, a hash computed from the deployed attributes so it can never drift, and a KQL query. The recovery messages were already written, the dashboard was already there; connecting them turns every agent mistake into an attributable data point against a specific version of your words. The prose is the interface, and now the interface has monitoring.

The code, the traffic driver, and the queries are in the companion repo; the whole change is one class and one helper in src/RestaurantBackend/Mcp/RestaurantMcpTools.cs, and measure/drive_confusion.py reproduces the traffic against any MCP server of your own.

The Model Was Never the Problem

Steef-Jan Wiggers — Thu, 24 Sep 2026 08:07:57 +0000

The shortlist post ended with a promise: stage two of the eval, selection within the shortlist, measured with a live model, scored so that a recall miss can never masquerade as a selection miss. This post pays that debt. The design came straight from the comment threads: same tool pools, same task pairs, shortlists from the same retriever, and every result row tagged with the description-set hash so the numbers stay attributable across changes. The runner is measure/eval_selection.py in the companion repo, stdlib Python against any Azure OpenAI deployment.

The method

Stage one showed recall failing at 5 percent on paraphrased requests, so scoring stage two on retrieved shortlists would mostly measure retrieval again. Instead, every shortlist here is forced: the correct tool plus its four strongest BM25 distractors, deterministically shuffled. That construction has a pleasant side effect. Because the distractors are whatever retrieval scores highest for the request, they are the correct tool's same-noun siblings. When the expected tool is approve_claim, the shortlist also contains cancel_claim, update_claim, and search_claim. The model does not get to coast on topic matching; it has to read verb semantics.

The model is gpt-5.4-mini, deliberately the small and cheap one, at its default temperature, sixty tasks per condition, half vocabulary phrasing and half paraphrase, across the four description conditions from the recall post. Scored on two things: did the model pick the expected tool, and did the task's reference value (the INV-2005 in the request) land in the arguments of a correct pick. The argument check is deliberately lenient; exact schema validation would punish harmless formatting.

One anecdote from setup that belongs in this series: the eval refused to run with an API key, because the Azure AI resource had disableLocalAuth set. The harness that measures keyless MCP tooling had to authenticate with an Entra bearer token itself. The 401 post's argument, arriving in person.

The numbers

Selection accuracy, forced shortlist of five, sixty tasks per condition:

Descriptions	Selection, vocabulary	Selection, paraphrase
Terse	97%	97%
Realistic	97%	93%
Verbose	90%	93%
Realistic plus aliases	97%	97%

Read that table as flat, and I mean that as a finding, not a shrug. At thirty tasks per cell, every difference in it is one or two tasks, and one run at default temperature is a sample, not a truth. The honest summary is a single sentence: selection accuracy is 90 to 97 percent everywhere. A small model picks the right tool from a five-tool shortlist about nineteen times in twenty, whether the descriptions are terse or verbose, whether the user echoed the tool's vocabulary or paraphrased it into synonyms, and with the tool's own siblings crowding the list. Argument filling lands slightly lower, 87 to 97 percent, under the lenient scoring.

Two null results hide in there, and both are useful. Description verbosity, which did nothing for recall, also does nothing for selection; terse held its own against descriptions three times its token price. And the paraphrase gap, which was catastrophic for recall, nearly vanishes at selection time: once the right tool is in front of it, the model bridges "sign off on the damage report" to approve_claim without difficulty. The vocabulary gap is fatal to retrieval and almost irrelevant to judgment.

Multiply the stages

The two-stage arithmetic is the real headline. End to end, task success is roughly recall times selection. For a paraphrased request against one hundred realistic tools:

Setup	Recall	Selection	End to end
Realistic descriptions	5%	93%	~5%
Realistic plus aliases	100%	97%	~97%

The entire end-to-end difference lives in stage one. Report one blended number, as most tool-calling benchmarks do, and you would conclude the model picks tools badly, when the model almost never saw the right tool at all. Measured separately, the model turns out to be nearly blameless, and the 13-token alias sentence from the recall post is carrying the whole system.

What this changes about the budget

The reflex when an agent picks wrong tools is to reach for a bigger model. At this shortlist size, that money is misspent, because selection was never the failing stage. The failing stage was vocabulary coverage in the descriptions, which is fixed with words that cost thirteen tokens per tool, not with a model that costs ten times more per call. There will be scales where that stops being true; a shortlist of twenty near-duplicates or genuinely ambiguous intents will stress judgment in a way five siblings do not, and that is a measurable follow-up. But the burden of proof has moved. Before spending on model size for tool calling, measure your recall, because the odds are the model never saw the tool you are blaming it for not picking.

The caveats, as always

One model, one run, sixty tasks per condition, differences within the table smaller than its noise floor. The forced shortlist isolates selection but flatters no condition, since all four saw identical lists. And the ceiling on the alias numbers inherits the recall post's caveat: my aliases match my paraphrases by construction. What survives all of that comfortably is the shape: selection high and flat, recall variable and decisive.

The runner, the results with their description hashes, and everything needed to reproduce this against your own deployment are in the repo under measure/; the whole run costs a few cents. Argue with the numbers directly, it is what the comment section of this series is for.