DEV Community: Patrick Hughes

3 Tests Before a GGUF Quant Runs Your Coding Agent

Patrick Hughes — Thu, 03 Sep 2026 16:05:10 +0000

3 Tests Before a GGUF Quant Runs Your Coding Agent

A model can fit in VRAM, answer one prompt, and still fail as a coding agent. I do not keep a quant because one chat reply looks good. I keep it after it completes the same small tool loop more than once.

Short answer: Test a local coding model on file reading, a bounded patch, and a real test command. Keep the prompt, tools, context, and pass rules fixed. Pick the smallest quant that passes every gate across repeated runs.

Why is VRAM fit only the first gate?

Fit tells me whether I can start the model. It does not tell me whether the model can follow a tool schema, respect a file boundary, or report a failed command.

I solve fit first. I check the file size, context target, and GPU offload plan. My GGUF quant guide covers that choice, and my --n-gpu-layers guide covers the offload check.

Then I stop changing runtime settings. A moving setup makes the result hard to compare.

This post decides one thing: which quant file I keep for a coding agent. My Q4_K_M vs Q5_K_M guide picks a quant on paper, before the download. This one picks the file by running it.

What three tests do I run?

I use one small repository fixture. It has a short source file, a focused test, and one rule that the model must preserve.

The first test is read. I ask the model to open the source and test files. It must state the rule before it proposes a change. This catches models that guess from filenames or from the task text.

The second test is edit. I ask for one bounded change in one named file. The model must not touch nearby code, add a package, or rewrite the test to make it pass.

The third test is proof. I ask the model to run the focused test and report the command, exit code, and result. A claim without command output does not pass.

How do I keep the comparison fair?

I keep the same model family, prompt, repository state, context limit, tool definitions, and backend version. I change only the quant under test.

I also reset the fixture before each run. Cached files and prior edits can hide a weak result. I use three runs per quant because one clean run can be luck. This is my acceptance rule, not a universal benchmark.

I set temperature to zero when the backend supports that setting. The output can still vary, so I score behavior instead of exact wording.

The stop reason is the part I can show from a measured run. On 2026-07-09 I ran llama3.1:8b and gemma4:26b, both Q4_K_M, on Ollama 0.31.1 with the agent-code-task-512 workload. Both used num_ctx=4096, num_predict=512, and temperature=0.

That pair shows why the proof gate matters. llama3.1:8b stopped on its own at 459 output tokens. gemma4:26b ran to 512 tokens and stopped because it hit the cap, not because it finished. A truncated patch can still read like a complete one, so I check the stop reason before I score the run.

What makes a quant fail?

I reject a run when the model invents file content, sends invalid tool input, edits outside the named file, changes a test without permission, or calls a failed test successful.

I also reject an incomplete run. A useful explanation does not replace the requested patch and command proof. The agent must finish the small loop.

One safety or boundary failure rejects that run. Style differences do not. I care about the action and its proof.

How do I choose between Q4_K_M and Q5_K_M?

I run the same acceptance set against both files. I do not assume the larger file wins. I keep the smaller quant if it passes all three cases in all three runs.

If Q4_K_M misses a tool call and Q5_K_M passes, I keep Q5_K_M when it still fits my target context. If both pass, I keep Q4_K_M and use the saved memory for context or another process.

If neither passes, I stop. More prompt text rarely repairs a model that cannot follow the basic loop. I test another model family or move that task to a stronger endpoint.

What should the test record contain?

I record the model name, quant, backend version, context limit, GPU layer setting, prompt, tool results, changed files, test command, and exit code. I keep failed rows.

The failed rows explain why a model did not enter the coding-agent lane. They also stop me from repeating the same download and setup work later.

This record turns model choice into a local decision. I can rerun it after a backend update, a new quant, or a tool-schema change. The gate stays the same even when the model changes.

Accompanying prompt

What the prompt does: It tests one GGUF quant against a fixed read, edit, and proof loop for a local coding agent.

Copy/paste this prompt:

Role:
You are testing a local coding model for safe repository work.

Context:
Repository path: [ ]
Source file: [ ]
Test file: [ ]
Rule to preserve: [ ]
Focused test command: [ ]
Allowed edit scope: [one file]

Task:
1. Read the source file and test file.
2. State the rule that the change must preserve.
3. Make the smallest change inside the allowed scope.
4. Run the focused test command.
5. Report the command, exit code, changed files, and result.

Output:
- Rule found
- Change made
- Changed files
- Test command
- Exit code
- PASS or FAIL

Constraints:
- Do not invent file content.
- Do not add a package.
- Do not edit a test unless the task permits it.
- Treat a nonzero exit code as FAIL.
- Stop and report the blocker if a required file or tool is missing.

Copy the block above.

Get the next measured local-AI field note by email: The 5090 Reports.

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=gguf-quant-coding-agent-acceptance-test-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

One model logged 14.5 and 6,178 prompt tokens a second

Patrick Hughes — Tue, 01 Sep 2026 16:05:08 +0000

One model logged 14.5 and 6,178 prompt tokens a second

Two rows in my 2026-08-27 report describe the same model on the same GPU. One says it read prompt tokens at 14.5 a second. The other says 6,178.9. Nothing about the model changed between them. The prompt size did.

Short answer: A local model has no single prompt ingestion rate. These runs used different input sizes, so their prompt rates do not support a direct speed ranking. Size the model with a prompt the size it will really get. Canonical URL: https://bmdpat.com/blog/local-llm-prompt-eval-rate-prompt-size-2026

What did the two runs record?

Both rows are gemma4:26b, quant Q4_K_M, on Ollama 0.31.1, temperature 0, on the RTX 5090.

The short-gen-256 run sent 53 input tokens with num_ctx=4096. Ollama reported prompt_eval of 53 tokens at 14.5 tok/s. That is about 3.66 seconds to read 53 tokens.

The long-context-summarize run sent 4,470 input tokens with num_ctx=8192. Ollama reported prompt_eval of 4,470 tokens at 6,178.9 tok/s. That is about 0.72 seconds to read 4,470 tokens.

The bigger prompt was 84 times larger and finished reading in a fifth of the time. As a rate, that is a 426x spread on one model in one report.

These are the same Gemma 4 26B rows I already published when I split input and output timing. That post showed why one blended speed is a bad score. This post is the next failure: ranking prompt rates after you split them.

Why can I not rank these prompt rates?

A rate is work divided by time. These rows divide different token counts by different prompt_eval intervals.

The rows do not isolate any fixed cost. They only record the prompt token count and the full prompt_eval interval. Ollama documents both fields in its generate API.

I do not have a field that says why the 53-token interval took 3.66 seconds. The honest read is narrower. The 14.5 tok/s row does not define gemma4:26b's prompt speed for a 4,470-token job.

Did the same thing happen on the 8B model?

The same report has another unmatched pair. llama3.1:8b Q4_K_M on the same engine read 46 tokens at 679.4 tok/s and 4,344 tokens at 12,108.7 tok/s.

That is a 17.8x spread, not 426x. Both pairs mix a short input with a long input. Neither pair supports a prompt-speed ranking.

Two models, four rows, and not one of them supports a sentence that starts with "this model reads prompts at."

Why does the output rate stay stable?

Compare the generation column for the same two gemma4:26b runs. Short prompt: 198.81 output tok/s. Long prompt: 180.18 output tok/s. That is about a 10 percent gap.

These two output rates stayed much closer than the prompt rates. That result does not explain the input spread. It shows why I compare each phase with matched work.

Splitting the phases is not enough if you then compare input rates across different prompt sizes. It is the same failure I hit with wall clock, where the faster run took 83x longer. A rate with a small denominator will lie to you in a table that looks orderly.

What does this change about sizing a local agent?

A retrieval worker or a coding worker sends a lot of input. A file, an error log, tool output, and instructions add up fast. If I size that job from a 53-token test prompt, I predict the wrong thing in both directions.

So I test at the prompt size the job will really send. If the agent will carry 4,000 tokens of context, the sizing run carries 4,000 tokens. A short smoke test proves the model answers. It does not measure ingestion.

I also stopped ranking models by prompt rate across rows that used different input sizes. That comparison has no meaning. If the input counts differ, the rates are not comparable, and no amount of averaging repairs it.

What do I record now?

Every prompt rate in my table carries its input token count and its num_ctx in the same row. A rate without its denominator is not a measurement I can reuse later.

I mark any prompt-eval rate under 1,000 tokens of input as indicative, not comparable. It still goes in the table, because failed and weak rows are worth keeping. It just cannot be quoted against a production-sized run.

The rule is short. Record the count next to the rate, and only compare rates that read prompts of the same size.

Accompanying prompt

What the prompt does: It checks whether two local model prompt-eval rates can be compared, and rejects the comparison when the input sizes differ.

Copy/paste this prompt:

Role:
You are auditing local LLM benchmark rows before anyone quotes them.

Context:
Row A: model, quant, engine, num_ctx, input token count, prompt rate
Row B: model, quant, engine, num_ctx, input token count, prompt rate
Row A values: [ ]
Row B values: [ ]

Task:
1. Confirm both rows name model, quant, engine, num_ctx, and input count.
2. Report the ratio of the two input token counts.
3. Mark any row with fewer than 1000 input tokens as indicative only.
4. Decide whether the two prompt rates are comparable.

Output:
- Missing fields per row
- Input count ratio
- Indicative rows
- COMPARABLE or NOT COMPARABLE
- One sentence of reasoning

Constraints:
- Do not average rates across different input sizes.
- Do not infer a missing input count from the context limit.
- Return NOT COMPARABLE when a required field is missing.
- Name a cause only when a timing field supports it.

Copy the block above.

Get the next measured local-AI field note by email: The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Ollama Load Time Can Hide a Fast Local LLM

Patrick Hughes — Mon, 31 Aug 2026 16:05:12 +0000

Ollama Load Time Can Hide a Fast Local LLM

My local LLM produced tokens at 188.16 tokens per second. The request still took 43.86 seconds. The model was fast, but the load was not.

The short answer: Do not use tokens per second as the only local LLM speed result. Split wall time into model load, prompt work, and output work. A slow load can hide fast output, while a resident model can make the same task feel much faster.

Canonical URL: https://bmdpat.com/blog/ollama-load-duration-local-llm-benchmark-2026

What did the RTX 5090 run record?

On August 4, 2026, I sent a small decision task to llama3.1:8b through Ollama. The request used a 1,024-token context limit. It read 44 prompt tokens and wrote 32 output tokens.

The first row in my August 27 report recorded 43,860 ms of wall time. Ollama reported 43,476.3 ms of model load, 183.2 ms of prompt work, and 170.06 ms of output work. The output rate was 188.16 tokens per second.

That output rate sounds fast because it is fast. It only describes the output phase. It does not include the long load that the user had to wait through.

The next request used the same model, token counts, and context limit. Its wall time fell to 530 ms. Ollama reported 142.43 ms of load and 208.98 ms of output work.

The report did not record the model residency state. I will not label the second row as a proven warm run. I can state the narrower result: the second request had far less load time, and its full wait was about half a second.

Which Ollama timing fields matter?

Ollama returns separate timing fields from its generate endpoint. Its API reference defines total_duration, load_duration, prompt_eval_duration, and eval_duration in nanoseconds.

I convert those values to milliseconds before I compare runs. Each field answers a different question.

total_duration shows Ollama's full request time. load_duration shows time spent on model load. prompt_eval_duration shows time spent on the input. eval_duration shows time spent on output tokens.

I also keep an outside wall clock. It includes client work and local transport around the Ollama call. If the wall clock and total_duration differ by a large amount, I check the client path before I blame the model.

This split extends the wall-clock test I use beside tokens per second. One rate cannot explain every part of a request.

Why can tokens per second give the wrong verdict?

Tokens per second answers a narrow question: how fast did the model write after output work started? A person waiting for a coding agent cares about the full delay before the useful result arrives.

A cold load can dominate a short task. A long prompt can move the delay into prompt work. A large output can make output work dominant. The same model can therefore need different changes for three slow requests.

If load time is high, I check model reuse, process churn, and memory pressure. If prompt time is high, I check context size and repeated input. If output time is high, I compare model size, quant, runtime settings, and hardware fit.

I do not tune all three at once. That would erase the cause. I change one boundary, run the same task again, and save another row.

How should a local LLM benchmark store timing?

Store the raw duration fields with the model, runtime version, prompt count, output count, context limit, and wall time. Keep the source unit or save the conversion rule beside the schema.

Do not replace total time with output rate. Keep both. A benchmark row should let another builder answer two separate questions: how fast was token output, and how long did the request make the user wait?

I also record missing state as missing. My August 4 rows say residency_state=unrecorded. That is better than changing an unknown into a warm-load claim.

This follows the same rule as my local LLM benchmark decision record. A measurement earns a decision only when the row keeps enough context to explain it later.

What should you change first?

Start with the largest measured part of the request. In my 43.86-second row, model load was the clear first target. Output speed was not the problem to solve.

Then repeat the exact workload. Keep the prompt, token cap, context limit, and model fixed. Compare the full timing split, not one attractive rate.

That small rule prevents wasted tuning. It also stops a fast token rate from hiding a slow user path.

Accompanying prompt

What the prompt does: Reviews one Ollama benchmark row and finds which measured phase should get the first change.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
Paste one benchmark row with model, wall time, total_duration, load_duration, prompt_eval_duration, eval_duration, token counts, and runtime settings.

Task:
1. Convert all durations to one unit.
2. Find the largest measured phase.
3. Name one change that targets only that phase.

Output:
- A timing table with wall, load, prompt, and output time.
- A first-change recommendation with a pass or fail rule.
- A list of missing fields that block a stronger claim.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.

Copy the block above.

I publish measured local AI tests in The 5090 Reports. Join the email list for the next result.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Split Local LLM Prose and JSON Jobs by Model

Patrick Hughes — Mon, 31 Aug 2026 16:05:08 +0000

Split Local LLM Prose and JSON Jobs by Model

One local model wrote better prose. Another followed a bounded JSON contract with less memory. I stopped asking one model to win both jobs.

The short answer: Route local models by the output contract. Use a prose model for drafts, a measured structured model for JSON scoring, and a separate embedding model for retrieval. Then test each lane with code before it can affect another system.

Canonical URL: https://bmdpat.com/blog/split-local-llm-prose-json-model-routing-2026

Why did I stop using one local model for every job?

My local AI stack has three different output contracts. A blog draft needs clear paragraphs and a steady first-person voice. A scoring job needs a small JSON object with fixed fields. Retrieval needs an embedding vector, not generated text.

Those jobs fail in different ways. Prose can sound flat or miss the point. JSON can omit a key, use the wrong type, or add text outside the object. An embedding route can return the wrong vector size or fail to load.

One model name cannot hide those differences. I now treat the output contract as part of the route.

Which models handle each local lane?

My current policy sends prose generation to gemma4:26b. It sends bounded scoring and triage to qwen3.5:9b. It keeps embeddings on nomic-embed-text.

This is a route choice for my machine and my tests. It is not a claim that one model is best at prose or JSON on every GPU.

The policy also limits the local host to one loaded model and one parallel request. That means a second lane is not free. A model change can cause a load delay and memory churn. I only keep the split because each lane has a different job and a different pass rule.

This extends the route rule in Your Local LLM Is a Different Tool. That post separates local work from frontier work. The same idea applies inside the local stack.

What makes the JSON lane different from prose?

Ollama supports a JSON schema in the format field. Its structured output guide shows the schema passed to the model, then validates the returned text with Pydantic.

I still treat the response as untrusted input. A schema request can guide the model. The application must parse the response, validate each required field, reject extra states it cannot handle, and fail closed when validation breaks.

A prose lane needs different checks. My draft path runs word count, voice, source, image, link, and paragraph gates. A valid JSON object would not prove any of those. The model route and the verifier must travel together.

How do I test a model before it gets a lane?

I start with a real job and a fixed acceptance rule. For structured work, the response must parse, match the schema, stay inside the allowed values, and survive a hostile or missing input. For prose, the draft must clear the same deterministic checks that run before publication.

I save failed attempts. A model that returns the right object once has not earned a permanent route. I repeat the same contract, keep the failures, and compare the cost of retries and load changes.

That is the same promotion rule I use in I Do Not Promote a Local Model Because It Fits. A model earns a route by passing the job, not by loading or producing a good demo.

What should the router record?

Each decision should record the lane, model, model tag, prompt or schema version, verifier result, stop reason, load time, and final action. The final action is important. A failed local attempt should become escalate, defer, or fail-safe, not a guessed answer.

Do not store only the accepted runs. The rejected rows show whether the lane is stable or merely lucky. They also make a future model change testable against the same contract.

The router should never turn a parse failure into prose and pass it downstream. JSON requested plus JSON invalid means the structured lane failed.

When should you add a second local model?

Add a lane only when the work has a distinct output contract and the candidate passes it. Do not add a model because it is new, small, or high on a general benchmark.

The split also has an operating cost. More model files mean more disk, more health checks, more load paths, and more chances to route the wrong job. If one measured model passes both contracts on your hardware, keep one.

My rule is simple: one lane per distinct job, one verifier per lane, and no promotion without replayable evidence.

Accompanying prompt

What the prompt does: Turns a mixed local AI workload into model lanes with a clear verifier and failure action for each lane.

Copy/paste this prompt:

Role:
You are a local LLM routing engineer.

Context:
Paste the jobs, candidate models, hardware limits, and sample outputs.

Task:
1. Group the jobs by output contract: prose, structured data, embeddings, or another exact type.
2. Assign a candidate model to each group only when evidence supports it.
3. Define a code-based verifier and a fail-closed action for every group.

Output:
- A route table with job, model, contract, verifier, and failure action.
- A replay test that a new model must pass before promotion.

Constraints:
- Keep it short.
- Use exact model tags and schema names when available.
- Do not invent missing measurements.

Copy the block above.

I publish measured local AI tests in The 5090 Reports. Join the email list for the next result.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

A 32 GB GPU Still Needs Host RAM Headroom

Patrick Hughes — Sat, 29 Aug 2026 16:05:08 +0000

A 32 GB GPU Still Needs Host RAM Headroom

My local model was installed. Ollama answered its model-list request. The RTX 5090 reported 32,607 MiB of VRAM. I still did not start the test.

The short answer: VRAM fit does not prove that the whole host is ready for a local LLM load. On 2026-08-29, my preflight found only 7.02 GiB of free host RAM against my 16.0 GiB cold-load floor. It stopped before inference and saved a clear reason.

Canonical URL: https://bmdpat.com/blog/rtx-5090-host-ram-headroom-local-llm-2026

Why did the run stop when the GPU was ready?

I ran config/ollama/health.py --json before a small Ollama test. The check reached the local API and found all eight installed model names. It also found gemma4:26b, the model required by my local writing path.

The nvidia-smi probe reported an NVIDIA GeForce RTX 5090, driver 610.88, and 32,607 MiB of total VRAM. Those checks passed.

The host-memory check did not. Windows had 7.02 GiB free. My cold-load policy needs 16.0 GiB free before the system can ask Ollama to generate. The health command returned ok: false and named the failed guard. It made no generation request.

That last detail matters. This was not an Ollama error. It was not a model error. It was a local admission rule that stopped a risky load. The generate HTTP call was never sent, so this run has no token rate, eval count, or duration to publish.

A later runtime budget such as AgentGuard can cap spend after a request starts. This floor is earlier. It refuses the load while host RAM is 7.02 GiB free against a 16.0 GiB policy.

What is the difference between VRAM and host RAM?

VRAM holds GPU work such as model weights, runtime state, and the active context. Host RAM supports the operating system, the inference process, file reads, model setup, and every other process on the machine.

A model can fit in VRAM while the host has little working room. The GPU total does not show how much memory Windows, Ollama, browsers, build tools, and other agents need at the same time.

This is why I keep host memory separate from the VRAM budget for shared local AI workloads. The two checks protect different boundaries.

The 16.0 GiB floor is my policy for this machine. It is not an Ollama rule. Another host can use a different floor after it measures its own cold loads and normal background work.

What should a local LLM preflight check?

I use four gates.

First, query the model catalog. A missing model is a setup result, not a speed result.

Second, read free host RAM before model load. Compare it with a fixed floor for that host and workload. If the check fails, stop before the request.

Third, load one model with a small request. Save the model name, runtime version, host, and the pass or fail result.

Fourth, run the measured workload. Capture wall time, prompt tokens, output tokens, output rate, VRAM, host RAM, and stop reason.

I keep these gates separate from the broader local AI runtime preflight. Build tools and runtime support come first. Memory admission comes just before model load.

Why not let Windows page memory to disk?

Paging can keep a process alive, but that is not the same as keeping an agent host healthy. A large memory load can slow unrelated work and make wall-time results hard to compare.

My 5090 machine also runs the vault, browser tasks, and scheduled agents. A local model test must share the host. It does not own the machine.

A fixed free-memory floor makes that choice explicit. The run starts only when the machine has enough working room. If the floor is too high, measured skips will show that. If it is too low, load-time and system-health receipts will show the cost.

I do not lower the floor to make a daily test turn green. I free memory, choose a smaller model, or record the skipped run.

What should the failure receipt contain?

Save the time, host, model, runtime version, GPU, driver, total VRAM, free host RAM, required floor, and the exact failed gate. Also save whether the generation request started.

The distinction between blocked and failed is important. On 2026-08-29 my health command returned generate_ok: false and named the host RAM guard as the reason. The request never started, so there is no failed generation to explain.

That receipt prevents a later report from showing a blank token rate with no reason. It also prevents someone from blaming the model for a request that never reached it.

A useful preflight names the failed gate and whether generate started. Mine named the host RAM guard and recorded generate_ok: false, so the table does not show a blank token rate.

Accompanying prompt

What the prompt does: It turns a local LLM load plan into a host-memory gate with a clear pass rule and failure receipt.

Copy/paste this prompt:

Role:
You are reviewing memory admission for a local LLM host.

Context:
Paste the host name, model, runtime, total VRAM, free host RAM, normal background work, and planned workload.

Task:
1. Separate the VRAM check from the host RAM check.
2. Set a host-specific free-memory floor from measured cold loads and background use.
3. Define the small request that proves model load before the measured workload.
4. Define the receipt for a blocked, failed, or completed run.

Output:
- A gate table with catalog, host RAM, model load, and workload rows.
- A pass or fail rule for each row.
- The fields to save when a gate stops the run.

Constraints:
- Do not treat total VRAM as free host RAM.
- Do not lower the memory floor only to make a run pass.
- Do not invent missing measurements.
- Mark every request that never started as not attempted.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Your Benchmark Row Never Saved the Driver Version

Patrick Hughes — Fri, 28 Aug 2026 16:05:12 +0000

Your Benchmark Row Never Saved the Driver Version

On 2026-08-27 I tried to attach a driver version to a benchmark row from June. I could not do it. The row does not carry one, and neither does any other row I have saved.

The short answer: Seven benchmark CSV files in my RTX 5090 notebook hold 17 rows, and none of the seven has a driver column. Two of the seven record a capture time. Three files carry a date and nothing finer. Two files carry neither a date nor a capture time. A row that did not save its own environment cannot support an environment claim later, and nvidia-smi output from 2026-08-27 is not a substitute.

Canonical URL: https://bmdpat.com/blog/local-llm-benchmark-row-missing-driver-version-2026

What did the file audit find?

I read every CSV in the RTX 5090 benchmark notebook on 2026-08-27 and checked the headers, not the numbers.

Seven files. Seventeen rows. Zero driver columns. Two files carry a captured_at field. Three files carry a date column and nothing finer. Two files carry neither. The header widths run from 9 columns to 17, so the files do not even agree on what a row is.

The engine column is the part that surprised me. One runtime appears under four labels across the seven files: Ollama 0.22.1, Ollama 0.31.1, ollama+code_verifier, and plain ollama. Two of those four hold a version. Four files use the bare word.

Why does a missing driver column matter?

I wrote earlier that a GPU driver is not a local LLM benchmark. That post argued a driver number printed in a report header does not show the driver changed model speed. This is the hole underneath it.

The report header is written when the report runs. The row is written when the test runs. On 2026-08-27 the header would say driver 610.88, because that is what nvidia-smi prints on this box right now. The June row ran on whatever driver was installed in June, and nothing recorded it.

So the header number binds to the report date and to nothing else. Copy it onto an older row and you have invented a fact.

Which version is the runtime actually running?

I ran ollama --version on the same box. It printed ollama version is 0.31.2 and then warned client version is 0.32.1.

Two answers, one machine, one command. On 2026-08-27 the server answering requests is on one version and the CLI I typed is on another. Two of my seven files did record a version, and they read Ollama 0.22.1 and Ollama 0.31.1, so at least three server builds have touched this notebook. A column that says ollama cannot tell those apart, and the difference is the kind that moves results. Runtime releases change defaults, and I keep a compatibility check between Ollama and llama.cpp for that reason.

Record the server version. The client version is the one you can read most easily and the one that did no work.

What four fields does every row need?

Four fields, written by the collector at capture time, never typed by hand later:

Runtime version at capture. The server build that answered the request.
Driver version at capture. Read from nvidia-smi in the same run, not the same week.
GPU name at capture. My fleet has more than one card, so the row must say which one.
Capture timestamp with timezone. A date is not enough when a driver updates midday.

None of these needs a new tool. Each is one command in the collector that already writes the row.

What do I do with the 17 rows I already have?

Do not backfill them. Writing 610.88 into a June row would make the file look complete and make the record false.

Mark them instead. I am adding an env_provenance field with the value unknown on every existing row. Those rows still hold their measured numbers, and they still support a narrow benchmark decision inside the file that produced them. What they cannot do is join a claim about a driver, a runtime release, or a card.

That is a real loss. Seventeen rows of measured work now sit outside every environment question I might want to ask. You pay for the missing column on the day you need it, which is always later.

How do I stop this from happening again?

Move the fields out of the writer's hands and into the collector. The script that calls the API is the only thing that knows the truth at the moment of the run.

Then fail the run when a field is empty. An empty driver column is not a small gap in a spreadsheet. It is a row that will look usable in six weeks and will not be.

Accompanying prompt

What the prompt does: It checks a benchmark file for the environment fields a row needs before that row can support any hardware or runtime claim.

Copy/paste this prompt:

Role:
You are a benchmark provenance auditor for local LLM test files.

Context:
I will give you the header and a few rows from one or more benchmark files.

Task:
1. List which environment fields each file records at capture time.
2. Name the missing fields: runtime version, driver version, GPU name, capture timestamp.
3. Flag any column whose values mix a bare tool name with a versioned tool name.
4. State which claims each file can and cannot support as it stands.

Output:
- A table of files and the environment fields each one records.
- A list of missing fields per file.
- The claims each file cannot support.
- The exact collector change needed to record each missing field.

Constraints:
- Do not fill a missing field from a current system reading.
- Do not treat a report header value as row provenance.
- Do not treat a date without a time as a capture timestamp.
- Do not delete rows that lack provenance. Mark them.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Your Local LLM CSV Needs a Schema Version

Patrick Hughes — Fri, 28 Aug 2026 16:05:08 +0000

Your Local LLM CSV Needs a Schema Version

My RTX 5090 benchmark notebook has measured data I cannot compare without hand repair. The models are not the problem. The CSV files disagree about what a benchmark row is.

The short answer: I audited seven local LLM benchmark CSVs on 2026-08-28. They contain 17 rows across four header widths, three power-field shapes, and zero schema_version fields. A benchmark file needs a versioned row contract before it needs another model run.

Canonical URL: https://bmdpat.com/blog/local-llm-benchmark-csv-schema-version-2026

What did the CSV audit find?

I read every CSV under my 5090 benchmark folder and compared the headers before I compared the results.

Seven files hold 17 data rows. Their headers have 9, 10, 15, or 17 columns. None has a schema_version field. None has a run_id field. Two files have captured_at; the other five do not.

Power has three shapes. Two files store watts_before and watts_after. Four files declare watts_avg, although one row leaves it blank. One file has no power column and puts 158W inside a notes sentence.

Those rows are still real measurements. The problem is that the reader must guess how each file maps into a common table.

Why does header drift break a benchmark notebook?

A CSV reader can join files only after someone decides which columns mean the same thing. watts_after is not the same measurement as watts_avg. A date is not the same as a timestamp. A runtime name is not a run identity.

Unit drift is the sharpest case. My 2026-06-14 and 2026-07-09 files record total_duration_s, eval_duration_s, and load_duration_s. My 2026-08-02 v2 and 2026-08-04 files record total_duration_ms, eval_duration_ms, and load_duration_ms. Same measurement, different unit, different name. The other three files carry no duration column at all. A reader that matches on the word duration and ignores the suffix is wrong by a factor of 1000.

Token rate splits the same way. Six files write tokens_per_second. One writes tok_per_sec. Those names map safely because both mean tokens per second. The values still do not compare, because that lone row is a five-task eval average and the others are single runs. A safe name mapping is not a safe comparison.

My daily 5090 report flattens all 17 rows into one display table. That makes the report readable, but it cannot recover fields that the collector never wrote. A blank power cell may mean not measured, not supported by that schema, or lost in free text.

I found the same boundary when I audited missing driver versions. The row must carry its own facts at capture time. A later report cannot add them without inventing history.

What belongs in a versioned row contract?

I now treat the CSV header as an API. Version one has four groups.

Identity fields name schema_version, run_id, and captured_at. Environment fields name the GPU, driver, runtime, runtime version, model, and quant. Workload fields record the task, input tokens, output tokens, context size, and stop reason. Result fields record wall time, load time, prompt time, generation time, token rate, VRAM, and power.

Each field needs one unit in its name or contract. load_duration_ms is clear. load_duration is not. watts_avg must mean an average sampled during the measured window, not the reading after the request ended.

The contract also needs explicit missing values. I use not_measured when the collector skipped a metric and not_supported when the runtime cannot expose it. An empty string is not enough because it hides the reason.

How should old benchmark rows move forward?

Do not rewrite old files to look current. Keep the original bytes and add a read-time adapter for each known header.

The adapter can map tokens_per_second and tok_per_sec into one output field because both source names remain visible in the mapping. It must not map watts_after into watts_avg. When no safe mapping exists, the normalized value stays unknown.

That preserves the measured result and the limits around it. It also lets a future collector write only the current schema while the report reader handles older files.

This is where power data needs a defined sample window. A schema name is useful only when its measurement rule is fixed too.

What should fail before a new row is saved?

The collector should reject a row with no schema version, run ID, capture timestamp, runtime version, or GPU name. It should reject numeric fields that carry units in the value instead of the contract. It should also reject unknown columns unless the schema version declares them.

Then test both sides. A complete current row should pass. A row copied from an old file should fail with the missing fields named in the error. That negative case matters because a permissive writer is how the notebook reached four header widths.

The fix is small. The collector owns one row contract. The report reader owns explicit adapters for old contracts. No human edits a benchmark row after the run.

Accompanying prompt

What the prompt does: It audits local LLM benchmark files for schema drift and writes a safe normalization plan without filling missing facts.

Copy/paste this prompt:

Role:
You are a schema auditor for local LLM benchmark files.

Context:
I will give you CSV headers and sample rows from several benchmark runs.

Task:
1. Group files by exact header shape.
2. Find fields that use different names, units, or measurement windows.
3. Separate safe mappings from mappings that would change meaning.
4. Draft one versioned row contract for new runs.

Output:
- A table of files, header widths, and schema groups.
- Safe field mappings with their source names preserved.
- Fields that must stay unknown.
- Required fields and fail rules for the new collector.

Constraints:
- Do not infer a missing value from a later system reading.
- Do not map point-in-time power to average power.
- Do not replace the original files.
- Use explicit units and missing-value reasons.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Preflight Local AI Before You Benchmark a Model

Patrick Hughes — Thu, 27 Aug 2026 03:45:42 +0000

Preflight Local AI Before You Benchmark a Model

My local AI benchmark failed before it loaded a model. The GPU was ready. The inference runtime was not.

The short answer: Test the toolchain, runtime build, and model load as separate gates. A setup failure is useful evidence, but it is not a model result. Record the exact failed gate and publish no speed number.

Canonical URL: https://bmdpat.com/blog/local-ai-runtime-preflight-before-benchmark-2026

What failed before the benchmark started?

On August 2, 2026, I tried to build WASTE on my RTX 5090 machine. WASTE is a C inference project with a Makefile. I wanted to test its NVMe expert offload path on hardware I own.

The test stopped at the build gate. cc, gcc, clang, and make were absent from the Windows host. WSL showed Ubuntu 22.04 as running, but each command returned Wsl/Service/E_UNEXPECTED with the text Catastrophic failure.

I recorded the failure in my 5090 report. I did not convert a model. I did not load weights. I did not get RAM, NVMe, latency, or token-rate measurements.

That distinction matters. A blank result row can mean that the model failed, the runtime failed, the host failed, or the test never started. Those outcomes need different fixes.

Which gates should run before model load?

I now split the preflight into three gates.

First, check the build tools. Ask the runtime what it needs. Then run version commands for each compiler, build tool, driver, and GPU API. Save stdout, stderr, and exit codes.

Second, build the runtime without a model. A clean build shows the source tree and host toolchain can produce the binary. It does not prove CUDA, model parsing, or inference works.

Third, load the smallest supported model. The goal is one known-good request, not a speed record. Capture the model file, quant, runtime commit, GPU, driver, load result, and stop reason.

Only then should the measured run begin.

Run these gates on every host, not once per project. My WASTE attempt stopped at the first gate, so the 5090 report carries the host state instead of an empty benchmark row. A gate that passes still earns a line in the receipt. That passed line is what lets a later failure point at the model rather than the machine.

Why is VRAM fit not enough?

VRAM answers one question: can the selected weights and runtime state fit on the GPU? It says nothing about whether the host can build or start the runtime.

I made the same point in VRAM Fit Is Not Runtime Support. That post dealt with feature and runtime support. This WASTE test exposed an earlier boundary. The binary did not exist, so VRAM never entered the decision.

This also changes how I read failed benchmark tables. In Why a Failed Local LLM Benchmark Row Still Matters, I kept a failed row because it showed a real limit. A preflight failure needs the same care, but it belongs in a different field.

Use preflight_status, build_status, load_status, and run_status. Do not put every failure under one error label.

What should the receipt contain?

My minimum receipt has four sections.

The host section names the operating system, GPU, driver, and available memory. The toolchain section stores the compiler and build-tool versions. The runtime section stores the repository URL, commit, build command, and build result. The model section stores the file, quant, load result, and measured-run result.

If a gate fails, I stop there. I keep the command, exit code, and error text. I mark all later measurements as not attempted.

This prevents two common errors. The first is to copy a number from another machine because the local run produced none. The second is to call a setup fault a model fault.

I did neither in the WASTE test. The report says that no inference numbers were produced. That is the only result the evidence supports.

When is the benchmark ready to start?

Start it after one small model completes one known-good request through the exact runtime binary that the measured run will use.

Keep that check cheap. It should catch a missing compiler, broken WSL session, bad CUDA link, unsupported model file, or failed load before the long test reserves the machine.

Then run the real workload. Capture input tokens, output tokens, wall time, token rate, VRAM, power, and stop reason. If the workload fails, the receipt can now place the fault after a passed preflight.

The point is not to avoid failed tests. The point is to make each failure name the boundary it reached.

Accompanying prompt

What the prompt does: It turns a local AI test plan into a fail-closed preflight with separate build, load, and measurement results.

Copy/paste this prompt:

Role:
You are a local AI benchmark preflight reviewer.

Context:
I will give you a host, inference runtime, model file, and test plan.

Task:
1. List the toolchain commands needed before the runtime build.
2. Define separate pass rules for build, model load, and measured run.
3. Stop at the first failed gate and save its command, exit code, and error text.
4. Mark every later metric as not attempted.

Output:
- A gate table with toolchain, build, load, and run rows.
- The exact evidence needed for each pass.
- One final status: ready to measure or blocked at a named gate.

Constraints:
- Do not copy measurements from another machine.
- Do not call a setup failure a model failure.
- Do not publish speed, memory, or power numbers unless the measured run started.
- Keep the runtime commit, model file, quant, GPU, and driver in the receipt.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

49W Average Hid a 338W Burst on Gemma 26B

Patrick Hughes — Wed, 26 Aug 2026 22:31:32 +0000

49W Average Hid a 338W Burst on Gemma 26B

A 49-watt average on my RTX 5090 looked cheap. The same Gemma 4 26B Q4_K_M run peaked at 338 watts.

The short answer: Keep GPU watts average and GPU watts max on the same row. I only divide watts by output tokens per second after I name the sample window. Naive math on this long-context run gives 0.272 joules per output token from the average and 1.876 from the max. Those are not two valid answers. They are one missing window.

Canonical URL: https://bmdpat.com/blog/local-llm-gpu-watt-avg-vs-max-2026

What did the RTX 5090 receipt record?

The August 20, 2026 5090 Report lists three gemma4:26b Q4_K_M rows from Ollama 0.31.1. Each note stores watts avg/max during the run.

Workload	Input	Output	Output rate	Watts avg / max
short-gen-256	53	256	198.81 tok/s	43 / 187
long-context-summarize	4,470	128	180.18 tok/s	49 / 338
agent-code-task-512	76	512	207.41 tok/s	48 / 320

The long-context row is the loud one. Average power was 49 W. Max power was 338 W. That is a 6.9x gap on one run of one Q4_K_M file.

nvidia-smi reports power.draw as a sampled GPU power reading, not as wall power for the whole PC. I treat it that way. The receipt does not name the sample interval. It also does not say whether idle samples sat in the average.

Why is the average a weak energy input?

On August 24, 2026 I showed how to turn a generation-phase average into joules per output token. Divide watts by output tokens per second. On a different workshop row, num_batch=256 used 187.6 W at 215.7 tok/s, or about 0.870 J per output token. That write-up is How to Calculate Local LLM Energy per Token.

The formula still needs a named window. If I plug the long-context average into the same formula, 49 / 180.18 = 0.272. If I plug the max into it, 338 / 180.18 = 1.876. I round those to three decimal places only for the table. I do not keep either number as the energy result for that row.

The 0.272 figure is 3.2x lower than the workshop 0.870 figure. The 1.876 figure is 2.2x higher. Same GPU. Same 26B class. Different watt windows. A ranking built from those mixed averages would pick a winner for the wrong reason.

How should I use the max reading?

Max power is a burst flag. It is not a substitute average.

338 W on a 575 W card is a real spike. It is not a 49 W workload. I keep the max so a later reader can see that the GPU left the low-power band. I still need a generation-phase average before I compute joules per output token.

The short-gen row shows the same split at a smaller scale: 43 W average, 187 W max, 198.81 tok/s. The agent-code row recorded 48 W average and 320 W max at 207.41 tok/s. Across these three Q4_K_M rows, the average stayed in a 43-49 W band while the max moved from 187 W to 338 W.

I do not know the cause from this receipt. Prompt work, decode work, and idle samples can share one "during run" average. The input and output benchmark split already treats those phases as separate clocks. Power needs the same split.

What belongs in the next power receipt?

Keep the raw pair. Save watts average and watts max. Save the model, quant, runtime, GPU, workload, context, input tokens, output tokens, and output rate. Then name three more fields:

The sensor (nvidia-smi power.draw, wall meter, or other).
The sample interval.
The phase covered (full request, prompt eval only, or generation only).

If any of those three is missing, I can still compare speed and VRAM. I cannot compare energy per token with a row that named the window.

My earlier power-data note argued that speed without watts hides operating cost. This row adds the next trap: watts without a window hides the burst.

For this long-context Gemma run, I keep 49 W and 338 W together. I do not publish a joules-per-token winner from them. The next useful experiment is a generation-only average on the same prompt, with the sample interval written on the row.

Accompanying prompt

What the prompt does: It checks whether a local LLM power row has a named watt window before anyone computes energy per token.

Copy/paste this prompt:

Role:
You are a local LLM power-receipt reviewer.

Context:
I will give you benchmark rows with model, quant, runtime, GPU,
workload, token counts, output rate, watts average, watts max,
and any sample-window notes.

Task:
1. Keep rows that include both watts average and watts max.
2. Mark energy-per-token math as blocked when the sample interval
   or phase (full request, prompt, generation) is missing.
3. If the window is named, compute joules per output token as
   generation-phase average watts divided by output tok/s.
4. Report the avg-to-max ratio as a burst flag, not as energy.

Output:
- A table with raw watt fields and the window status.
- One line that says compute, block, or rerun.
- The missing fields needed for a valid energy figure.

Constraints:
- Do not treat GPU watts as wall power.
- Do not use watts max as the energy input.
- Do not compare energy across rows with unnamed windows.
- Keep full precision for the calculation and round only the shown result.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

How to Calculate Local LLM Energy per Token

Patrick Hughes — Tue, 25 Aug 2026 16:05:12 +0000

How to Calculate Local LLM Energy per Token

The faster of two matched gemma4:26b settings was not the most efficient. One run gained 1.3% output speed and used 23.3% more energy per output token.

The short answer: Divide average GPU watts by output tokens per second. The result is joules per output token. In my RTX 5090 test, num_batch=256 used about 0.870 joules per output token. The 512 setting used about 1.072. Speed alone hid that difference.

Canonical URL: https://bmdpat.com/blog/local-llm-energy-per-token-2026

How do watts become joules per token?

NIST defines a watt as one joule per second. A model rate is output tokens per second. Divide the first rate by the second rate, and seconds cancel.

joules per output token = average GPU watts / output tokens per second

This number answers a narrow question. It estimates GPU energy during the measured generation period. It does not include the rest of the computer, model load time, prompt work, cooling, or idle power.

What did the two RTX 5090 rows show?

The August 20, 2026 5090 Report includes two gemma4:26b workshop rows. Both used Ollama, a 2,048-token context, 91 input tokens, and 482 output tokens. Both passed the fixed tasks.

Setting	Output rate	Average GPU power	Energy per output token
`num_batch=256`	215.7 tok/s	187.6 W	0.870 J/token
`num_batch=512`	218.5 tok/s	234.3 W	1.072 J/token

The arithmetic is direct. 187.6 / 215.7 = 0.8697. 234.3 / 218.5 = 1.0723. I round each result to three decimal places in the table.

The 512 row generated 1.3% faster. Average GPU power rose 24.9%. Energy per output token rose 23.3%.

Why is energy per token better than watts alone?

Watts measure a rate of energy use. They do not say how much work finished during that time. A 200-watt run can use less total energy than a 150-watt run if it finishes much sooner.

Tokens per second have the opposite gap. They say how fast output arrived, but not how much power the GPU drew to produce it. The fastest row can also be the least efficient row.

Joules per token join those two fields. The unit still does not say whether the answer was correct. I only compare rows after the task check passes. My earlier failed-row note shows why a fast row with a failed task stays out of production.

Can I turn this into electricity cost?

Yes, but I label the estimate. One million output tokens at 0.870 joules each equals 870,000 joules. Divide by 3.6 million joules per kilowatt-hour. The generation estimate is about 0.242 kWh.

The 512 row reaches about 0.298 kWh per million output tokens by the same method. That is a difference of about 0.056 kWh for the measured generation phase.

I do not call either number a full local inference cost. A full receipt also needs wall power, prompt work, load time, idle time, hardware cost, and the electricity rate. My local AI workbench note uses the same boundary when it converts one measured run into a rough electricity estimate.

When should I use this metric?

I use it after two rows pass the same task check. I keep the model, quant, runtime, workload, context, prompt, and output cap fixed. I also record the token counts so a later reader can see whether the runs did the same work.

Then I ask three questions. Did the task pass? Did the faster setting remove useful wait time? Did energy per output token improve or get worse?

For this workload, num_batch=256 is the better measured setting. It produced 98.7% of the 512 row's output rate while using less average GPU power and less energy per output token. That is a result for this RTX 5090 test, not a default for every model or GPU.

What should the benchmark receipt keep?

Keep the raw fields before the derived metric. Save the model, quant, runtime version, GPU, workload, context, output cap, input tokens, output tokens, output rate, average GPU power, and task result.

Calculate joules per output token from those saved values. Do not save only the rounded answer. The raw row lets another builder check the math and change the boundary.

Also name what the power sensor measured. GPU power is not wall power. A generation average is not a whole-request average. A clear label stops a small metric from becoming a large claim.

Accompanying prompt

What the prompt does: It calculates energy per output token from local LLM rows and rejects comparisons that lack a task check.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
I will give you benchmark rows with model, quant, runtime, GPU,
workload, context, token counts, output rate, average GPU power,
and task result.

Task:
1. Keep only rows that passed the same task check.
2. Calculate joules per output token as watts divided by output tok/s.
3. Compare speed, average GPU power, and energy per output token.
4. State which measurement boundaries remain outside the result.

Output:
- A table with the raw fields and calculated joules per token.
- One setting choice for this measured workload.
- A list of missing fields needed for a full cost estimate.

Constraints:
- Do not treat GPU power as wall power.
- Do not include failed or unmatched task rows in the recommendation.
- Keep full precision for the calculation and round only the shown result.
- Say unknown when average power or output rate is missing.

Copy the block above.

Get the next measured local AI build note from The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Why a Failed Local LLM Benchmark Row Still Matters

Patrick Hughes — Mon, 24 Aug 2026 16:05:11 +0000

Why a Failed Local LLM Benchmark Row Still Matters

I used to delete failed benchmark rows. That was wrong. A failed row marks the test boundary: the point where a fixed check stopped passing.

The short answer: Keep failed quality rows in the receipt. On my RTX 5090, one gemma4:26b setting hit 156.9 tokens per second and scored 0% on the fixed tasks. Two nearby settings scored 100%. The failed row is what makes that line visible.

What did the failed local model row show?

The August 20, 2026 5090 Report lists 17 benchmark rows and no runner failures. One gemma4:26b row ran on Ollama at 2,048 context with num_batch=128. Speed was 156.9 tokens per second. Fixed-task quality was 0%.

The next two rows kept the same model and the same workload label. The num_batch=256 row reached 215.7 tokens per second at 100% quality. The num_batch=512 row reached 218.5 tokens per second at 100% quality. Average power was 187.6 W and 234.3 W.

That does not prove 128 always fails. One machine, one task set: enough to reject this row, not enough to generalize.

Why keep a zero-quality row?

A passing row cannot answer one question: where did the test stop working?

Delete the 0% row and the report shows two passing speeds. The reader never learns that a nearby setting failed the same check. Someone then lowers num_batch to cut memory pressure and assumes the task still passes.

The row also guards future comparisons. Say a new runtime reaches 200 tokens per second at num_batch=128. Then I ask: did quality improve, or did the test change? Without the old row I lose that check. The same logic drives my stop reason field.

What changed between the rows?

The batch setting changed. The model stayed gemma4:26b. The context stayed 2,048 tokens. The workload label stayed the workshop mix.

Two more fields moved, and the receipt shows it. The failed row logged 100 input and 600 output tokens. Both passing rows logged 91 input and 482 output. The report also notes the 128 row came from a fallback pick, not a deliberate choice.

So this is not a clean one-variable test. I could leave that out and tell a neater story. The neater story would be false. Three fields moved, so the honest label is a new experiment, not a controlled comparison.

That is the second argument for keeping the failed row. The full field set showed me what my own test failed to hold fixed. A trimmed report would have hidden that too.

Quality still moved from 0% at 128 to 100% at 256 and 512. Speed rose as well. The report gives no average power for the failed row, so I leave that field empty instead of estimating it.

Speed alone cannot settle any of this. My VRAM fit notes cover whether a model loads, and my phase timing receipt covers why a total time hides the slow phase. This post covers a third question: did the fixed task pass?

How do I use failed rows in a local eval?

Save the exact prompt, model, runtime, context, setting, output, and verifier result. A percentage without the task definition is not a receipt.

Change one variable at a time. If several fields move, call it a new experiment, as I just did with the 128 row.

Keep the failed row next to the next result. Do not overwrite it. Add one note that states what failed and what changed.

Pick the production setting from passing rows only. Treat the failed row as a boundary marker, not as a candidate.

What should a local LLM benchmark report?

My minimum row carries these fields:

Model and quantization.
Runtime and version.
Workload and context.
The setting under test.
Tokens per second.
Power and VRAM when measured.
The task result.
The stop reason or verifier note.

The last two fields change how I read every number above them. A fast row that fails the task is not a candidate. A slower row that passes may be the better production choice. The receipt should make that call obvious.

Accompanying prompt

What the prompt does: It turns a local benchmark log into a row set that keeps failed tests and exposes the setting boundary.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
Model: [model and quantization]
Runtime: [runtime and version]
GPU: [GPU name and memory]
Workload: [prompt set, context, and setting]
Run log: [paste the log or report path]

Task:
1. Extract speed, power, VRAM, output, and verifier fields.
2. Mark each row as pass, fail, or unverified.
3. Keep failed rows in the comparison.
4. Identify the smallest setting change that moves a row from fail to pass.

Output:
- A markdown table with one row per run.
- A short note for every failed row.
- One production-setting recommendation from passing rows only.

Constraints:
- Keep exact measured values.
- Do not estimate missing power or VRAM values.
- Do not call a speed result a quality result.

Copy the block above.

Read the next measured local AI build note by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

My local LLM eval hid four token caps

Patrick Hughes — Sat, 22 Aug 2026 16:05:18 +0000

My local LLM eval hid four token caps

I compared two local eval scores and treated them as one test. They were not. One path stopped at 50 tokens. Another allowed 256.

Short answer: My 5090-rig eval stack uses four output caps: 50, 180, 192, and 256 tokens, under three field names. A score that omits the cap is not a model result. Canonical URL: https://bmdpat.com/blog/local-llm-eval-four-token-caps-2026

Where are the four caps in my eval code?

I listed every default output limit in 5090-rig. Four values are still in the source and logs.

The router gate posts to /v1/chat/completions with "max_tokens": 50. That lives in 5090-rig/phase1/gates/gate_01_router.py. The prompt asks for one Python function. Fifty tokens can be enough for that job.

Phase 1 and phase 2 call Ollama /api/generate with "num_predict": 180. That default is in 5090-rig/phase1/phase1_eval.py and again in 5090-rig/phase2/data_flywheel.py. The Ollama generate API documents num_predict as the max tokens to generate after the prompt.

The sweep probe uses a different default. 5090-rig/sweep/probe.py sets num_predict=192 on ollama_generate. Same API. Different budget. A table that mixes probe rows with phase-1 rows is already mixing caps.

Phase 3 is a third API. I did not find max_new_tokens in the Python source. The run log 5090-rig/phase3/SCRATCH/phase3-final-run.log shows Hugging Face generate used max_new_tokens=256. The same lines say max_length=131072 was also set, and that max_new_tokens takes precedence. Hugging Face text generation is the independent page for that field.

Why do four caps hide a model ranking?

The field names do not match. max_tokens, num_predict, and max_new_tokens all mean "stop after this many new tokens." A reader who only sees a score cannot tell which one ran.

The jobs do not match either. A 50-token router check is a short function test. A 180-token Ollama generate can write more. A 256-token Hugging Face generate can write more still. Ranking those scores ranks the budget.

I already covered the receipt side of this in The 26B Model Hit the Cap. The 8B Finished.. That post is about done_reason. This post is the other half: find the configured cap in the eval code before you trust the table.

A fast row can still be a truncated row. Honest local eval numbers start with the settings that shaped the answer.

What should every eval row record?

I now want these fields on every local eval row:

The cap value (50, 180, 192, or 256 in this stack).
The field name (max_tokens, num_predict, or max_new_tokens).
The API path (/v1/chat/completions, Ollama /api/generate, or Hugging Face generate).
Tokens produced.
Stop reason when the runtime gives one.
Task pass or fail.

If two rows differ on any of those, they are not the same test. Do not average them. Do not pick a winner.

The speed and quality gate already treats quality as a release check. The cap belongs in that check. A pass under 256 tokens is not a pass under 50 tokens.

Short caps are useful. The failure is silent mixing. If production uses 50, the table must say 50.

How do I test a model against its production cap?

I use two runs on the same prompt, model, quant, and runtime path.

Run 1 uses the production cap for that path. For the router gate, that is 50. For phase 1, that is 180.

Run 2 uses a larger cap on the same path. If run 1 fails the task and run 2 passes, the model needed more tokens than production allows. That is a product limit. It is not a mystery quality drop.

Do not raise the eval cap and then ship the tight one. If production will send num_predict=180, the keep/reject decision must use 180. The large-cap run is evidence, not a substitute score.

A model that hits the cap can look fast because it never had to finish. Record the cap, the tokens produced, and the task check together.

Accompanying prompt

What the prompt does: Lists every output-token cap in a local eval codebase and flags scores that omit the cap.

Copy/paste this prompt:

Role:
You audit a local LLM eval codebase for output-token caps.

Context:
Paste the eval files, configs, and one result table.

Task:
1. Find every max_tokens, num_predict, max_new_tokens, and n_predict default.
2. Group them by API path and by file.
3. Match each result-table row to the cap that produced it.
4. Flag any score whose cap, stop reason, or token count is missing.

Output:
- A table of file, field name, cap value, and API path.
- Rows that mixed two caps.
- The two-run check: production cap, then a larger cap on the same path.

Constraints:
- Use the exact numbers from the files.
- Do not invent a cap that is not in the source or logs.
- Do not rank models across different caps.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.