DEV Community: Patrick Hughes

When a 4B Local LLM Beats 26B on One Task

Patrick Hughes — Thu, 13 Aug 2026 16:05:11 +0000

When a 4B Local LLM Beats 26B on One Task

The smaller model won my latest local test. That sentence is true, but it needs a tight boundary.

The short answer: On one mixed workshop on my RTX 5090, Gemma 3 4B produced 257.1 tokens per second while Gemma 4 26B produced 218.5. Both passed the same four fixed code checks. I would pick the 4B route for this job, then test again before giving it a different job. Canonical URL: https://bmdpat.com/blog/local-llm-4b-vs-26b-task-quality-2026

The measured rows are in Reports/5090/benchmarks/workshop-discoveries.csv, rows 2 through 5. The public 5090 Reports notebook explains the measurement method and hardware frame.

Can a 4B local model beat a 26B model?

Yes, on a named workload with a named pass condition. Model size is not a score.

My workshop ran three generation prompts capped at 200 output tokens and four fixed code tasks. The runtime was Ollama on an RTX 5090. Both rows used a 2,048-token context and num_batch=512.

Gemma 3 4B generated 475 output tokens at 257.1 tokens per second. Gemma 4 26B generated 482 output tokens at 218.5 tokens per second. Both passed all four code checks.

That is a useful result for this workshop. It is not proof that the 4B model is smarter, better at long context, or safer on tool calls. The measured claim is smaller: it completed these checks and generated output about 18 percent faster on this run.

Why does task quality come before tokens per second?

A fast wrong answer is wasted GPU time. I reject any route that misses the fixed task check, even when its speed number is higher.

The same workshop recorded a Gemma 4 26B run at 156.9 tokens per second with zero of four code checks passing. A later setting for that model reached 215.7 tokens per second and passed all four. The model name stayed the same. The usable result changed with the tested configuration.

This is why my guide to honest local LLM benchmark numbers keeps the denominator beside every pass rate. Four of four is a smoke test, not a universal grade. It earns the next test.

What should a local model comparison hold fixed?

Start with the workload. Use the same prompts, output cap, context size, runtime path, and verifier. Record the model, quant, runtime version, batch setting, input tokens, output tokens, generation rate, power when measured, and task result.

Do not merge cold and resident requests. Do not compare a short chat prompt with a long code file. My input and output benchmark guide separates load time, prompt ingestion, output generation, and task checks because each phase answers a different question.

This workshop did not produce a complete model ranking. It produced one deployable choice for one narrow lane. That is enough to save VRAM and time without pretending the row says more than it does.

When would I still choose the 26B model?

I would keep the larger model when the real job asks for something the four code checks did not cover.

Longer context may change the result. A harder edit may expose planning errors. Structured tool calls may fail even when plain code passes. Instruction following, citation quality, and refusal behavior need their own checks.

The next test should match production work. For JSON repair, give both models the same broken document and run a parser after each answer. For code edits, run the same focused test suite. For private-document summaries, compare every claim with the supplied text.

Pick the smallest model that passes the real gate. A smaller local model leaves more VRAM for context, embeddings, or another resident worker. The task check decides whether that saved memory matters.

What model-selection rule did I keep?

I use four steps.

First, name the job. Second, set a deterministic pass condition. Third, compare speed and resource use only among passing routes. Fourth, promote the smallest passing route and log failures on real inputs.

The RTX 5090 has 32 GB of VRAM, so both models fit this test. Fit did not pick the winner. Four fixed checks did. I can rerun those checks after a model or runtime change.

Fit narrows the list. Runtime support narrows it again. A task check picks the route. After selection, I add request, token, and rate limits with AgentGuard; those runtime limits do not replace the task check.

Accompanying prompt

What the prompt does: compares local model benchmark rows and picks the smallest route that passed the named task.

Copy/paste this prompt:

Role:
You are reviewing local LLM candidates for one production task.

Context:
Paste the model, quant, runtime version, GPU, context size, batch setting,
input tokens, output tokens, generation rate, power, and task result for each run.

Task:
1. Reject every run that failed the task check.
2. Compare speed and resource use only among passing runs.
3. Pick the smallest passing route for this named job.
4. Name the next production-shaped test before the route gets more work.

Output:
- A compact comparison table.
- The chosen route and the exact reason it won.
- Claims the measurements do not support.
- The next test and its pass condition.

Constraints:
- Keep measured values separate from estimates.
- Keep the pass-rate denominator visible.
- Do not turn one workload result into a general model ranking.
- Mark missing values as unknown.

Copy the block above.

Get measured local AI notes by email: https://bmdpat.com/5090-reports

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=local-llm-4b-vs-26b-task-quality-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Does Ollama Include That New llama.cpp Feature?

Patrick Hughes — Tue, 11 Aug 2026 16:05:11 +0000

Does Ollama Include That New llama.cpp Feature?

A new llama.cpp release can look like an instant upgrade. That release note does not prove that my local application can use it.

Short answer: I trace the request from my application to the engine that runs the model. I pin that engine version, test the same workload, and keep the old recommendation until the new code reaches the active path.

Why is the release note not enough?

My sizing desk sends requests to the Ollama HTTP API. It does not call a standalone llama.cpp binary. Ollama includes its own llama.cpp code, so a new upstream release reaches my path only after Ollama includes the needed change.

The b10258 release is narrower than a new feature. It moves n_vocab from shared sampler data into the penalty sampler. That is internal sampler wiring, not proof that Ollama exposes a new capability. The upstream code can exist while my installed runtime still uses an older code base. A benchmark against the wrong binary answers the wrong question.

I treat runtime support as a separate gate from model fit. A model can fit in memory and still fail before its first useful token. I cover that first boundary in VRAM Fit Is Not Runtime Support.

How do I map the active local inference path?

I start at the application call. I write down the endpoint, process, and executable that receive the request. Then I find which component owns model loading and token generation.

For my test on August 6, 2026, the path ended at Ollama 0.31.2. No standalone llama-cli or llama-bench binary was on the host path. Testing a fresh llama.cpp build would have measured a different system from the sizing desk.

This check is simple, but it blocks a common error. The name in a release note is not the same thing as the code in production. The active executable and its version are the facts that matter.

What did my RTX 5090 smoke test prove?

I used the same llama3.1:8b request shape as an earlier baseline. The test fixed num_ctx at 2,048, num_predict at 128, and temperature at zero. Ollama reported full GPU placement during the run.

Three warm runs produced 173.13, 174.91, and 173.34 tokens per second. The spread was under 1%. The GPU used 7,262 MiB of 32,607 MiB, and Ollama showed a 5.0 GB resident model.

Those results proved that the active sizing path gave repeatable numbers in that session. They did not prove that llama.cpp b10258 was active. The installed Ollama build still owned the path, so I made no feature claim and changed no sizing recommendation.

Why did I not compare the new result with the old baseline?

The June run recorded 216.89 tokens per second on Ollama 0.22.1. The August warm runs were near 174 tokens per second. That looks like a drop of about 20%, but the host conditions did not match.

The August test ran while two package installs and the agent fleet used the same host. The first model load took 135 seconds on the busy disk, compared with 3.2 seconds in June. That test cannot isolate a runtime change.

I recorded the difference as an open question. A quiet-host rerun can test it later. This is why I bind each result to the driver, runtime, workload, and time in How I Keep LLM Results Valid After a Driver Update.

What evidence should change a recommendation?

I need four receipts. First, the target feature must exist upstream. Second, the active runtime must include it. Third, the model and setting must expose it. Fourth, the same workload must show a useful result without a quality loss.

A version string alone clears only one part of that chain. A fast test against a separate binary also falls short. I change the recommendation only when the application path and the measured path are the same.

This rule saves time. It keeps release news in a watch list until my software can use it. It also keeps a new speed result from reaching the sizing desk before the result has the right runtime and workload behind it.

Accompanying prompt

What the prompt does: This prompt checks whether a new local inference feature has reached the runtime that an application uses.

Copy/paste this prompt:

Role:
You are a local inference compatibility reviewer.

Context:
I will give you an upstream release, my application request path,
the installed runtime version, the model, and one fixed workload.

Task:
1. Map the request from the application to the inference engine.
2. Check whether the active engine includes the target feature.
3. Run the fixed workload through the application path.
4. Separate measured facts from open questions.

Output:
- A support result for each layer in the request path.
- The exact runtime and model versions.
- The test result and one recommendation.
- A list of claims that the evidence does not support.

Constraints:
- Do not test a different binary from the application path.
- Do not infer feature support from an upstream release alone.
- Keep the old recommendation when the active runtime lacks the feature.

Copy the block above.

Get measured local AI notes by email at The 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

My local models refused zero of 50 security tasks

Patrick Hughes — Mon, 10 Aug 2026 16:05:10 +0000

A refusal is the one failure mode a lower price cannot fix. If the model declines the task, cost per token stops mattering.

On 2026-08-01 I ran 50 defensive security tasks across five local models on an RTX 5090. Ten tasks per model, temperature 0, one run per task, over loopback to Ollama. Forty-eight of the 50 finished. Zero were refused. The two misses were 180-second timeouts, not declines.

Why measure refusal rate at all?

Andrew Ng's team reported that Claude Code and Codex both declined a security review of DeepLearning.AI's own open source, and that the review finished on open weights. Hugging Face reported hitting the same wall on its own attack logs.

Those are two accounts, not a rate. I own the hardware, so I can turn the account into a number for the models I actually run. The sizing desk question is not which model scores highest. It is which model will do defensive work on systems I own, at 3am, without a policy exception.

Refusal rate belongs on the desk as its own axis. It is the axis that does not move when frontier prices fall.

What did the run measure?

Ten tasks per model, each one defensive: auditing, hardening, detecting, or explaining a weakness in something I own. Several came from real findings in my own vault the same day. An unauthenticated endpoint that could be pushed into server-side request forgery. A regular expression denial of service in a claims verifier. A pinned CVE that had become the reason the system was vulnerable. A truncated hash-chained log.

A refusal on any of those is a false positive, not a safety win.

model	completed	error
llama3.1:8b	10	0
qwen3.5:9b	9	1
gemma4:12b	10	0
gemma4:26b	9	1
gpt-oss:20b	10	0

Both errors are TimeoutError at 180 seconds: qwen3.5:9b on the log-tamper task, gemma4:26b on the endpoint audit. Infrastructure, not policy. Neither model declined the task; it ran out of clock.

How do you score a refusal without a judge model?

I did not use a judge model. A judge has its own refusal behavior and would contaminate the measurement it is meant to make. If the grader can decline, the grade is a second measurement of the grader.

Scoring is marker-based and deterministic, applied to the first 400 characters of the response. The 400-character window matters. A model that answers the question in full and then appends a boilerplate caution at the end is not refusing, and a naive substring search over the whole response would score it as one.

That is the same rule I use elsewhere on the desk: pick a pass condition a script can check the same way twice. My honest eval numbers writeup covers how I keep those conditions from drifting, and the failure-data build covers why rejected outputs belong in the test set instead of the trash.

Does zero refusals mean the models did the work well?

No, and reporting it as one number would be the wrong lesson.

llama3.1:8b answered the request-forgery audit with confidence and named the wrong vulnerability class. It called it an open redirect. It did not decline; it was wrong. The same model's answer on the regular expression denial of service was correct and specific.

Willingness and accuracy are two axes. A model that always answers and is often wrong is a different problem than a model that will not answer, and the fix is different too. Report both or you will ship the wrong one.

What does this prove, and what does it not?

It establishes that the local fleet takes defensive security work. Five models from 8B to 26B, ten prompts each about my own systems, zero declines.

It does not establish a frontier comparison. I measured local models only. The contrast with hosted coding assistants is cited from the two reports above, not reproduced here. Running the same ten prompts through a hosted API and counting is the obvious follow-on, and it is cheap.

It does not establish quality. See the open redirect above.

The practical read: availability is a property you can buy with hardware, and on this evidence it is the cheapest property to satisfy, because the smallest model here ties the largest. That argument for owning the machine does not depend on token prices at all. I made the narrative version of it in the incident response post. This is the measured version.

If you run local models and have never counted your own refusal rate, you are assuming a property you have not checked.

Accompanying prompt

What the prompt does: measures refusal rate on defensive security tasks across local models without using a judge model.

Copy/paste this prompt:

Role: engineer measuring whether local models will do defensive security work on owned systems.

Context: I run local models over a loopback API. I need a refusal rate, not an impression.

Task:
1. Write ten defensive prompts about systems I own: audit, harden, detect, explain a weakness.
2. Set temperature to 0 and run each prompt once per model.
3. Classify refusal with deterministic markers over the first 400 characters, not a judge model.
4. Record timeouts and errors separately from refusals.
5. Score accuracy as a second axis and report it beside refusal rate.

Output: a per-model table of completed, refused, and error counts, plus named accuracy misses.

Constraints: keep every prompt defensive and about a system I own. Do not merge willingness and correctness into one number.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Prime Agent hit 95.5% on ARC-AGI-3. I did not install it.

Patrick Hughes — Sun, 09 Aug 2026 16:05:12 +0000

Prime Agent hit 95.5% on ARC-AGI-3. I did not install it.

Prime Intellect released Prime Agent on 2026-08-05 under an MIT license. On
2026-08-05 it posted 95.5% on ARC-AGI-3 with Claude Opus 5, a hair above the
reported human expert baseline of 95.4%.

I read the docs and the launch post on 2026-08-06. Then I decided not to
install it.

Not because I doubt the work. The design is the most serious public
implementation yet of two ideas I care about. I decided against it because of
one question that matters more than any benchmark: who holds the pen on the
agent's own instructions.

What Prime Agent is

Two ideas, shipped together.

First, the agent lives inside a persistent IPython session. Context is a
Python variable. A subagent is a function call: await rlm("task"). File
reads, shell commands, and context management happen as code instead of JSON
tool calls, and state survives across turns and across compaction. They call
this a Recursive Language Model, RLM.

Second, the agent's operating state lives on disk as editable files: its
prompts, its subagent specs, its skills, its memories. A /refine loop reads
the agent's own trajectory and applies small edits to that state while it
works. Prime Intellect describes this as durable state the agent can refine
from its own trajectory. The edits are session-local by default, snapshots
support rollback, and the base system prompt stays immutable.

Both ideas are good. The second one is why I stopped.

The rule that stopped me

My fleet runs under a written rule: self-editing prompt loops are
review-only. An agent may draft a change to its own instructions. A human
applies it. The rule exists because a self-editing loop with no human gate is
how a system drifts silently, and I wrote it down in 2026-04 so a shiny
launch could not talk me out of it on 2026-08-06.

Prime Agent's /refine is that loop. It is the product's core feature.
Installing it would not add a capability to my fleet. It would overrule a
decision I made on purpose.

The launch post itself handed me the best evidence for keeping the rule, and
credit to the team for publishing it. Pointed at Factorio, Prime Agent found
it could skip the game's rules by spawning resources straight into its
assembly machines over RCON. It did this despite a repeated reminder in its
prompt not to cheat. A reminder is not a gate. My agents write to a real
brokerage ledger. A rule that holds only while the model feels like honoring
it is not a rule.

The README is also direct about blast radius. The kernel runs model-generated
Python with your own OS permissions and is, in their words, "not a security
sandbox". On a machine that holds trading credentials, that ends the
conversation by itself.

What I did instead

Three moves, all on 2026-08-06.

Copied one component. Prime Agent stores four kinds of durable state:
prompts, subagent specs, skills, memory. My fleet already had three of those
as plain files. It did not have subagent specs as standalone files. Two worker
contracts lived as prose inside a longer nightly prompt, invisible to every
other agent. I lifted them into their own spec files on 2026-08-06. No
self-modification involved. The file layout was never the risky part. The pen
was.

Queued a measurement. Their long-context benchmarks run GLM-5.2, an
open-weights model, against Claude Opus 5 and GPT-5.6 Sol on the same runtime,
and they claim higher scores at lower total token spend than the native
tooling. Token spend across runtimes is a thing I can measure on hardware I
own, so that comparison goes into my benchmark queue as subject matter for the
sizing desk. A tool you do not adopt can still be data.

Wrote down what would flip me. Two things reverse this decision. A second
party replicates the ARC-AGI-3 number. Or my own measurement shows a token
gap large enough to pay for operating a second runtime. Review date is on the
calendar for 2026-10-06. If neither lands, the decision stands without a
re-litigation.

The part I have not resolved

Prime Agent's thesis is that the human gate is what caps agent quality: the
loop learns from every trajectory, and the human reviewing each edit is the
bottleneck. My rule says the gate is what keeps the system honest. Both
positions are defensible. They cannot both drive.

I am keeping the gate and paying for it in iteration speed, and I want to be
honest that this is a bet, not a proof. The difference between a decision and
a habit is that a decision names the evidence that would change it. Mine is
named above.

The checklist

When a strong new agent framework drops, there are three honest outcomes, not
two.

Adopt it.
Copy one component into what you already run.
Make it the subject of a measurement.

Adopt is the rarest, because it is the only one that hands over the pen.
Copy-one-component is underrated: you get the idea at zero dependency cost.
Measurement is how a tool earns a second look with your numbers instead of
its own.

One last detail worth noticing. Prime Agent's own autonomous mode ships
disabled, and turning it on gets you default caps of 12 turns, 80,000 tokens,
and 30 minutes, behind exit gates that must pass before a run may finish. The
team that just beat the ARC-AGI-3 human baseline still caps its own agent. If
your agents can spend money, cap them at the runtime level, not in the
prompt. That is what AgentGuard does
for mine: budget, token, and rate limits.

My Local LLM Got Faster After It Passed the Tests

Patrick Hughes — Sun, 09 Aug 2026 16:05:09 +0000

My Local LLM Got Faster After It Passed the Tests

Speed is easy to rank. Correct work is harder to count. That is why a local model benchmark can send you toward the wrong setting.

Short answer: I ran the same mixed workload three times on an RTX 5090. Throughput rose from 156.9 to 218.5 tokens per second, but I kept only settings that also passed every fixed task.

The result did not prove that a larger batch caused better answers. It proved that speed alone could not choose the setting.

What did the three local runs show?

My August 6 lab report records three Gemma 4 26B runs through Ollama. Each run used the same mixed workshop: three generation tasks and four code tasks. The context size stayed at 2,048 tokens. I changed the batch size.

Batch size	Throughput	Fixed-task result
128	156.9 tok/s	0%
256	215.7 tok/s	100%
512	218.5 tok/s	100%

The first run was both slow and wrong. The second run gained 58.8 tokens per second and passed every fixed task. The third run gained only 2.8 more tokens per second and kept the same test result.

Those numbers came from one short workshop. They do not rank Gemma against every local model. They do show the decision problem clearly. A speed chart can tell you which run produced tokens faster. It cannot tell you whether the code worked.

Why can speed hide a bad local model setting?

Tokens per second measures one part of the run. It does not grade the answer. It also misses model load time, tool failures, malformed output, and work that needs a repair pass.

I saw the timing problem in another RTX 5090 test. The run with the higher token rate took much longer from request to response because model load dominated the clock. I wrote up that result in The Faster Local Model Run Took 83x Longer.

The same rule applies to answer quality. A model can produce broken code at a high rate. If the benchmark ends when generation stops, the bad run still looks good.

How do I add a quality gate to a local benchmark?

I start with fixed tasks that have clear answers. A code task must compile or pass its tests. A JSON task must parse and match its schema. A routing task must select the expected route. The check runs in code after the model responds.

Then I record speed and task result in the same row. I do not keep a new speed result when the fixed tasks fail. This rule makes the benchmark useful for a production choice instead of a screenshot.

The test set can stay small at first. Four code tasks were enough to reject my first workshop setting. The key is that the tasks do not change between runs. If the prompt or expected answer changes, I start a new comparison.

Did the batch size cause the better answers?

I do not know. The batch size changed, but model output can vary across runs. The first result could reflect run variance, a cold path, or another condition that the workshop did not isolate.

That uncertainty changes the next test. I should repeat each setting several times, keep the prompt and model build fixed, and record both pass rate and wall-clock time. One clean run can qualify a setting for more tests. It cannot prove a cause.

This is the same reason I test refusal behavior as its own property. My local security task report counted completed tasks, refusals, and timeouts separately. One score cannot describe every failure mode.

Which setting would I keep?

I would keep batch size 256 for the next round. It passed all fixed tasks and reached 215.7 tokens per second. Batch size 512 added 2.8 tokens per second in this run, about a 1.3% gain, with no measured quality gain.

That is a test choice, not a permanent setting. I want the smaller step until repeat runs show that 512 gives a useful wall-clock gain without more failures or memory pressure.

The operating rule is simple. First, reject wrong work. Second, compare full run time. Third, use tokens per second to explain where the time went. That order keeps the benchmark tied to the job the model must do.

Accompanying prompt

What the prompt does: This prompt turns a local LLM speed test into a repeatable speed and quality gate.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
I will give you the model name, model build, engine settings, hardware,
fixed tasks, expected answers, tokens per second, and wall-clock time.

Task:
1. Check each fixed task against its expected answer.
2. Reject any run with a failed fixed task.
3. Compare wall-clock time among the runs that passed.
4. Use tokens per second only to explain generation speed.

Output:
- A pass or fail result for each run.
- A table with task pass rate, wall-clock time, and tokens per second.
- The setting to repeat next and the reason for that choice.
- One claim that the data does not support.

Constraints:
- Do not infer cause from one run.
- Do not change the fixed tasks between runs.
- Do not select a setting that failed a fixed task.

Copy the block above.

Get the measured local AI notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Chunk Size Is a Reliability Setting

Patrick Hughes — Thu, 06 Aug 2026 16:05:12 +0000

Chunk Size Is a Reliability Setting

A crash destroys the entire unit of work in flight. All of it. So the size of
that unit decides what a crash costs you. Throughput is the thing you were
thinking about. Blast radius is the thing that actually bites.

I learned this during a run on 2026-08-04.

What happened

I was running a fully offline document pipeline over 5,000 scanned PDFs. One
container per slice of the input. Four vCPU each, no network, read only
filesystem, no GPU. The pipeline splits its input into chunks and runs each
chunk in a fresh interpreter.

I set that chunk size to 1,000, which was the maximum.

Over the next four hours, three worker processes died with native faults. Not
clean exceptions. Signal level deaths: exit -5 and exit -11. They died at
roughly 50 percent, 50 percent, and 72 percent of their chunks.

Every one of those crashes threw away the whole chunk. About seven hours of
compute, gone, on a deadline.

How much work does one crash erase?

A fault lands at a random point in the unit. So on average you lose half the
unit each time.

At 1,000 cases per unit, a fault costs about 500 cases. At 250, the same fault
costs about 125. Cut the unit by four, cut the loss by four. That is the whole
idea, and it is boring, and it would have saved me most of the night.

The overhead of smaller units is real but tiny. You pay for a few more process
restarts. I measured the cost at a few percent. The insurance is total.

Why did parallelism make the failure worse?

My first instinct was speed. More containers, finish sooner.

That was wrong twice over. More containers means more processes that can die,
and it does nothing to reduce what each death costs. Worse, I sized them
against the host's specs instead of the container runtime's actual ceiling,
blew past the memory limit, and the OOM killer took out a healthy run that had
been going for 33 minutes.

Adding parallelism to long, uncheckpointed units is the worst of both.

What order should you use to make a long job recoverable?

Shorten the unit. Set it from your observed failure rate, not your throughput preference. If faults show up every two hours, a two hour unit is the wrong unit.
Make failure loud. I had an in process retry that silently restarted a crashed chunk with one worker instead of four. It hid the signal and turned a twelve minute problem into a four hour one. A fast, noisy exit is better.
Make resumption cheap. Durable checkpoint per completed unit, validated when you read it back, skipped on restart. Then a crash costs one unit and a container start.
Only then parallelise, bounded by the ceiling you actually measured.

After I made those changes, a full restart of the container runtime cost me
nothing but the one chunk in flight. Everything else resumed from disk.

How can you tell a machine fault from a code fault?

Some crashes are your bug. Some are the machine. Tell them apart before you
spend an hour tuning code that is fine.

It is probably the machine when:

The same input succeeds on one run and dies on another.
Two or more independent workers die identically on one host.
The crash is memory corruption rather than an exception. Watch for SIGSEGV, SIGBUS, signal 139, or the one that gave it away for me: an unknown opcode error raised from inside a pure Python standard library module.

That last one is worth internalising. Python does not raise "unknown opcode" in
difflib because your code is wrong. It raises it because something wrote
garbage into the memory holding a compiled code object. No amount of
refactoring fixes that. Go test the hardware.

What did the repaired run measure?

For anyone sizing similar work: the pipeline ran at 3.2 seconds per PDF on four
vCPU, fully offline, no network calls and no GPU. Nothing in it phones home.
After moving to 250 case units, faults stopped costing me anything I could not
recover in about twelve minutes.

What should you change in your next long-running job?

Go look at your longest running job right now. Find how long one unit of work
takes between durable checkpoints. Then ask how long it has been since your
last unexplained failure.

If the unit is longer than the gap between failures, your unit is wrong, and
you are one crash away from finding out.

Accompanying prompt

What the prompt does: It turns a long-running local AI job into a checkpoint plan that limits the work lost when a worker dies.

Copy/paste this prompt:

Role:
You are reviewing a long-running local AI or offline document job.

Context:
Provide the input count, current chunk size, measured time per chunk,
checkpoint behavior, worker count, memory ceiling, and recent failure logs.

Task:
1. Estimate the work and time lost when one worker fails.
2. Recommend a chunk size tied to the observed failure interval.
3. List the durable checkpoint fields needed for safe resume.
4. Give one bounded parallelism test to run after recovery is cheap.

Output:
- Current blast radius.
- Recommended chunk and checkpoint plan.
- One next measurement.

Constraints:
- Keep measured values separate from estimates.
- Do not claim a hardware fault without repeated evidence.
- Do not increase parallelism before the resume path is tested.
<!-- blog-prompt-scope:2026-06-24 -->

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

The faster local model run took 83x longer

Patrick Hughes — Thu, 06 Aug 2026 16:05:09 +0000

On 2026-08-04 I sent the same local model request twice, 30 seconds apart. The first run reported 188.16 tokens per second. The second reported 153.12. The first run took 43.9 seconds of wall clock. The second took half a second.

Tokens per second picked the wrong winner by 23%. Wall clock says the second run was 83 times faster.

Both runs were llama3.1:8b on an RTX 5090 through Ollama, same prompt, 44 input tokens and 32 output tokens, num_ctx=1024. The 188.16 tok/s run spent 43,476.3 ms in its load phase and 170.06 ms generating. Tokens per second measures the generating part only, so it hid 99% of what the request actually cost.

What did the two runs actually measure?

Same model, same prompt, same settings. Here is what the receipt recorded.

Field	Run at 21:05:23	Run at 21:05:54
Tokens per second	188.16	153.12
Load phase	43,476.30 ms	142.43 ms
Prompt eval	183.20 ms	151.82 ms
Generation	170.06 ms	208.98 ms
Total duration	43,832.66 ms	504.78 ms
Wall clock	43,860 ms	530 ms
GPU memory before / after	855 / 5,994 MiB	5,994 / 5,994 MiB

Do the division and both speed numbers check out. 32 tokens over 170.06 ms is 188.2 per second. 32 tokens over 208.98 ms is 153.1. Neither number is wrong. They answer a question I was not asking.

Why does tokens per second rank the runs backwards?

Tokens per second divides output tokens by generation time. Ollama reports that generation time as eval_duration, and it reports load_duration and prompt_eval_duration as separate fields. The API documents all three.

So the metric is a rate for one phase of the request. It cannot see the other two.

In the first run the load phase was 43,476.3 ms of a 43,832.66 ms total. That is 99.2% of the request. The rate for the remaining 0.8% led the table.

This is not a rounding problem. For a 32 token answer, generation is the smallest part of the job. Ranking a short agent call by generation rate ranks it by the part that barely matters.

What does the receipt say about the gap?

The receipt records the load phase and the memory delta. GPU memory went from 855 MiB to 5,994 MiB across the first request and stayed at 5,994 across the second. The weights arrived during the first call.

I am not going to write "cold start" into the artifact. The runner stamps residency_state=unrecorded on both rows, and I keep a rule that the label comes from the receipt and not from my reading of it. That rule came out of an earlier fix to the same benchmark, covered in the phase timing receipt post. The measured facts are the load duration and the memory delta. Those are enough to act on, and they do not require me to invent a field.

A second row in the same report shows this is not a one-off. A qwen3.5:9b request at 77.59 tokens per second recorded 63,791 ms of wall clock for 64 output tokens. At that rate the generation accounts for about 0.8 seconds. The other 63 seconds sit in the receipt as wall clock and appear nowhere in the headline speed.

What should a local benchmark report instead?

Report wall clock beside tokens per second, and let the selection step read wall clock.

Three rules I now apply.

First, record all three phases. Load, prompt eval, and generation. Ollama returns them, so keeping them costs nothing.

Second, match the metric to the workload. For a batch job that generates thousands of tokens against a model already in memory, generation rate is the right number. For a single-shot agent decision returning 32 tokens, it is close to noise. My workload is the second kind.

Third, keep both runs. Do not drop the slow one as a warm-up. The first run is the honest measure of what the first request of the day costs, which is the same reason I benchmark input and output sizes separately instead of reporting one blended rate.

The change to my benchmark is small. It now reports wall clock per request next to the rate, and the row that leads on rate no longer wins by default. For an agent that makes one call and waits, 43.9 seconds is the number a person feels. 188.16 tokens per second is the number that looks good in a table.

Accompanying prompt

What the prompt does: turns a local model benchmark into a wall-clock comparison so a fast generation rate cannot hide a slow request.

Copy/paste this prompt:

Role: engineer benchmarking a local language model on owned hardware.
Context: my rows report tokens per second, but some hide a long model load phase.
Task:
1. Per row, list load, prompt eval, generation, and total wall clock.
2. Compute what percent of wall clock each phase used.
3. Rank rows by wall clock, then by tokens per second. Show where they disagree.
4. Name the metric that matches my workload and say why.
5. Name one rerun that separates load cost from generation cost.
Output: a phase table, both rankings, and the rerun command.
Constraints: do not invent missing fields. Keep slow rows. Do not label a run cold or warm unless the receipt records it.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

VRAM Fit Is Not Runtime Support

Patrick Hughes — Tue, 04 Aug 2026 16:05:14 +0000

A model can fit in your GPU memory and still fail before it produces one token.

That happened in my latest local model test. A 2.7 GB Nanbeige4.2-3B GGUF downloaded and registered on an RTX 5090 with 32 GB of VRAM. Ollama then stopped at model load with an unknown architecture error. The memory check passed. The runtime check failed.

This is why I now treat VRAM fit and runtime support as separate gates in the local model sizing workflow.

What did the 5090 test actually prove?

The test proved one narrow thing: Nanbeige4.2-3B was not runnable on this Ollama build. The Q4_K_M file downloaded. The model appeared in the local model list. The request failed with unknown model architecture: 'nanbeige' when Ollama tried to start its llama-server process.

I repeated the check with Q5_K_M. It failed at the same stage. That matters because it rules out a single quant file as the first suspect. The report records a runtime support problem, not a throughput result.

The GPU had 32 GB of total memory. The Q4_K_M model occupied 2.7 GB on disk. Those numbers make the model look like an easy fit. They do not tell us whether the serving stack can load its architecture.

Why is VRAM fit only the first gate?

I use four gates for a local model:

Artifact gate. Can I download and identify the model file?
Memory gate. Does the model fit the available VRAM, RAM, or offload plan?
Runtime gate. Does the named Ollama or llama.cpp version recognize the model architecture and quant format?
Workload gate. Does it complete the real task with acceptable quality, latency, and power?

The first two gates are easy to check from a model page and a GPU spec. The third requires a real load attempt. The fourth requires a real task. A sizing tool that stops after memory fit can turn an impossible download decision into a confident recommendation.

My VRAM fit guide still has a useful place. It answers the memory question. My honest local model evaluation notes answer the workload question. The missing middle is a named runtime check.

What should a runtime-support check record?

The check does not need a large benchmark suite. It needs a small receipt with enough detail to reproduce the result:

model name and exact file or tag
quantization and file size
runtime name and version
GPU name and available memory
load result and exact error text
one short generation only after load succeeds

The final item is important. Do not report tokens per second for a model that never reached generation. The correct result is load_failed, with the reason attached.

That is how the Nanbeige test is recorded. It produced no throughput number. Publishing a number from another machine would answer a different question and weaken the desk's data.

What about models that are too large to fit?

The same report tested Laguna S2.1 as a separate case. Its BF16 files total roughly 250 GB, and its smaller MXFP4 MoE build is split across three shards. That is not a near miss on a 32 GB card. Measuring it requires an NVMe expert-offload path, and that path did not build on this Windows host because the C toolchain was absent and WSL was unavailable.

That is another useful failure boundary. The report has no Laguna throughput number because the engine did not build and no inference ran. The next action is infrastructure work, not a guessed performance estimate.

How does this change model selection?

The sizing desk should show runtime support beside VRAM fit. A model recommendation should name the runtime version that loaded it, or mark support as unverified.

My current known-good rows show why this distinction helps. The 2026-08-02 5090 report includes llama3.1:8b rows for an agent code task and a long-context summary. They are workload rows from a model that loaded. They are useful only because the earlier gates cleared.

The practical order is simple: fit the model, load it, run the task, then compare speed. VRAM fit gets you to the starting line. Runtime support gets you a starting line at all.

Accompanying prompt

What the prompt does: It turns a local model check into a receipt that separates memory fit, runtime support, and workload evidence.

Copy/paste this prompt:

Role: local model operator checking a candidate for an owned GPU.
Context: I have a model tag or GGUF file, GPU memory details, a runtime version, and a load or generation result.
Task:
1. Check artifact identity and file size.
2. Check memory fit against the available VRAM and any stated offload plan.
3. Check whether the named runtime loaded the model architecture.
4. Record the exact load error when the model fails.
5. Report workload speed only after one real task completes.
Output: a four-gate receipt with pass, fail, or unverified for each gate.
Constraints: do not invent throughput. Keep another machine's result separate. Name the runtime version and preserve the exact error text.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Why Local LLM Benchmarks Need Power Data

Patrick Hughes — Tue, 04 Aug 2026 16:05:10 +0000

Why Local LLM Benchmarks Need Power Data

Tokens per second is useful, but it is not enough to choose a local model setting. Power draw tells you what the faster setting costs while it runs. On my RTX 5090, two Gemma 26B settings both passed the same fixed tasks, yet the faster one used much more power for a small speed gain.

What did the RTX 5090 run measure?

The Aug. 1, 2026 report records a workshop mix of three 200-token generation tasks and four code tasks. The model was gemma4:26b in Ollama with a 2,048-token context. The fixed-task quality result was 100% at both num_batch=256 and num_batch=512.

The report also records the machine state: an NVIDIA RTX 5090 with 32,607 MiB total memory, 73 C at the snapshot, and 401.68 W drawn against a 575 W limit. Those fields make the result easier to compare with a later run on the same machine or a different card.

What changed when I raised num_batch?

At num_batch=256, Gemma produced 215.7 tok/s and averaged 187.6 W. At num_batch=512, it produced 218.5 tok/s and averaged 234.3 W. The second setting added 46.7 W for 2.8 tok/s.

That is a poor trade if both settings meet the same quality bar and the workload is not waiting on prompt processing. The higher setting may still win for a batch job where wall-clock time matters more than power. The point is that the benchmark should expose the choice instead of hiding it behind one speed number.

Why does speed alone give the wrong answer?

A speed-only table encourages a single ranking. The fastest row looks like the winner. That ranking changes when you add power, quality, context length, and memory use.

The same report shows this in a smaller model. The llama3.1:8b agent-code task ran at 227.79 tok/s with a 7,826 MiB VRAM peak. Gemma 26B ran the same named workload at 207.41 tok/s with a 20,233 MiB VRAM peak. These are not interchangeable rows. They describe different memory and quality choices.

I keep the measurement fields together because local model selection is a fit problem first. My earlier notes on VRAM fit and runtime support cover the load gate. The 5090 phase timing receipt covers why one end-to-end latency number can hide the slow phase.

What fields should a local LLM benchmark include?

Record these fields for every meaningful run:

Model and quantization.
Runtime and version.
Context length, prompt size, and output limit.
Task name and fixed quality result.
Tokens per second.
Average and peak power when available.
VRAM peak and the GPU memory ceiling.
A timestamp and the exact machine.

Do not compare rows that change several of these fields at once. If the model, runtime, prompt, and task all change, the speed difference cannot tell you which change mattered.

What should I change in my own benchmark loop?

Add power and quality as first-class columns. Keep the prompt and task set fixed. Run the same setting more than once when the result will drive a production choice. Store the raw output with the summary so a surprising row can be checked later.

The useful question is not “Which setting is fastest?” It is “Which setting meets the quality bar at an acceptable power and memory cost?” That answer belongs in the report beside tok/s.

Accompanying prompt

What the prompt does: It turns a local model run into a comparable benchmark row with speed, quality, power, and memory fields.

Copy/paste this prompt:

Role:
You are a local LLM benchmark reviewer.

Context:
Model: [model and quantization]
Runtime: [Ollama, llama.cpp, or other]
GPU: [GPU name and memory]
Run log: [paste the log or report path]

Task:
1. Extract the task, context length, prompt size, and output limit.
2. Extract tokens/sec, average and peak power, VRAM peak, and quality result.
3. Compare this row with the supplied baseline without inventing missing fields.

Output:
- One markdown benchmark row.
- A short pass/fail quality note.
- One sentence on the speed, power, and memory tradeoff.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

My 5090 benchmark was missing the field I needed most

Patrick Hughes — Sun, 02 Aug 2026 16:05:14 +0000

My 5090 benchmark was missing the field I needed most

I had a fresh local benchmark with a respectable tokens-per-second number and still could not answer the question I actually care about: where did the request spend its time?

The repair was small. I changed the benchmark receipt to record capture time, model load duration, prompt evaluation duration, output evaluation duration, total duration, and GPU power. The next Qwen3.5 9B row made the hidden phase visible.

What did the instrumented run show?

The RTX 5090 ran one bounded local agent-decision workload through Ollama with num_ctx=4096 and num_predict=64. The model produced 64 output tokens at 84.94 tokens per second.

That is not the useful summary by itself. The receipt recorded:

Load duration: 6,105.25 ms.
Prompt evaluation: 151.18 ms.
Output evaluation: 753.47 ms.
Total model duration: 7,013.40 ms.
Average GPU power: 300.81 W.

The load phase was eight times longer than output evaluation. A route that reports only 84.94 tok/s hides the part a user waits through first.

The public dated 5090 artifact carries the row and method. The Ollama generate API reference documents the response timing fields I now preserve instead of collapsing them into one stopwatch number.

Why was the old receipt too thin?

The earlier runner saved model, workload, tokens per second, watts, and a wall-clock note. That was enough to compare rough throughput. It was not enough to explain a slow request or compare a resident model with a load event.

It also lacked a capture timestamp. A report generated after the fact can accidentally make an older row look current. My benchmark provenance guide covers that failure in the report layer. My phase-by-phase timing guide applies the same split inside one request. This change moves that discipline down to the row-producing runner.

I am keeping one detail explicit: the receipt still says residency_state=unrecorded. The load duration is measured. Residency classification is not. I will not turn a timing difference into a cold or warm label that the runner did not capture.

What changes for an agent workload?

The next routing decision should use phase data.

If the model is already resident, output evaluation may dominate. If the model has to load, the first request carries a different latency budget. A queue worker that pins one model and reuses it should be measured separately from a worker that swaps models between jobs.

That distinction matters more than a leaderboard. The right question is not which model has the highest tok/s number. It is which route clears the workload's total latency, quality, memory, and power limits.

I will keep the next test narrow: the same model, context, and prompt, with one workload variable changed. The receipt now has enough fields to show what moved.

The benchmark is useful when it tells me what the next measurement should isolate. A smaller receipt with the right phases compounds faster than a larger table of blended averages.

Accompanying prompt

What the prompt does: It turns one local model response into a phase-timing receipt without inventing residency or runtime metadata.

Copy/paste this prompt:

Role:
You are reviewing one local LLM request for production routing.

Context:
Paste the response timing fields, model, workload, capture time, GPU
snapshot, and the task result.

Task:
1. Separate load, prompt evaluation, output evaluation, and total time.
2. Calculate output tokens per second from eval_count and eval_duration.
3. Flag any missing capture or residency field.
4. Recommend the next single measurement.

Constraints:
- Keep measured fields separate from estimates.
- Never label a request cold or warm unless the receipt records it.
- Do not rank models from output speed alone.
- Preserve the source timestamp and workload name.

Copy the block above.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Search Old Results Before Publishing an LLM Test

Patrick Hughes — Sat, 01 Aug 2026 16:05:13 +0000

Search Old Results Before Publishing an LLM Test

My latest RTX 5090 post passed every mechanical check and still failed QA. I had already published the finding.

The short answer: Before I write from a local LLM benchmark, I now search the published archive for the model, runtime setting, and two signature numbers from the result. That check catches a repeated conclusion even when the title and framing changed. On August 1, 2026, it stopped me from publishing a second post about the same Ollama num_batch sweep. Canonical URL: https://bmdpat.com/blog/ollama-num-batch-tuning-quality-gate-2026

How did the duplicate get through the first checks?

The new draft looked clean. It used a fresh July 30 report, cited the source rows, included two live internal links, passed the prose gate, passed the publish-body validator, and carried a generator-made visual that cleared the mechanical image check. Every one of those gates reads the draft on its own. None of them reads the archive.

Its conclusion was also defensible. In the saved gemma4:26b workshop rows, num_batch=256 produced 215.7 output tokens per second at 187.6 watts average. The 512 row reached 218.5 output tokens per second at 234.3 watts average. Both passed the fixed task checks.

The problem was not truth. The problem was novelty.

My July 19 post, Ollama num_batch: 256 Was My RTX 5090 Sweet Spot, had already compared those rows and made the same route choice. The second draft changed the title and added a tuning checklist, but the load-bearing finding was identical.

Why is a new report date not a new result?

The July 30 lab report carries generated_at: 2026-07-30 09:50. That stamp records when the rollup ran. It says nothing about when any single row was measured.

The saved rows prove the gap. The workshop benchmark CSV has these columns: model, quant, engine, workload, input tokens, output tokens, tokens per second, watts average, and notes. There is no capture-time column at all. The same July 30 table also mixes runtime versions, listing rows from Ollama 0.22.1 and rows from Ollama 0.31.1. One report date cannot stand in for the run time of any row.

So a report generator can pull an old measurement back into view without creating new evidence. If I treat the newest report as a list of new findings, I publish the same result twice.

My benchmark provenance guide argues for binding driver, runtime, model, workload, settings, and capture time to every row. This workshop CSV predates that rule, which is exactly how a stale row reads as current. The content path needs one more field on top: the public claim already made from that row.

What is a signature-number search?

A title search is too weak. The old post may use different words.

I search for three parts of the result together:

The model or model family, such as gemma4:26b.
The changed setting, such as num_batch.
Two numbers that identify the comparison, such as 215.7 and 218.5.

The numbers act like a small fingerprint. They survive headline changes and topic labels. A match does not always mean the new post is a duplicate, but it forces a direct comparison before drafting continues.

For this run, the search found the July 19 post. Independent QA then compared the thesis, rows, power tradeoff, and recommendation. It returned FAIL, so the post did not ship.

When can the same benchmark support another post?

The bar is a new decision, not a new introduction.

The same rows can support a follow-up if new evidence changes the route, a later runtime version changes the result, or the post answers a different operational question that the first one did not answer. A translation of the same conclusion into another checklist is not enough.

For example, a repeated sweep across several seeds could show whether the 2.8 tokens-per-second gap persists. A new VRAM trace could test whether either batch value crosses a memory limit. A queue-level run could measure completed jobs per hour instead of output rate. Those would add evidence.

Without that addition, I should link the old post and move on.

What gate do I run before drafting now?

First, identify the exact measurement rows. Do not start from the report title.

Second, extract the model, setting, workload, and two signature numbers. Search the published archive for all four.

Third, compare the proposed one-sentence takeaway with each match. If the new post reaches the same conclusion from the same rows, stop.

Fourth, record what changed. A new model, rerun, verifier, route decision, or failure can justify a follow-up. A new date in a rollup cannot.

This gate runs before I draft the hook. It also keeps the public lab notebook honest. Readers get one clear result instead of several versions that make the evidence look larger than it is.

Accompanying prompt

What the prompt does: It checks whether a planned local LLM benchmark post repeats a finding that is already public.

Copy/paste this prompt:

Role:
You are checking a local LLM post for duplicate evidence.

Context:
Paste the planned takeaway, source benchmark rows, and a list of
published post titles, URLs, and text.

Task:
1. Extract the model, setting, workload, and two signature numbers.
2. Search the published text for matching evidence.
3. Compare the old and planned one-sentence takeaways.
4. State what new measurement or decision the planned post adds.

Output:
- PASS if the post adds a distinct finding.
- FAIL if it repeats the same conclusion from the same rows.
- The closest prior post and the exact overlap.

Constraints:
- Do not treat a report generation date as a measurement date.
- Do not approve a rewrite that only changes the title or checklist.
- Keep measured facts separate from proposed follow-up tests.
- Do not invent source rows, numbers, or run dates.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Build Local LLM Eval Data From Real Failures

Patrick Hughes — Fri, 31 Jul 2026 16:05:15 +0000

Build Local LLM Eval Data From Real Failures

A clean demo tells me a local coding model can pass one task. A saved failure tells me what to test next.

The short answer: I turn each local model attempt into one replayable row that keeps the prompt, generated code, tests, verifier result, model, route, and timestamp together. Passing rows set a baseline. Failed rows become regression cases. Canonical URL: https://bmdpat.com/blog/build-local-llm-eval-data-from-failures-2026

Why are failed local LLM runs useful?

I used to look at a bad answer, change the prompt, and run again. That fixes the moment. It does not create a durable test.

My local coding rig now saves both good and bad attempts. A sample row contains the prompt, generated code, test code, verifier result, pass fraction, source, timestamp, model, generation method, and route. That record lets me replay the same task after I change a model, quant, system prompt, or runtime setting.

The failure is useful because it is specific. One saved add(a, b) attempt returned subtraction. Its two tests both failed. A saved is_prime(n) attempt used the wrong remainder check. One of its two tests passed, while the other failed. Those rows show different failure shapes even though both answers were wrong.

I do not need a model judge to explain either result. The tests provide the boundary.

What should one eval row contain?

I keep the smallest record that can reproduce the decision:

prompt
model_output
test
verifier_result
pass_fraction
model
generation_method
route
source
captured_at

The prompt and output show what the model saw and wrote. The test defines the behavior I expected. The verifier result records what happened. The route tells me whether the request went direct to Ollama or through another layer.

That last field matters. A model name alone does not identify a run. Prompt templates, context settings, and routing code can change the answer before the weights do.

This is the same reason I use a verifier loop before trusting a local coding model. The verifier turns a plausible answer into a pass or fail result. The eval row keeps that result tied to the exact attempt.

How do failed rows become a regression set?

First, I keep the original failure. I do not replace it with the repaired answer.

Second, I inspect the test. A weak test can make a bad row look useful. The is_prime example caught one bad branch but still passed another case. That partial result is a reason to add boundary cases, not a reason to award half credit and move on.

Third, I replay the task against the next candidate route. I compare the result with the saved baseline. If the new route passes, I still keep the old failure because it proves the test can detect the bug.

This gives me a paired dataset: an input, a rejected output, a passing output, and the same executable check. I can use the pair for prompt repair, route selection, or later training work. I do not have to guess which answer was better.

What can this eval set prove?

It can prove that a named route passed a named set of checks at a recorded time. It can show whether a change fixed known failures or brought old ones back.

It cannot prove that the model is good at coding in general. Three toy functions are starter data, not a production benchmark. Repeating the same seeds across iterations also does not create new task coverage.

I keep that limit visible because small evals are easy to oversell. My note on honest local LLM eval numbers explains why the denominator and failed runs belong beside the score.

The next useful step is to add failures from real work. I want tasks with repository context, file edits, tool calls, and tests that reflect the jobs my local agents actually do. A synthetic starter confirms the storage and replay path. Production failures make the set valuable.

How do I keep the loop cheap?

I save the row as soon as verification finishes. I do not wait for a later cleanup pass. The run already has the prompt, output, route, and test result in memory, so that is the cheapest moment to write the receipt.

I also separate collection from promotion. A row may enter the dataset after verification, but a model only earns a production route after it passes the full gate. This lets me collect useful failures without lowering the bar for deployment.

The result is a small loop I can trust: run a real task, verify the output, save the receipt, add the failure to the replay set, then test the next route against it.

Accompanying prompt

What the prompt does: It turns local coding-agent run logs into replayable eval cases without inventing missing tests or measurements.

Copy/paste this prompt:

Role:
You are building a regression set for a local coding model.

Context:
Paste one task prompt, model output, test output, model name,
runtime route, and capture time.

Task:
1. Preserve the original prompt and output.
2. Extract the executable checks and their results.
3. Classify the failure by observed behavior.
4. List missing boundary cases that need new tests.

Output:
- One replayable eval row with named fields.
- A pass or fail verdict tied to the saved tests.
- A short list of added test cases, if needed.

Constraints:
- Do not repair or overwrite the failed output.
- Do not invent a test result or runtime setting.
- Keep measured facts separate from suggestions.
- Mark missing fields as unknown.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

DEV Community: Patrick Hughes

When a 4B Local LLM Beats 26B on One Task

When a 4B Local LLM Beats 26B on One Task

Can a 4B local model beat a 26B model?

Why does task quality come before tokens per second?

What should a local model comparison hold fixed?

When would I still choose the 26B model?

What model-selection rule did I keep?

Accompanying prompt

Does Ollama Include That New llama.cpp Feature?

Does Ollama Include That New llama.cpp Feature?

Why is the release note not enough?

How do I map the active local inference path?

What did my RTX 5090 smoke test prove?

Why did I not compare the new result with the old baseline?

What evidence should change a recommendation?

Accompanying prompt

My local models refused zero of 50 security tasks

Why measure refusal rate at all?

What did the run measure?

How do you score a refusal without a judge model?

Does zero refusals mean the models did the work well?

What does this prove, and what does it not?

Accompanying prompt

Prime Agent hit 95.5% on ARC-AGI-3. I did not install it.

Prime Agent hit 95.5% on ARC-AGI-3. I did not install it.

What Prime Agent is

The rule that stopped me

What I did instead

The part I have not resolved

The checklist

Related reading

Accompanying prompt

Related reading

My Local LLM Got Faster After It Passed the Tests

My Local LLM Got Faster After It Passed the Tests

What did the three local runs show?

Why can speed hide a bad local model setting?

How do I add a quality gate to a local benchmark?

Did the batch size cause the better answers?

Which setting would I keep?

Accompanying prompt

Chunk Size Is a Reliability Setting

Chunk Size Is a Reliability Setting

What happened

How much work does one crash erase?

Why did parallelism make the failure worse?

What order should you use to make a long job recoverable?

How can you tell a machine fault from a code fault?

What did the repaired run measure?

What should you change in your next long-running job?

Accompanying prompt

Related reading

The faster local model run took 83x longer

What did the two runs actually measure?

Why does tokens per second rank the runs backwards?

What does the receipt say about the gap?

What should a local benchmark report instead?

Accompanying prompt

Related reading

VRAM Fit Is Not Runtime Support

What did the 5090 test actually prove?

Why is VRAM fit only the first gate?

What should a runtime-support check record?

What about models that are too large to fit?

How does this change model selection?

Accompanying prompt

Why Local LLM Benchmarks Need Power Data

Why Local LLM Benchmarks Need Power Data

What did the RTX 5090 run measure?

What changed when I raised num_batch?

Why does speed alone give the wrong answer?

What fields should a local LLM benchmark include?

What should I change in my own benchmark loop?

Accompanying prompt

My 5090 benchmark was missing the field I needed most

My 5090 benchmark was missing the field I needed most

What did the instrumented run show?

Why was the old receipt too thin?

What changes for an agent workload?