DEV Community: Patrick Hughes

Ollama num_batch: 256 Was My RTX 5090 Sweet Spot

Patrick Hughes — Mon, 20 Jul 2026 14:45:09 +0000

Ollama num_batch: 256 Was My RTX 5090 Sweet Spot

I changed one Ollama setting and got 37.5 percent more generation speed from the same local model. Then I doubled it again, burned more power, and gained almost nothing.

The short answer: In my four-run gemma4:26b workshop, moving num_batch from 128 to 256 raised measured throughput from 156.9 to 215.7 tokens per second. Moving from 256 to 512 reached 218.5 tokens per second, only 1.3 percent faster, while average GPU power rose from 187.6 to 234.3 watts. For this RTX 5090 workload, 256 was the practical setting. Canonical URL: https://bmdpat.com/blog/ollama-num-batch-rtx-5090-2026

What is num_batch in Ollama?

Ollama exposes num_batch inside the options object on the generate API. The official API reference lists it beside settings such as num_ctx, num_gpu, and num_thread.

That makes it a runtime setting to measure, not a quality score. I did not assume a larger value would produce a better completed task. The useful value can change with the model, context, GPU, prompt shape, and runtime version.

I wanted a measured setting for my own machine, not a copied default. The workshop changed one configuration at a time and recorded speed, average power when available, and a fixed coding check.

What did the RTX 5090 workshop measure?

The model was gemma4:26b through Ollama. Each run used three short generation prompts capped at 200 output tokens plus four fixed coding tasks with deterministic checks. The first three runs held context at 2,048 and tested batch values of 128, 256, and 512.

At batch 128, throughput measured 156.9 tokens per second. The coding checks scored zero percent on that run, so I do not treat its speed delta as a clean batch-only benchmark. It still exposed a bad configuration for the whole job.

At batch 256, throughput reached 215.7 tokens per second and all four checks passed. Average GPU power measured 187.6 watts. At batch 512, throughput reached 218.5 tokens per second with the same passing checks, while average power measured 234.3 watts.

These are workshop measurements from July 16 through July 18, 2026. They are not vendor claims and they do not predict another GPU.

Why did I keep 256 instead of 512?

The extra 256 batch units bought 2.8 tokens per second. That is a 1.3 percent speed gain. Average power increased by 46.7 watts, about 24.9 percent.

I will take a large speed gain when it removes real wait time. I will not pay a quarter more power for a change I cannot feel in the workflow. Batch 256 captured 98.7 percent of the measured batch-512 speed at the same context, while keeping the passing task result.

This is why I tune for completed work instead of the largest allowed number. The setting with the highest throughput was technically 512. The setting I would deploy was 256.

What happened when context doubled?

The July 19 run kept batch at 512 and moved context from 2,048 to 4,096. Throughput fell from 218.5 to 183.6 tokens per second. The four coding checks still passed, and average power measured 212.4 watts.

That run was discarded because the earlier 2,048-context setting was faster at the same quality score. It also reinforced a rule from my Ollama context reload test: context belongs in the measured configuration. A batch result without its context value is incomplete.

The prompt totals were small, so the larger context did not help the task. I would only pay for it when the input needs it.

How should you tune num_batch on your own GPU?

Hold the model, quant, context, prompts, output cap, and runtime version steady. Test a small ladder such as 128, 256, and 512. Record throughput, loaded memory, average power, and a task-level pass or fail.

Do not keep a setting because it wins one speed column. Reject any run that breaks the output contract. Then choose the smallest setting that captures nearly all of the useful speed with enough memory headroom for the real workload.

The same rule applies to model choice. My Ternary Bonsai comparison kept the larger Gemma default because instruction compliance beat file size. Runtime tuning should use the same gate: fit, run, verify, then keep.

Accompanying prompt

What the prompt does: It turns local Ollama benchmark rows into a measured num_batch recommendation without inventing missing data.

Copy/paste this prompt:

Role:
You are a local LLM runtime tuning reviewer.

Context:
Paste benchmark rows for one model and GPU. Include model, quant,
runtime version, num_ctx, num_batch, tokens per second, power,
loaded memory, and task verifier result when available.

Task:
Compare the settings and reject any run that fails the task verifier.

Output:
- A compact comparison table.
- The smallest setting that captures nearly all useful speed.
- The exact measurement missing before deployment.

Constraints:
- Compare only runs with the same workload.
- Keep context attached to every batch result.
- Separate measured values from estimates.
- Do not invent missing power or memory readings.

Copy the block above.

I publish measured local AI build notes in The 5090 Reports. Join the email list for the next result.

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=ollama-num-batch-rtx-5090-2026

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

A 7 GB 27B Model Lost to My 17 GB Default

Patrick Hughes — Sun, 19 Jul 2026 14:45:08 +0000

A 7 GB 27B Model Lost to My 17 GB Default

A new 27B model landed on my RTX 5090 in a 6.66 GiB GGUF file. It generated 123 tokens per second. It answered all three factual checks correctly. I still did not make it my default.

The short answer: Ternary Bonsai 27B proved that a large model can fit in a small memory budget, but fit did not decide the job. My local test favored gemma4:26b because it generated faster and followed the requested output format. File size gets a model onto the machine. Task completion keeps it there. Canonical URL: https://bmdpat.com/blog/ternary-27b-vs-gemma4-26b-local-ai-2026

How did a 27B model fit in 6.66 GiB?

I tested the official Ternary-Bonsai-27B-Q2_0.gguf on July 16, 2026. llama-bench reported a 6.66 GiB file and 26.90 billion parameters. The model ran with 99 GPU layers on an RTX 5090 at a 4,096-token context.

The compact file comes from ternary weights. Instead of storing ordinary high-precision weights, the model uses values of negative one, zero, or positive one, plus shared scale factors. The official model card describes a roughly 7.2 GB deployed footprint for the language model. My downloaded file measured 7,165,121,600 bytes.

That is the first win. A 27B-class model that fits under 8 GB opens options for laptops, smaller GPUs, and multi-model servers. It also makes model size a poor proxy for task quality.

Was the smaller file fast on an RTX 5090?

Yes, with a caveat. Bonsai generated 123.37 tokens per second in three llama-bench repetitions. Prompt processing measured 3,885.47 tokens per second on the same run. Those are my local measurements, not vendor estimates.

My existing gemma4:26b default generated the same factual prompt at 179.95 tokens per second through Ollama. That comparison crosses two runtimes, so I treat it as a job-level spot check, not a clean model benchmark. It was still enough to answer the operator question: which model should keep the drafting slot on this machine?

The answer stayed Gemma. Bonsai was compact and fast enough to serve. It was not faster for the task I needed.

Why did instruction following decide the winner?

I asked both models three factual questions and required one exact FINAL line. Both models found Canberra, Ursula K. Le Guin, and Python 3.7. Bonsai put the correct answers in its reasoning, then used the 384-token cap before it emitted the required line. Gemma returned the exact line.

That difference matters in an agent loop. A correct answer in hidden reasoning is not the same as a parseable result. If the next step expects one line, the model that emits that line finishes the job. The model that explains past the cap creates a retry, a repair step, or a manual check.

This is why I separate model intelligence from model usefulness. Benchmarks can tell me whether a model has capability. My verifier tells me whether it completed my task.

What should a local model selection test measure?

Start with fit, but do not stop there. Record the actual loaded footprint at the context you plan to use. Context and cache settings can change the memory total after the weights fit.

Then measure the real output path. I care about generation speed, cold-load behavior, exact format compliance, and whether the result passes a deterministic check. My Q4_K_M versus Q5_K_M comparison uses the same idea: the quant label is an input, not the verdict.

Test the workload at its natural length too. My 8B long-form failure test showed that a quick answer can hide a length-control failure. A model that wins a short prompt may lose a full draft.

For each candidate, I now want one row with five fields: loaded memory, warm generation speed, cold-load time, verifier result, and failure reason. The default slot goes to the model with the best completed-task result, not the smallest download.

When would I still choose Ternary Bonsai 27B?

I would test it again on a memory-constrained machine or a server that needs several resident models. A 6.66 GiB file leaves much more room than my 17 GB Gemma default. That can matter more than raw speed.

I would also run it as a persistent server. My captured Bonsai load took 68.99 seconds, and other launches from the Windows E: drive took longer. Paying that cost for every request would erase the value of fast generation.

The model earned a place in the sizing data. It did not earn the default drafting job. That is a useful result. A good model test should be allowed to end with "fits, runs, does not replace."

Accompanying prompt

What the prompt does: It turns a local model comparison into a task-based keep, test, or reject decision.

Copy/paste this prompt:

Role:
You are a local LLM model-selection reviewer.

Context:
Paste the GPU, runtime, model files, context size, benchmark output,
task prompt, expected output format, and verifier result.

Task:
1. Compare loaded memory and cold-load time.
2. Compare warm generation speed on the same task.
3. Check exact output compliance and verifier results.

Output:
- A table with fit, speed, compliance, and failure reason.
- One decision: keep, test again, or reject for this job.

Constraints:
- Separate local measurements from vendor claims.
- Do not compare different runtimes as a clean model benchmark.
- Do not invent missing measurements.

Copy the block above.

I publish measured local model decisions in The 5090 Reports. Join the email list for the next result.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

My Agents Have to Prove What They Did

Patrick Hughes — Sat, 18 Jul 2026 20:48:25 +0000

I stopped trusting "done."

One of my agents reported three completed actions. Two were false. The model produced a convincing final answer. The repository did not match it.

The tool trace looked busy. The summary sounded sure. Nothing in that chat proved the side effects landed.

So I made a rule for the operation: every material completion needs a check that can fail.

That layer is now open source as showwork. It is on PyPI.

pip install showwork

What does showwork verify?

Observability answers, "What did the agent do?"

showwork answers, "Is the outcome it claimed actually true?"

The first question needs traces. The second needs a falsifiable assertion and a deterministic checker. No LLM judging an LLM.

showwork start --session fix-timeout --agent claude-code

showwork claim --session fix-timeout \
 --claim "bumped billing-sync API timeout to 30s" \
 --type file_contains --path config/api.yaml --pattern "timeout: 30"

showwork finish --session fix-timeout --status ok

If a claim is RED, the clean close is refused. Exit code 2. Fix reality, retract the claim, or finish as blocked.

The command checker has no shell. It only permits a Python script under the project root. It rejects shell metacharacters, path escape, and recursive verification. Other checks cover files, content, path moves, frontmatter, and file counts. Vacuous patterns (regex that match empty strings, always-true counts) are rejected, not blessed.

Why is the ledger append-only?

Agents will be wrong. Operators will also be wrong. Deleting a bad assertion makes the record look cleaner while destroying the most useful evidence.

showwork appends a retraction that names the earlier claim and explains the correction. The original record stays. Failures become training material for the operation.

Does it gate the session?

Yes, through the explicit finish command. Observation and gating stay separate.

showwork finish --session fix-timeout --status ok

The Claude Code Stop-hook adapter records a verdict when a session stops, but always exits 0. It observes. It does not block. The gate is the finish call your prompt contract requires.

What proof came before the package?

I derived a sanitized snapshot from the private ledger that runs the operation:

2,158 claims recorded
2,152 backed by deterministic checks
842 sessions represented
152 append-only retractions
One malformed ledger line surfaced
Captured verdict: RED, 54 of 60 verified

I kept the RED result. A proof system that only publishes green is not proof.

The derivation publishes aggregates plus a source fingerprint. It does not copy private claim text, paths, strategy, or financial values.

Why publish a specification?

The package can be copied. The record format can outlive it.

spec-v0.1 defines the JSONL schema, checker semantics, anti-vacuous rules, retractions, verdict algebra, session events, and exit-gate contract. Every normative requirement names a behavioral test.

Another team can implement the same evidence format in Go or TypeScript. That is more useful than forcing every agent stack through one Python process.

What is the artifact?

Installable CLI and Python API: pip install showwork
Portable ledger specification
Claude Code Stop-hook adapter
Genesis receipt where the repository verified its own tests
Sanitized case study and derivation script
Marketing demo cut: agents say done, finish is REFUSED, then GREEN after reality matches

Code: github.com/bmdhodl/showwork
PyPI: pypi.org/project/showwork

Where runtime limits fit

Verification is one half of trusting an unattended agent. The other half is bounding what the agent can do before it does it.

showwork proves what an agent did after the fact. It does not stop an agent from looping, over-calling a paid API, or draining a token budget on the way to a wrong answer. Those are runtime questions, and they need a runtime limit.

That is the job of AgentGuard: hard caps on tokens, cost, and call rate around an agent run, enforced while the agent is still running. AgentGuard bounds what an agent can spend. showwork proves what an agent actually produced. Run both and an unattended agent has a ceiling on the way in and a receipt on the way out.

Your Agent's Audit Trail Cannot Be Retrofitted

Patrick Hughes — Sat, 18 Jul 2026 20:48:17 +0000

My agent said it completed three actions. Two had not happened.

The chat transcript looked fine. The tool trace showed activity. The final answer sounded certain. None of that proved the files and state matched the answer.

That failure led me to build a separate verification layer. Every material completion claim now needs a deterministic check. The session cannot close cleanly when a RED claim fails.

Logs are not outcome proof

An agent trace can show that a model called a file-writing tool. It cannot prove the write landed in the right repository, contained the required value, passed the relevant test, or survived another process changing the file.

Those are outcome questions.

I wanted each claim to be falsifiable. "Updated the configuration" became a file-content check. "Moved the queue task" became a source-absent and destination-present check. "Tests pass" became a locked command with an expected exit code.

No model grades the model. The checker reads reality.

The record has three parts

The first part is the claim:

{
 "session": "deploy-fix",
 "claim": "the API timeout is configured",
 "severity": "RED",
 "artifact": "config/api.yaml",
 "check": {
 "type": "file_contains",
 "path": "config/api.yaml",
 "pattern": "timeout: 30"
 }
}

The second part is deterministic verification. The current checker set covers file existence, file content, path moves, frontmatter, glob counts, and locked Python commands. Vacuous checks fail. A regex that matches every file is not proof. A count of at least zero is not proof.

The third part is an append-only receipt. Corrections become retraction records. The original assertion remains in history.

The exit gate changes agent behavior

A normal stop hook runs after the agent has already stopped. It can record a verdict, but it cannot send the agent back to fix its work.

The explicit finish command can.

showwork finish --session deploy-fix --status ok

If a RED claim fails, the command exits with code 2 and refuses the clean close. The agent must fix the outcome, retract the claim truthfully, or report the session as blocked.

The Stop-hook adapter still records the final verdict. It always exits zero because it observes rather than gates. Keeping those roles separate prevents a reporting hook from becoming an availability risk.

The production ledger is not perfectly green

The ledger runs next to the agents themselves: my fleet operates on owned hardware, and the receipts live on the same machine as the work. The sanitized source ledger contains 2,158 claims from 842 sessions. Deterministic checks back 2,152 claims. It also contains 152 retraction records and one malformed line that the parser surfaced instead of dropping.

The captured audit was RED at 54 of 60 verified.

That is useful evidence. A verification system that converts its own gaps into a green launch number is not doing verification.

The public case study contains aggregates only. It excludes claim text, private paths, strategy, financial values, and trading records. A SHA-256 fingerprint ties the aggregate to the exact source ledgers used for the derivation.

The AI Act makes evidence an engineering problem

The European Commission says the AI Act becomes fully applicable on 2 August 2026, with named exceptions and longer transitions for some regulated-product systems. That does not make this package a compliance solution. I am not offering legal advice.

It does make one engineering question harder to ignore: can you reproduce the evidence behind an agent's claimed outcome?

Action logs are useful. Model traces are useful. Neither replaces a check that compares the assertion with the resulting state.

The official application timeline is on the European Commission's AI Act page.

The portable part is the specification

The Python package is one implementation. The longer-lived artifact is spec-v0.1, which defines the JSONL record, six check types, retractions, verdict rules, session events, and exit-gate behavior.

Someone can implement the same format in Go or TypeScript without reading the Python source.

The repository is bmdhodl/showwork. It includes the Stop-hook adapter, conformance tests, genesis receipt, and sanitized production case study.

Bound the agent before you audit it

A receipt proves what happened. It does not cap what an agent can spend on the way to a wrong outcome. Evidence and limits are two sides of one problem: trusting work no human watched in real time.

AgentGuard is the limit side. It puts hard ceilings on tokens, cost, and call rate around an agent run and enforces them while the run is live, so a looping agent hits a wall instead of your bill. showwork is the evidence side: the append-only receipt that proves the outcome after the fact. Bound what the agent can do, then prove what it did.

Ollama Raised $65M. What Builders Get

Patrick Hughes — Fri, 17 Jul 2026 14:45:09 +0000

Ollama Raised $65M. What Builders Get

Ollama just turned local AI from a developer preference into a funded product category. The money matters less than what it will buy: better model access, more predictable runtimes, and a clearer path between a workstation and cloud capacity.

The short answer: TechCrunch reported a $65 million Series B, while Ollama says it has raised $88 million in total and serves 8.9 million developers. That does not prove every workload should move local. It does mean builders can expect the local model on-ramp to keep improving. Canonical URL: https://bmdpat.com/blog/ollama-raises-65m-what-it-means-for-local-ai-builders-2026

What did Ollama's funding actually prove?

The round proved that access to open models is now a business, not just a GitHub project. Ollama's official announcement centers on ownership, privacy, local execution, and an optional cloud path for models that do not fit on one machine. That is a practical product map.

I read the signal this way: local inference no longer has to win every benchmark. It only has to own a useful class of work. Drafting, extraction, classification, private retrieval, and repeatable agent steps are good candidates. Hard reasoning and occasional long-context work can still go to a frontier API.

That split is already how I use my RTX 5090. Ollama handles repeatable local work. A frontier model handles the final QA gate when the output will publish or change an important system. The local model buys capacity. The verifier buys trust.

Does local inference remove token costs?

No. It changes the bill.

A local request does not create a provider token charge. The real costs are the GPU, electricity, model load time, cooling, maintenance, and the attention needed when the runtime fails. If the hardware already exists and the workload runs often, the marginal request cost can be small. If the workload is rare or needs the best available reasoning, an API may still be cheaper.

This is why I avoid the phrase "free inference." Owned compute is prepaid compute. The useful comparison is workload by workload: requests per day, latency target, privacy boundary, model size, and how much operator time the local path consumes.

My consumer GPU production guide covers the operating side. The deciding question is not "local or cloud?" It is "which steps are stable enough to own?"

What should builders expect Ollama to improve?

Ollama says its funding will support hybrid inference, faster access to new open models, and cloud access without giving up the local product. Those are useful priorities because model serving has three separate problems.

First, the model must fit. Quantization, context size, and KV cache decide whether a run stays inside VRAM. Second, the runtime must stay loaded and answer consistently. Third, the surrounding agent must fail safely when the model stalls, repeats, or calls the wrong tool.

The funding can improve the first two. Builders still own the third.

For example, changing context size can force a full model reload. I documented that failure in my Ollama context reload field note. A friendlier runtime does not remove the need to pin settings, measure reloads, and keep a bounded fallback.

How should a local agent fail safely?

Treat the model server and the agent as separate systems. Ollama produces tokens. Your agent decides what those tokens are allowed to do.

I put timeouts around generation, cap retries, log tool calls, and require a verifier before writes. If a task needs filesystem or network access, the runtime also needs explicit limits. A loop that costs no provider tokens can still burn an afternoon, corrupt files, or hold a GPU indefinitely.

That is where AgentGuard's runtime limits fit. The point is not to make Ollama safe by association. The point is to bound the code around any model, local or hosted, before it receives tools.

The best local-first design is boring. Route repeatable work to the owned GPU. Measure it. Escalate the hard cases. Verify every consequential result.

What changes for builders now?

The local path has stronger financial backing and a large developer base. That should improve the on-ramp. It does not change the engineering decision.

Start with one bounded workload. Record model, quant, context, latency, failure mode, and verifier result. Compare that row with the same task on an API. Keep the route that wins on cost, privacy, availability, and operator time.

Ollama's round is useful because it gives that experiment a longer runway. Your own measurements still decide where local inference belongs.

Accompanying prompt

What the prompt does: It turns one workload into a measured local-versus-cloud routing decision with explicit pass and fail rules.

Copy/paste this prompt:

Role:
You are a local AI workload evaluator.

Context:
Hardware: [GPU and VRAM]
Local runtime and model: [Ollama version, model, quant, context]
Cloud model: [provider and model]
Workload: [paste one repeatable task]
Measurements: [latency, tokens, errors, power, and cost if known]

Task:
1. Check whether the local model fits and completes the workload.
2. Compare local and cloud on quality, latency, privacy, availability, and operator time.
3. Choose local, cloud, or a verified hybrid route.

Output:
- A comparison table using only supplied measurements.
- A routing recommendation with one pass rule and one fallback trigger.
- The next measurement needed to reduce uncertainty.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.

Copy the block above.

I publish measured local runs, failure notes, and routing decisions in The 5090 Reports. Join the email list to get the next field note.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Why production AI is moving to open weights

Patrick Hughes — Thu, 16 Jul 2026 14:45:12 +0000

Why production AI is moving to open weights

Short answer: I do not need a frontier model to write every first draft. I need one where mistakes are expensive. On my RTX 5090, a local 26B model can do the high-volume writing pass. Deterministic checks and an independent frontier reviewer decide whether the draft is safe to publish.

That is not a theory. I ran the split on this post.

Gemma 4 26B wrote the draft through Ollama. The model was fully resident on the 5090. The accepted local generation took 6.2 seconds. Before the file entered review, code checked the metadata, post length, banned phrases, internal links, prompt block, CTA, and every percentage or dollar claim.

Then a separate frontier model reviewed the result. It rejected the first draft.

That rejection is the important part.

What did the local model get wrong?

The local draft had three real problems.

First, it split the digits in the 41% figure. Second, it omitted the closing line required by the reusable prompt block. Third, the source card did not contain the primary links needed to support its Vercel numbers, even though those numbers were in the writer's verified source pack.

The deterministic gate caught unsupported percentages in an earlier attempt. The frontier reviewer caught the corrupted number and the provenance gap in the accepted attempt. The repair script refused to make a subjective source decision and left the post blocked.

That is the behavior I want. Local generation can be cheap and fast because every later stage is allowed to say no.

Why move the writing pass to a 5090?

The economics now support this split outside my machine too.

Hugging Face reports that Chinese models accounted for 41% of model downloads on its platform. Vercel reports open-weight models handled 29% of AI Gateway tokens in June 2026 on under 4% of spend. Frontier labs still retained 95% of spend.

Those numbers do not say frontier models are obsolete. They say production volume and premium judgment are separating.

Writing a first draft is high-volume and reversible. Factual review is lower-volume and higher consequence. Running both steps on the most expensive model wastes the frontier budget on work my hardware can already do.

The same logic applies to classification, extraction, summaries, support drafts, and log analysis. Start with tasks where a bad local result can be rejected before a customer sees it.

For the machine details, read my 5090 local inference guide. For the budget side, see AI agent cost control.

Where should frontier models stay in the loop?

I am keeping frontier models on four jobs:

Claim verification against primary sources.
Voice review for public writing.
Decisions where a false positive can publish, charge money, or change production state.
Recovery when the local model fails the deterministic contract repeatedly.

I am moving routine drafting to the 5090. The local writer cannot approve its own work. It cannot publish. Its only output is a draft in a queue.

This boundary matters more than the model name. A local model with direct publish access is a bad design. A frontier model with direct publish access can still be a bad design. The controls need to live in code, not in a prompt asking the model to be careful.

What makes the pipeline permanent?

The old pipeline started at QA. If the morning digest created a content task but no draft, QA saw an empty folder and reported success. The publisher also saw an empty folder and reported success. Every process was green while the actual outcome was red.

The new chain starts earlier:

The digest routes a source-backed content task.
At 7:45 CT, Gemma 4 26B converts one eligible task into a draft.
Code rejects malformed metadata, weak structure, unsafe language, and unsupported numeric claims.
At 8:30 CT, an independent frontier provider reviews the surviving draft.
The repair and publisher stages keep their existing gates.
At 9:35 CT, the healer reruns the full chain and checks the live site.

Provider limits are scoped too. If the Claude subscription is exhausted, Claude is removed from the QA chain. That no longer disables Codex, Grok, or Gemini. Known Codex quota failures are skipped instead of consuming the whole review window.

The target is not zero frontier use. It is spending frontier attention where it changes the result.

Accompanying prompt

What the prompt does: This prompt helps you design a two-tier workflow where a local model generates content and a frontier model audits it for accuracy.

Copy/paste this prompt:

Role:
You are an AI Workflow Architect specializing in hybrid local-frontier pipelines.

Context:
I am running high-volume text generation on a local RTX 5090 setup using Ollama. I use frontier models (like Claude or GPT) only for final quality audits to keep costs low.

Task:
1. Analyze the provided raw text generated by my local model.
2. Identify any factual hallucinations or logical inconsistencies.
3. Check if the tone meets my brand guidelines.
4. Provide a "Pass" or "Fail" grade for each section.

Output:
- Error Log: List specific errors found.
- Audit Score: A score from 1 to 10.
- Corrected Version: The final text ready for publication.

Constraints:
- Do not rewrite the entire text if it is already correct.
- Focus only on factual accuracy and risk mitigation.
- Keep the feedback concise.

Copy the block above.

Weekly measured local runs: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Devlog 2026-07-13: local drive git corruption stalls the q

Patrick Hughes — Thu, 16 Jul 2026 14:45:08 +0000

Devlog 2026-07-13: local drive git corruption stalls the q

What happened

The nightly queue sweep aborted for the fourth straight night. The root cause is a critical failure in the vault health check. I discovered a private path worktree corruption affecting agent47, agent47-dashboard, and bmdpat.

This corruption prevents sandbox provisioning because the metadata is missing or corrupt. The 04:45 retry task failed to fire as expected today. I am also dealing with an open Request from 2026-07-07 that has left the bmdpat main branch stuck on a stash-pop conflict.

What shipped

Closed the blog infographic quality backlog in config/visuals/todo.md (P2-2 through P2-4 cancelled).
Added several blog infographics for different layouts via bmdpat commits 06145c02, 1a1be6a4f, 4af87a05, a5ef3eee, 040bc94a, and 8117da61.
Replaced the rebuild-vector-index infographic with an aligned card layout via bmdpat commit ef861315.
Corrected embedding rebuild infographic provenance via bmdpat commit 1f772ad6.
Fixed blog dates to America/Chicago and removed UTC next-day labels via bmdpat commits bd55734f, a6b22c02, 357d067e, and aba807e4.
Logged Kraken manager flat reconciliation at 18:16 CT via autotrader commit 6c4248d.
Logged Kraken manager flat reconciliation at 17:16 CT via autotrader commit a7a319.
Logged Kraken manager flat reconciliation at 16:16 CT via autotrader commit 6c139ec.
Logged Kraken manager flat reconciliation at 15:16 CT via autotrader commit a19f212.
Logged Kraken position manager flat reconciliation at 14:15 CT via autotrader commit 9d01475.
Set kraken-entry-scout to stand down on macro blackout for multiple timestamps via autotrader commits 6645d32, 0efcdd4, e62ae5b, 7eb9ac5, 6c139ec, 9adfce6, 64ddb70, a19f212, and b88315d.
Set kraken-position-manager to FLAT with 0 closes via autotrader commits e0fb1ca and 165e97b.

Cannon state

CC debt is not pulled today.
Autotrader equity is a private amount with a +0.45% P&L.
SPY alpha is -6.72.
BTC alpha is -4.37.

What the machine did overnight

The nightly queue sweep aborted because sandbox repo provisioning failed for three repositories.
Vault-health and security-analyst both fired today after yesterday's scheduler drop.
SecurityAnalyst returned a YELLOW verdict with 0 P0, 15 P1, and 80 P2 issues.
Gitleaks scanned 5/5 repos and found 1 unique leak.
Brain worker drained the queue from 6 to 2 tasks.
Prompt-evolver logged 5 patterns but zero prompt-shaped candidates.

What I thought about

The infrastructure is breaking under its own weight. The local drive drive corruption is a blocker that prevents the entire automated pipeline from progressing. I cannot move forward with the queue until these git trees are repaired.

Tomorrow

Repair the corrupt local drive git work trees for agent47, agent47-dashboard, and bmdpat.
Run queue-sweep once the sandbox can provision.

Get the artifact-backed local AI lab notes by email: https://bmdpat.com/5090-reports

Accompanying prompt

What the prompt does: Explains how this public daily log is produced from the private vault devlog with deterministic redaction.

Take today's private BMD HODL devlog. Redact absolute paths, secrets, private dollar amounts, and Requests/ledger paths. Keep first-person builder voice. Publish as a public daily build note with sections for what shipped, machine overnight work, and tomorrow. Link 5090 Reports.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

My 8B Model Failed a 400-Word Task

Patrick Hughes — Wed, 15 Jul 2026 14:45:23 +0000

I gave Llama 3.1 8B a simple job: turn a set of verified daily artifacts into a 400-to-700-word devlog. It returned 208 to 299 words across three live runs. A retry collapsed the required structure before the post reached the floor.

The short version: Small local models are good when the task is bounded and the output has a hard verifier. Three failed 8B devlog trials gave me the evidence to keep that model on short scoring work. I assigned long-form synthesis to Gemma 4 26B and gate both task classes with code.

Canonical URL: https://bmdpat.com/blog/local-llm-long-form-model-selection-2026

What failed in the three 8B runs?

The prompt asked for six required sections and a 400-word minimum. Llama 3.1 8B produced 208 to 299 words across three runs on my RTX 5090. It could identify the source facts. It could not keep the full long-form shape intact.

That distinction matters. The result did not make the model useless. The July 10 handoff records the same 8B model as a fit for research triage and scoring, where the answer is short, structured, and easy to check.

The measured numbers in this post come from my July 10 local-inference handoff report. That report records the three 8B attempts, the passing 26B result, and the runtime settings that changed the outcome.

Why did another retry not solve the problem?

Retries help when failure is noisy. They help less when the model repeats the same limit.

The July 10 handoff records a structure collapse on retry. It does not prove why the model collapsed, so I do not guess. The useful fact is simpler: three attempts stayed below the contract or lost the required shape.

This is where a verifier earns its keep. I did not ask the model whether its draft was complete. Code checked the required sections, word range, forbidden terms, and output status. The task assignment changed because the artifact failed, not because the prose felt weak.

The same principle applies to generation after retrieval. I preview the files before local inference, then verify the generated artifact after inference. Both gates isolate the failed stage.

What changed when I routed the task to Gemma 4 26B?

The first larger-model attempts exposed a different failure. Gemma 4 26B is a thinking model in Ollama 0.31. With its thinking mode left on, it consumed the output budget internally and returned an empty response.

Setting think to false fixed that boundary. Increasing num_predict to 2600 kept path-heavy sections from being cut off. The configured model produced a 485-word devlog with all six required sections and passed the deterministic gate.

The lesson was not "use the biggest model." The lesson was "route to the smallest model that can pass the artifact contract, then configure that model for the job." Model size, context, thinking mode, and output budget are one runtime shape.

That runtime shape must stay stable. My RTX 5090 context-size sweep showed that changing num_ctx can force a full reload. A good route includes those settings instead of treating the model name as the whole key.

Why not send every task to the 26B model?

The 8B model is still my choice for bounded work. In a separate July 9 sweep, it peaked at 7,826 MiB of VRAM while the 26B model peaked at 20,233 MiB. A classifier that returns a score and one reason does not need the larger memory footprint of my long-form model.

I separate tasks by output shape. Short extraction, scoring, tagging, and command selection go to the 8B route. Multi-source narrative work with six sections goes to the 26B route. The verifier decides whether either result may advance.

The task classes are assigned before inference. Triage and scoring use the 8B route. The six-section devlog uses the 26B route. Each output carries its own verifier result, so the system does not silently swap models and call both outcomes equivalent.

What should the verifier check before a route advances?

Start with rules that code can judge. Check required sections, minimum and maximum length, forbidden phrases, empty output, truncated output, and whether cited paths came from the approved source manifest. Record the model, quant, context size, thinking mode, output budget, and verifier result with the artifact.

Do not turn the verifier into another model unless the rule truly needs judgment. Word counts and section names belong in code. A human or separate model can review tone later. The first gate should be fast, exact, and repeatable.

The resulting policy is boring: test each task shape, keep the smallest model that passes, pin that assignment, and run the same gate on every output. That is enough to stop a weak result from shipping and a larger model from handling every task by default.

Accompanying prompt

What the prompt does: This prompt turns local-model run records into a verifier-driven routing rule for short and long-form tasks.

Copy/paste this prompt:

Role:
You are a local LLM routing engineer.

Context:
Paste the task, model settings, output artifact, and verifier result for each local run.

Task:
1. Group runs by task shape and required output.
2. Identify the smallest model that passed every deterministic rule.
3. Name the exact failure that justifies each larger-model route.

Output:
- A routing table with task shape, model, runtime settings, and pass rules.
- One default route and any justified escalation routes.
- A list of checks that should run in code before human review.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.
- Do not recommend a larger model without a recorded verifier failure.

Copy the block above.

I publish the measured local-model runs, failures, and fixes in The 5090 Reports. Join the list to get the next artifact.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Q4km vs Q5km: Q4_K_M vs Q5_K_M

Patrick Hughes — Wed, 15 Jul 2026 14:45:19 +0000

Q4km vs Q5km: Q4_K_M vs Q5_K_M

Short answer: q4km and q5km are informal spellings of GGUF quants Q4_K_M and Q5_K_M. For most local LLM loads on a single consumer GPU, start with Q4_K_M. Move to Q5_K_M only when you still have VRAM after weights plus KV cache.

What do people mean by q4km vs q5km?

Search and chat often drop the underscores: q4km, q5km, q8. In llama.cpp and Ollama file names the real tags are Q4_K_M, Q5_K_M, and Q8_0 (or close cousins like Q5_K_S). Same decision: size vs quality vs speed under a fixed VRAM budget.

When should you pick Q4_K_M?

Pick Q4_K_M when:

You want the model to load fully without CPU fallback.
You still need context headroom for the KV cache.
You care more about tokens/sec than the last few quality points.

Q4_K_M is the default local quant for a reason: it is usually the best balance of file size and usable quality on 8-24 GB cards.

When should you pick Q5_K_M?

Pick Q5_K_M when:

Q4_K_M already fits with clear headroom.
You see quality issues on coding or long-form tasks at Q4.
You measured that Q5 still keeps full GPU offload at your context length.

If Q5 forces layers back to CPU, you often lose more quality (and speed) than you gain from the higher quant.

How do you decide in five minutes?

Note GPU VRAM and target context length.
Open quant compare and VRAM calculator.
Check weight size for Q4_K_M and Q5_K_M against free VRAM after OS and cache.
Prefer the smaller quant that stays fully on GPU.

For the full matrix (Q4/Q5/Q6/Q8, K_S vs K_M, and edge cases), read the long guide: GGUF Q4_K_M vs Q5_K_M vs Q8. For offload flags once the file fits, see llama.cpp --n-gpu-layers.

Accompanying prompt

What the prompt does: It turns your GPU VRAM, context length, and two GGUF file sizes into a Q4_K_M vs Q5_K_M pick with a hard pass/fail.

Copy/paste this prompt:

Role:
You are a local LLM quant picker.

Context:
GPU VRAM (GB): [ ]
Target context length: [ ]
Model: [ ]
Q4_K_M file size (GB): [ ]
Q5_K_M file size (GB): [ ]
Need full GPU offload: yes/no

Task:
1. Estimate free VRAM after OS and KV cache.
2. Say whether Q4_K_M and Q5_K_M fit fully on GPU.
3. Pick one quant and one fallback if load fails.
4. List the single next measurement to run.

Output:
- Pick: Q4_K_M or Q5_K_M
- Fit: full GPU / mixed / no
- Fallback plan
- Next command or check

Constraints:
- Prefer full GPU offload over a higher quant with CPU fallback.
- Do not invent sizes I did not provide.
- Keep the answer under 12 lines.

Copy the block above.

Weekly measured local runs: https://bmdpat.com/5090-reports

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

I Rebuilt 5,383 Embeddings After a Dimension Change

Patrick Hughes — Tue, 14 Jul 2026 14:45:22 +0000

I changed one embedding model and had to rebuild the whole vector index.

The old path used OpenAI text-embedding-3-small at 1,536 dimensions. The new path uses nomic-embed-text through Ollama at 768 dimensions on my RTX 5090. The model swap was easy. The data migration was the real job.

The short version: An embedding dimension change is a schema change, not a model setting. I changed the vector column, cleared the old rows, rebuilt the HNSW index, re-embedded 5,383 chunks from 1,605 notes, and checked retrieval with three known queries after the rebuild.

Canonical URL: https://bmdpat.com/blog/rebuild-vector-index-embedding-dimensions-2026

Why can I not mix 1,536 and 768 dimension vectors?

My Supabase table stored each embedding in a vector(1536) column. The local model returned 768 values. Those are different data shapes.

Changing only the application code would leave the database contract behind. Keeping the old rows would also leave corpus vectors and query vectors produced by different models. Even if I copied them into a looser storage format, the similarity scores would not represent one shared vector space.

I treated the change like a database migration. The new column became vector(768). The search function accepted a 768-value query. The HNSW index was rebuilt for the new column. Old embeddings did not survive the cutover.

What did the rebuild include?

The July 10, 2026 handoff report records the full pass: 5,383 chunks from 1,605 notes. One background run on the RTX 5090 generated every replacement vector with nomic-embed-text.

I kept the corpus and query paths on the same model. config/brain/embed_vault.py embeds notes. The vault search server embeds each query. Both now call the same Ollama model and expect 768 dimensions.

That symmetry matters more than the model name. A corpus built by one embedding model and queried by another can return numbers, but those numbers do not prove useful retrieval. I want one encoder, one dimension, and one migration boundary.

The process was simple:

Apply the 768-dimension schema and search function.
Clear the old corpus rows.
Rebuild every chunk with the local embedding model.
Run known queries and inspect the top results.

How did I test retrieval after the migration?

I used three queries with obvious expected answers. The AgentGuard query returned its entity note first at 0.694 cosine similarity. The Raspberry Pi researcher query returned its deployment README first at 0.676. The workflow audit query returned Offer.md first at 0.637.

Those scores are not universal quality targets. They are results from this corpus and this model. The useful test was whether each known query returned the expected source at the top after the rebuild.

This is the same reason I preview retrieval before a local LLM runs. Retrieval should produce an inspectable artifact before generation begins. A model cannot repair a missing source it never received.

I also keep durable facts in files before they enter semantic search. That design is covered in why I write agent memory to files. The vector index makes those files easier to find. It is not the source of truth.

What should fail closed during an embedding migration?

The embedding helper now checks that the local response has exactly 768 values. A wrong-length vector stops the run. The database schema provides a second boundary. The query path uses the same expected dimension.

I would also stop the cutover if the corpus count was far below the source count, if the embed service returned empty vectors, or if the known-query checks missed their expected files. A completed migration is not a successful migration unless retrieval still works.

The safest rollout is a full rebuild with a small parity suite. Keep the old index available until the new one passes. Then switch the query path once. Do not let two embedding models write into the same active corpus.

When is a full rebuild worth it?

I rebuilt because the change removed a paid API dependency from a daily vault search path. It also let me re-embed the corpus on hardware I own whenever the source files change.

That benefit justified one clean migration. It would not justify swapping models every week. Each embedding model change creates a new corpus version, a new query contract, and another parity run.

My rule is now explicit: pin the embedding model and dimension together. If either changes, version the index and rebuild the corpus. The application should never discover that contract drift during a live query.

Accompanying prompt

What the prompt does: This prompt turns an embedding-model change into a fail-closed vector-index migration plan with retrieval checks.

Copy/paste this prompt:

Role:
You are a vector search migration reviewer.

Context:
Paste the current embedding model, new model, dimensions, table schema, corpus counts, and three known queries with expected results.

Task:
1. Identify every schema, index, corpus, and query-path change.
2. Define the order for rebuilding vectors without mixing models.
3. Write parity checks that must pass before the query path switches.

Output:
- A numbered migration plan with rollback points.
- A pass/fail table for vector length, corpus count, and known-query results.
- The exact evidence to record after the rebuild.

Constraints:
- Keep it short.
- Use exact numbers and file paths when available.
- Do not invent missing measurements.
- Treat a model or dimension change as a new corpus version.

Copy the block above.

I publish measured local-model migrations, failures, and fixes in The 5090 Reports. Join the list to get the next artifact.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

I Raised the QA Bar on Blog Infographics

Patrick Hughes — Tue, 14 Jul 2026 14:45:17 +0000

I ship measured local-model posts with charts. Generative image tools garble numbers. So the fleet uses one matplotlib renderer under config/visuals/. Exact text. Fixed brand colors.

That still was not enough. A chart could pass "it rendered" and still look sloppy: numbers in corners, tiny labels, digits that never appear in the post. I raised the bar.

What failed before

The first defect class was layout. Step numbers sat in the top-left of a card while the label sat in the center. Peer cards did not share baselines. Long labels shrank below a readable floor.

The second class was honesty. A graphic could invent a comparison that no metrics file contained. Vision QA only checked whether the image looked clean, not whether the numbers were true.

The third class was publish mechanics. Committing a PNG from a feature branch worktree was easy to get wrong. Agents need a path that always targets main for the one asset file.

The gates

Every blog infographic now hits the same chain before an embed line is printed.

Char budget. Budgets scale with step count. A short chain (n≤3) gets wider cards and longer labels. A four-step row stays tight (18/22). Five or more steps use a two-row grid so cards do not crush.
Layout then claims. Numbers, labels, and subs share peer baselines. Step indexes sit in orange pills with clearance so they never paint over the words. Digits in the graphic require --claims <post or metrics> or publish exits 8.
Vision log. The vision gate still fail-closes. Full stdout and stderr land under Reports/Visuals/ so a timeout is not a mystery.

Footer text stays at the same readable floor as body labels (7pt minimum). Titles sit on a dark header scrim so ambient textures cannot wash them out.

What I measure

The unit suite under config/visuals/ is at 29 tests as of 2026-07-13. Selftest renders every file in examples/ and specs/ (13 specs after this post). Geometry QA checks containment, min size, peer baselines, and peer line counts. Char lint runs before draw.

I also re-rendered the live vector-migration post asset earlier the same day after the first stack fix. New posts get the full gate set automatically when they use publish_blog_asset.py.

What I will not do

I will not put labels inside a diffusion model. Charts need crisp, correct English. Ambient textures stay under the diagram. Text stays on the matplotlib layer.

I will not ship a PNG that fails any gate. Fail closed means ship text-only or fix the spec. That is intentional.

When a graphic is worth it

Use a chart when the post has a process, a comparison, or a set of numbers. Write the short labels first. Point --claims at the draft. If the story needs long phrases, use fewer steps or a stats/flow template. Do not fight the 4-up width with paragraphs inside cards.

The contract lives in Prompts/Includes/blog-publish.md and config/visuals/README.md. Agents read it before inventing a one-off plot.

Accompanying prompt

What the prompt does: This prompt turns a draft post into a fail-closed infographic publish checklist with char budgets and claims.

Copy/paste this prompt:

Role:
You are a blog infographic publisher for exact-text charts.

Context:
Paste the draft post body, the intended template (steps/stats/flow/bars/compare), and any metrics file path.

Task:
1. Write a JSON spec that fits char budgets (4-up steps: label <=18, sub <=22).
2. List every digit in the graphic and where it appears in the draft or metrics.
3. Run publish with --claims and report geometry/vision outcomes.

Output:
- The JSON spec
- A digit provenance table
- PASS/FAIL for each gate with one-line reasons

Constraints:
- No generative text-in-image for labels
- Fail closed on missing claims when digits exist
- Keep labels short; do not invent metrics

Copy the block above.

I publish measured local-model work and the systems around it in The 5090 Reports. Join the list for the next artifact.

For related layout and local-model posts, see I rebuilt 5,383 embeddings after a dimension change and the num_ctx reload tax.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

Pin Your Context Window or Pay the Reload Tax

Patrick Hughes — Sun, 12 Jul 2026 14:45:19 +0000

I changed one setting on a local agent and it went silent for 140 seconds.

No output. No error. Just a fan spinning up while a 17 GB model reloaded from disk. The actual answer, once the model was back, took under two seconds to generate. I had paid a two-minute tax for a two-second task.

The short version: Ollama reloads the entire model when you change num_ctx, the context window size. On my RTX 5090, gemma4:26b took about 140 seconds to reload every time num_ctx changed. Generation itself ran near 200 tokens per second. Pin num_ctx once and the tax goes away.

Why did my local model freeze for 140 seconds?

I was benchmarking two models on one card: an RTX 5090 running Ollama 0.31.1. The sweep sent three workloads to each model, and one of them used a longer context, so my script bumped num_ctx from 4096 up to 8192 and back down again.

Every one of those changes forced a full model reload. gemma4:26b is a 17 GB file at Q4_K_M. Each reload took between 138 and 142 seconds. Same model, same GPU, already warm. The only thing that changed was the context number, and that was enough to send the whole thing back to disk.

What actually triggers the reload?

The context window is not a runtime knob you can turn for free. Ollama allocates the KV cache when it loads the model, and the cache size depends on num_ctx. Change num_ctx and the previous allocation no longer fits, so the runtime tears the model down and loads it again from scratch.

This is easy to trigger by accident. An agent that sets context per request, a script that summarizes long documents then answers short questions, a chat loop that grows its window as the conversation runs. Each of those can flip num_ctx without anyone deciding to, and each flip is a fresh reload.

How much does the reload cost?

The numbers from that run are blunt. gemma4:26b generated at about 198 to 207 tokens per second once it was loaded. A 256-token answer is done in under two seconds. The reload before it took about 140 seconds.

So the model spent roughly 98 percent of that request loading and 2 percent working. There is a second, smaller tax hiding in the same place. The first prompt evaluation right after a load is a cold-start artifact: gemma's first reading measured 15 tokens per second, and the very next request hit 6,179. If you reload constantly, you also eat that cold start every single time.

How do I stop paying the tax?

Pick one context size and pin it. Decide the largest window your workload actually needs, set num_ctx to that value once, and never change it mid-session. A fixed 8192 that loads one time beats a "smart" window that reloads on every other request.

In practice that means setting num_ctx at model load and leaving it. If different jobs genuinely need different windows, keep a separate warm model per window instead of retuning one. The reload is the expensive part. Paying it once at startup is fine. Paying it every few requests is the whole problem.

What does this mean for a local coding agent?

This is the trap for anyone running an agent loop on a local model. Agent stacks love to adjust context. They pad the window for a big file, shrink it for a quick tool call, grow it as history accumulates. On a small 8B model the reload is quick enough to ignore. On a 17 GB model it is a 140-second stall in the middle of your loop, and the agent has no idea it is waiting on I/O instead of thinking.

June 12 I had this exact failure and did not understand it. A gemma4:26b run timed out at a 5-second limit. A 5-second timeout can never survive a 140-second load. The fix was not a faster GPU. It was pinning the context so the model stopped reloading.

What do you cap after you pin the context?

Pinning num_ctx removes one hidden cost. It does not remove the other one: an agent looping on that model with no ceiling. A local agent that retries in a tight loop burns GPU time and wall-clock the same way a stray reload does, except it can run until you notice.

So I put a hard limit around the loop. Before a local agent runs, I give it a budget, a token cap, and a rate limit, so one bad loop stops itself. That is what AgentGuard does: runtime budget, token, and rate limits for AI agents, in one pip install. Pin the context so the model stops reloading. Cap the loop so the agent stops running. Two different meters, same discipline: find the hidden cost, then put a hard limit on it.

Accompanying prompt

What the prompt does: It reviews how your local agent or script sets the context window and flags any place that would force Ollama to reload the model mid-run.

Copy/paste this prompt:

Role: You are a local LLM performance reviewer.

Context: I run models locally through Ollama on a single GPU. Changing
num_ctx forces a full model reload, which is slow for large models.

Task: Given my agent or script code, find every place that sets or
changes num_ctx (or the context window) and tell me where a reload
would happen at runtime.

Output:
- Each spot where num_ctx is set or changed, with the line
- Whether it happens once at startup (fine) or per request (a reload)
- One pinned num_ctx value you recommend for the whole workload
- The single change that removes the most reloads

Constraints: Assume a reload costs 100+ seconds for a large model.
Prefer one fixed context size over a dynamic one. Be specific about
lines. No general advice about prompt engineering.

Copy the block above.

Two minutes of pinning beats 140 seconds of dead air on every context change. Cap the reload by pinning num_ctx, then cap the agent with AgentGuard. Predict the hidden cost, then put a hard limit on it. That is the whole habit.

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.

DEV Community: Patrick Hughes

Ollama num_batch: 256 Was My RTX 5090 Sweet Spot

Ollama num_batch: 256 Was My RTX 5090 Sweet Spot

What is num_batch in Ollama?

What did the RTX 5090 workshop measure?

Why did I keep 256 instead of 512?

What happened when context doubled?

How should you tune num_batch on your own GPU?

Accompanying prompt

A 7 GB 27B Model Lost to My 17 GB Default

A 7 GB 27B Model Lost to My 17 GB Default

How did a 27B model fit in 6.66 GiB?

Was the smaller file fast on an RTX 5090?

Why did instruction following decide the winner?

What should a local model selection test measure?

When would I still choose Ternary Bonsai 27B?

Accompanying prompt

My Agents Have to Prove What They Did

What does showwork verify?

Why is the ledger append-only?

Does it gate the session?

What proof came before the package?

Why publish a specification?

What is the artifact?

Where runtime limits fit

Related reading

Accompanying prompt

Your Agent's Audit Trail Cannot Be Retrofitted

Logs are not outcome proof

The record has three parts

The exit gate changes agent behavior

The production ledger is not perfectly green

The AI Act makes evidence an engineering problem

The portable part is the specification

Bound the agent before you audit it

Related reading

Accompanying prompt

Ollama Raised $65M. What Builders Get

Ollama Raised $65M. What Builders Get

What did Ollama's funding actually prove?

Does local inference remove token costs?

What should builders expect Ollama to improve?

How should a local agent fail safely?

What changes for builders now?

Accompanying prompt

Why production AI is moving to open weights

Why production AI is moving to open weights

What did the local model get wrong?

Why move the writing pass to a 5090?

Where should frontier models stay in the loop?

What makes the pipeline permanent?

Accompanying prompt

Devlog 2026-07-13: local drive git corruption stalls the q

Devlog 2026-07-13: local drive git corruption stalls the q

What happened

What shipped

Cannon state

What the machine did overnight

What I thought about

Tomorrow

Related

Accompanying prompt

My 8B Model Failed a 400-Word Task

What failed in the three 8B runs?

Why did another retry not solve the problem?

What changed when I routed the task to Gemma 4 26B?

Why not send every task to the 26B model?

What should the verifier check before a route advances?

Accompanying prompt

Q4km vs Q5km: Q4_K_M vs Q5_K_M

Q4km vs Q5km: Q4_K_M vs Q5_K_M

What do people mean by q4km vs q5km?

When should you pick Q4_K_M?

When should you pick Q5_K_M?

How do you decide in five minutes?

Accompanying prompt

I Rebuilt 5,383 Embeddings After a Dimension Change

Why can I not mix 1,536 and 768 dimension vectors?

What did the rebuild include?

How did I test retrieval after the migration?