Unload Local LLMs After Every Test

A local model test is not over when the model returns text. It ends after I unload the model and read the idle GPU state.

Short answer: I explicitly unload a local model after each measured test. I record the loaded VRAM, the idle VRAM after unload, and the runtime setting that kept it loaded. This stops one warm model from changing the next test.

Canonical URL: https://bmdpat.com/blog/unload-local-llms-after-tests-2026

Why unload a local model after a test?

Ollama can keep a model in memory after a request. That is useful when the next request uses the same model and settings. It is a problem when I want a clean comparison.

A warm model can make the next run look better than it is. It can hide load time. It can also hold VRAM that another model needs. I cannot call a new result a cold-load result if the previous model still owns memory.

On 2026-08-15, I ran hf.co/unsloth/Qwen3.8-27B-GGUF:Q3_K_S through Ollama on the owned RTX 3070 in FLUARMN. I explicitly stopped the model after the run and recorded the next idle reading in the local receipt.

That reading belongs in the receipt. It shows whether the target runtime released the model state. It does not show that every process on the host stopped. The idle baseline can include the display, drivers, and other work.

What does an unload check prove?

An unload check proves a narrow fact: the target runtime released the model state it held for that request. It does not prove model quality, stable speed, or a useful context size.

I keep those questions separate. The local LLM benchmark decision record keeps task quality, speed, VRAM, power, and failure state in the same decision. The unload result is one field in that record.

The check also helps find a bad comparison. If VRAM stays high after I ask the runtime to stop the model, I do not start another model test and hope for a clear result. I record the state, stop the test sequence, and find the process that still holds memory.

How do I make the check repeatable?

I use the same short sequence for each run.

1. Record the idle GPU memory before the model loads.
2. Run one named model, runtime, context, and prompt.
3. Record the memory during the request and the model's keep-alive setting.
4. Stop the model through the target runtime.
5. Read GPU memory again and save the post-unload result.

The sequence makes a warm run explicit. If I want warm-request latency, I leave the model in memory and name the result warm. If I want cold-load time, I unload first and record the idle state. I do not mix the two rows.

This matters most on a small card. A model can leave little headroom for a second resident model or a large display workload. The VRAM headroom guide explains why a model that barely fits is not yet a useful route.

Which Ollama setting should I record?

Record keep_alive with the request. It tells the runtime how long it may keep the model ready after the response. A test can use a short keep-alive value, but I still send an explicit unload before I label the next run cold.

Also record the model tag, context size, output cap, and runtime version. A model can use a different amount of VRAM after a context or runtime change. The unload receipt does not replace those fields.

I treat a missing idle reading as missing evidence. The run may still show generation speed. It cannot support a claim about clean release, idle memory, or cold-load conditions.

When should I keep a model loaded?

Keep a model loaded when the next request is part of the same measured warm lane. That can reduce wait time for a real workload. State the lane in the record.

Unload before a new model, quant, context, or runtime test. Unload before a fit decision on a small GPU. Unload before a test that must measure first-request time.

This is a small step, but it keeps local measurements honest. A model that looks fast only because another run left memory warm has not earned a route.

Accompanying prompt

What the prompt does: This prompt makes a local model test receipt that separates loaded, unloaded, warm, and cold GPU states.

Copy/paste this prompt:

Role: Local LLM test reviewer

Context:
I will give you a model tag, runtime version, GPU memory readings, request settings, and an unload result.

Task:
1. Label the run as cold, warm, or unknown.
2. List the evidence for loaded and idle GPU memory.
3. State whether the runtime released the model.
4. List the missing facts before another model comparison.

Output:
- A short test receipt.
- A pass, hold, or rerun decision.
- The next exact measurement.

Constraints:
- Keep model, quant, context, and runtime details in the receipt.
- Do not infer quality from a memory result.
- Do not call a run cold without an idle reading after unload.

Copy the block above.

Get the next measured local AI run in the 5090 Reports email list.

---

Get the local AI lab notes (benchmark rows, VRAM fit, quant choices, what runs on consumer GPUs), M-F only when there is something worth sending: https://bmdpat.com/newsletter?utm_source=blog_md&utm_medium=aeo&utm_campaign=unload-local-llms-after-tests-2026

---

Originally published on bmdpat.com. I run a one-person AI agent company and write about what actually works.

Want these in your inbox? Subscribe to the newsletter - no spam, unsubscribe anytime.