Running a language model on your own machine has obvious appeal: your data stays local, there's no per-token bill and you can work offline. But the choice usually starts in the wrong place, with a parameter count or a leaderboard. A model that excels at code completion may be a poor choice for summarising long documents, and a model that "fits" on paper may run out of memory as soon as the conversation grows.
Here's a more reliable process, in six steps.
1. Start with the work, not the model
Before comparing models, define what you need them to do. Four questions settle most of it:
- Privacy: can the data leave the device at all?
- Latency: do you need interactive responses, or is this a batch job?
- Quality: which errors would make a result useless?
- Mode: text only, or also images, audio and files?
Different scenarios also call for different measurements:
| Scenario | Priority | What to measure |
|---|---|---|
| Chat and writing | Perceived latency | Time to first token, consistency |
| Long documents | Usable context | Retrieval of details and citations |
| Programming | Correctness | Tests passed, unnecessary changes |
| Batch processing | Throughput | Tokens per second, energy cost |
2. Budget memory, with margin
Available memory (RAM or VRAM, depending on how you run the model) is usually the bottleneck. The key mistake is treating the model file size as the whole cost. Context, cache and the runtime itself all need extra space, and the operating system needs some too.
Plan in this order: define inputs and quality, set a real memory budget, choose a size and format that fits within it, then test with your own cases. Don't plan right up to the limit; leave room for the system.
3. Quantization is a trade-off, not a free shortcut
Reducing numerical precision makes a model smaller and can speed up inference, but the effect on quality isn't uniform. Two models with the same quantized size can behave differently on your task.
A simple rule: compare at least two quantization levels using the same questions and the same settings. Save the outputs so you can review actual differences rather than relying on impressions.
4. The maximum context window isn't the usable context
The advertised context limit describes how much a model can accept. It doesn't tell you how accurately it will use all of that input, or whether it fits in your memory. The attention cache grows as the conversation gets longer and can become the largest consumer of memory, even after the weights are loaded.
Start with the smallest context that solves the task. For long documents, compare sending the whole file with retrieving only the relevant passages. Then measure:
- Memory budget: record weights, cache, runtime and system margin separately.
- Retrieval: check the passages, the overlap and the quality of the references returned.
- Degradation: put key facts at the start, middle and end of the input to detect where attention drops off.
- Latency: measure prefill and generation time at realistic lengths.
A fluent answer doesn't prove the model actually used the whole input correctly.
5. Check licence, provenance and safety
Before integrating a weights file, read the licence for that specific model and variant. "Open" can mean access to the weights without the same rights to commercial use, redistribution, modification or training derivatives. Keep a record of the repository, the version and the licence text you checked.
Treat models, prompt templates and downloaded files as external dependencies. Verify where they come from, be careful with formats that can execute code, limit what the process can access, and don't connect tools with broad privileges during early testing. Running locally gives you more control over your data, but it doesn't remove the risks of generated content or of the supply chain.
A minimal pre-install record: file identifier and hash, licence, author or repository, format, compatible runtime, download date, process permissions and how to remove it.
6. Run a reproducible test
Gut feeling is not an evaluation. Make the test repeatable:
- Freeze the set. Use the same ~20 real tasks for every variant.
- Record the environment. Note hardware, runtime version, context size and parameters.
- Grade blind. Hide the model name when you score responses.
If you use Ollama, you can inspect a model's details before running it:
ollama show model-name
ollama run model-name
For lower-level control and GGUF formats, llama.cpp is the usual reference project.
Putting it together
The best local model is the one that solves your task within your memory budget, under a licence you can live with, and that holds up in a test you can repeat. Parameter count and leaderboard rank are inputs to that decision, not the decision itself.
Inferama's guide to choosing a local model expands on each step, and its benchmarks page helps you shortlist candidates before you run your own tests.
Originally covered in more depth at https://www.inferama.com/en/guias/elegir-modelo-local
Top comments (0)