Pick a small model for your actual workload
Start with a model that supports your input. Try Qwen3 4B for a small text baseline, Gemma 3 4B if you also need image input, or Phi-4-mini as a second text baseline for math or structured tasks. These are candidates to test on your machine. Their relative quality depends on your prompts, language, and acceptance criteria.
Estimate the memory you can use
Four billion parameters at exactly four bits each would take about 2 GB of raw weight storage. The files can be larger, and loaded models also need room for runtime buffers and the key-value cache. Leave memory for the operating system and any other work running alongside inference. A larger context needs more memory even when the downloaded weights are unchanged. Ollama context-length documentation
For a repeatable comparison, try the same prompts first with a short context, for example 2,048 tokens, and then with the context your workload actually needs. Record any swapping or CPU offloading along with latency. A model fitting at the first setting does not establish that it fits at the second.
Run a first comparison with Ollama
ollama run qwen3:4b
ollama run gemma3:4b
ollama run phi4-mini
ollama ps
Run the model commands one at a time and exit the previous chat before starting the next. Use ollama ps while a model is loaded to inspect its allocated context and CPU/GPU split. Keep the same context and prompt settings when comparing memory and runtime. Test image inputs separately against models that actually support them.
Keep the smallest model that passes
Prepare a small fixed set of representative prompts and expected outcomes. Examples include summarising a short passage with required facts and producing JSON that validates against the schema you use. Measure time to the first token and output rate on the same machine. Record failed outputs and peak memory rather than averaging them away. Save the exact model tag, quantization when available, and context setting so the result can be reproduced.
This edition was prepared with an AI agent. The technical references were checked during editing.
Top comments (0)