Give a language model a CV and ask for a score from 1 to 10, and you get a number and a confident reason. Ask again and the number can move. The usual advice is "set temperature to 0", which is correct but leaves two questions open: how much does the score actually move with default settings, and what does temperature 0 not fix? Both can be measured on a laptop in a few minutes, because the score is a single token and the model hands you its probability directly.
The setup below uses Qwen2.5-3B-Instruct (4-bit, MLX) with mlx-lm 0.32.2 on an 8 GB M1 Mac. The prompts are the ones from a CV-scoring demo in the osFoundry desktop app's Models → Playground: a six-line CV for a junior data analyst role, followed by "Score this CV from 1 to 10 for a junior data analyst role. Start with the score, then one reason." The app runs local models through llama.cpp's llama-server, so the sampler settings come from there.
The defaults you get when you don't choose
When a request carries no temperature, llama-server samples with its own defaults. The build bundled with the app (version 10358) lists them in --help:
--temp, --temperature N temperature (default: 0.80)
--top-k N top-k sampling (default: 40, 0 = disabled)
--top-p N top-p sampling (default: 0.95, 1.0 = disabled)
--min-p N min-p sampling (default: 0.05, 0.0 = disabled)
-s, --seed SEED RNG seed (default: -1, use random seed for -1)
These apply to every request that doesn't set its own values. When the app starts llama-server, it only adds --temp if a temperature has been configured for the server (local_inference.rs), so a request without a temperature samples at 0.8 with a random seed, which is also what most llama.cpp front ends do out of the box.
Making the score the next token
A free-form answer can start with "Score: 7", "7/10" or "I would rate this CV a 7", which makes the score hard to locate. The fix is to pre-fill the start of the reply, so the very next token the model produces is the digit:
import mlx.core as mx
from mlx_lm import load, generate
from mlx_lm.sample_utils import make_sampler
model, tok = load("mlx-community/Qwen2.5-3B-Instruct-4bit")
DIGITS = {d: tok.encode(str(d))[0] for d in range(1, 10)}
def ids_for(cv, question, prefill="Score: "):
msgs = [{"role": "user", "content": cv + "\n\n" + question}]
text = tok.apply_chat_template(msgs, add_generation_prompt=True, tokenize=False)
return tok.encode(text + prefill)
def digit_probs(ids, T=1.0):
logits = model(mx.array([ids]))[0, -1].astype(mx.float32)
p = mx.softmax(logits / T)
return {d: float(p[t]) for d, t in DIGITS.items()}
Qwen's tokenizer splits numbers into single digits, so "7" is one token and "10" would start with the token "1". One forward pass over the 173-token prompt gives the whole distribution of the score.
The score is a distribution, not a number
For the English CV, at temperature 1 (the model's raw probabilities):
| Score | Probability |
|---|---|
| 7 | 74.2% |
| 6 | 23.3% |
| 8 | 1.3% |
| 5 | 1.2% |
The model doesn't "think" the CV is a 7. It thinks it is probably a 7, with a one-in-four chance of a 6. Temperature divides the logits before the softmax, which sharpens or flattens this:
| Temperature | P(7) | P(6) | P(8) |
|---|---|---|---|
| 1.0 | 74.2% | 23.3% | 1.3% |
| 0.8 | 80.2% | 18.9% | 0.5% |
| 0.5 | 91.0% | 9.0% | 0.0% |
| 0.2 | 99.7% | 0.3% | 0.0% |
llama-server doesn't apply temperature alone. Its default chain runs top-k 40, then top-p 0.95, then min-p 0.05, and temperature last. Applied to these logits, top-p already cuts everything after 7 and 6 (together they are over 95%), so the default sampler picks between exactly two scores:
def llamacpp_default(logits, temp=0.8, top_k=40, top_p=0.95, min_p=0.05):
order = mx.argsort(-logits)[:top_k]
kept = logits[order]
p = mx.softmax(kept)
n = int(mx.sum(mx.cumsum(p) < top_p).item()) + 1
idx = [i for i in range(n) if p[i].item() >= min_p * p[0].item()]
q = mx.softmax(kept[mx.array(idx)] / temp)
return {tok.decode([int(order[i])]): float(q[j]) for j, i in enumerate(idx)}
# -> {'7': 0.8093, '6': 0.1907}
Then I sampled it for real: 200 generations from the same prompt with mlx-lm's sampler set to the same four values (make_sampler(temp=0.8, top_p=0.95, min_p=0.05, top_k=40)).
| 7 | 6 | |
|---|---|---|
| Predicted by the chain | 80.9% | 19.1% |
| Observed, 200 runs | 162 (81%) | 38 (19%) |
So with default settings, about one run in five scores this CV a 6 instead of a 7. If you score two candidates once each and one gets a 6 and the other a 7, there is a real chance the gap is the draw.
A flatter case: the same CV in Japanese
The same CV and question in Japanese (222 tokens, pre-fill 「点数：」) gave a much less decided model:
| Score | Probability at T=1 | Observed, 200 runs at the defaults |
|---|---|---|
| 8 | 39.8% | 108 |
| 9 | 23.8% | 44 |
| 7 | 19.4% | 40 |
| 10 | (starts with "1": 4.1%) | 8 |
Here the most likely score wins only about half the time, and four different scores came out of 200 runs. How much a score wobbles is not a fixed property of the model. It depends on how close the top candidates are for this particular input, which you can only see by looking at the probabilities.
What temperature 0 fixes
With temperature 0 (greedy decoding: always the most likely token), I generated the full 40-token answer 20 times for each language. There was 1 distinct text per language, character for character. For the English CV it is "7" followed by the same reason every time, and for Japanese 「8点」.
A fixed seed does the same for sampling: 10 runs with mx.random.seed(42) reset before each gave 1 distinct text. That is useful when you want variety but need to reproduce a particular run. It is not a fairness tool, though. With a seed, a different CV gets a different sequence of draws, so two candidates are still scored by different luck.
One condition applies to all of this: every run here was a single request on an idle model. Servers that batch several requests together can produce tiny floating-point differences between runs, which only matter when two tokens are nearly tied. That was not measured here.
What temperature 0 doesn't fix
Greedy decoding makes the output a function of the input. It doesn't make it a stable function. I made four edits that should not change a CV's score and took the greedy score and the probabilities each time:
| Edit (English) | Greedy score | P(top) | Runner-up |
|---|---|---|---|
| none | 7 | 74.2% | 6 at 23.3% |
| Skills line moved up | 7 | 69.6% | 6 at 28.1% |
| Name changed (Mia Chen → James Miller) | 7 | 68.5% | 6 at 28.5% |
| One extra blank line | 7 | 71.0% | 6 at 26.1% |
| Question reworded | 6 | 67.5% | 5 at 19.3% |
The reworded question was "Rate this CV for a junior data analyst role on a 1 to 10 scale. Give the score first, then one reason." Same task, same CV, and the greedy score drops from 7 to 6. In Japanese, all five versions stayed at 8 at temperature 0, but the top probability never went above 44%, and the runner-up switched between 9 and 7 depending on the edit.
So temperature 0 removes the randomness between runs, not the sensitivity to the prompt. For comparing CVs this means:
- Use temperature 0 and the exact same question for every item you compare, character for character. Change the question, and re-score everything.
- Keep the probability, not only the score. llama-server's
/completionendpoint returns token probabilities if you passn_probs, and with the pre-fill trick above the first token is the score. A 7 at 74% and an 8 at 40% are not the same kind of answer. - Treat a one-point gap at low confidence as a tie, and read the reason the model gives before trusting either number.
Doing it in the app
In the osFoundry desktop app, the same experiment is three panels with one local model in Models → Playground. Run the prompt once with the defaults to see the spread, then open the settings row and set Temp to 0.
The temperature and top-p glossary entries cover the two settings in more detail. The app is free from the osFoundry download page.
Top comments (0)