Stop Using Temperature 0 for LLM Evals: Why It Breaks Benchmarks

#llm #benchmarking #temperature #evaluation

Temperature 0 Isn't Deterministic — and That Ruins Your Evals

Everyone does it. You're setting up an LLM benchmark, maybe comparing GPT-4 against Claude on some reasoning task, and you dutifully set temperature=0 because you want "reproducible results." The API docs say it's deterministic. The tutorials all do it. Your team lead insists on it.

Here's the problem: temperature 0 doesn't give you what you think it does, and it's quietly destroying the validity of your evals.

I'm not talking about the obvious issues like reduced output diversity or mode collapse. I'm talking about something more fundamental: temperature 0 fundamentally changes the inference process in ways that don't match production use, and the "determinism" you're chasing is mostly a myth anyway. You're benchmarking a different system than the one your users will interact with.

Let me show you why this matters, using real experiments with actual variance numbers.