LLM Accuracy vs Reproducibility: Are We Measuring Capability or Sampling Luck?

Why identical prompts can produce different reasoning paths — and why that matters for evaluation

LLM Accuracy vs Reproducibility: Are We Measuring Capability or Sampling Luck?

When working with LLMs, we often rely on metrics like accuracy, pass rates, or benchmark scores to evaluate performance.

But a simple experiment reveals something that’s easy to overlook.

The Setup
Same prompt
Same model snapshot
Same temperature
Same sampling configuration

Run the same input multiple times.

The Observation

The outputs don’t just vary slightly.

They often follow completely different reasoning paths.

In some cases, the structure of the response changes significantly — different intermediate steps, different logic, different phrasing.

And yet:

The final answer may still be the same.

Why This Matters

Most evaluation frameworks implicitly assume:

Same input → consistent reasoning process → comparable outputs

But what we actually observe looks more like:

Same input → multiple competing generation paths → occasional convergence to a correct answer

This introduces a subtle but important issue

If outputs are path-dependent, then:

A correct answer does not necessarily imply a stable reasoning process
A passing result does not guarantee reproducibility
Aggregate benchmark scores may hide significant variability
A Practical Question for Developers

If your system depends on LLM outputs:

How do you define reliability?
Is a single correct response enough?
Or do you need consistency across runs?
A Deeper Concern

Are we measuring model capability —
or the probability of sampling a favorable trajectory?

Closing Thought

This may not be a problem of “better benchmarks.”

It may be a question of:

what we assume benchmarks are actually measuring.