I set temperature to 0, ran the same document through the same model twice with
the same prompt, and got different answers. Not subtly different. Roughly a third
of the extracted records changed.
The cause is not sampling. It is batching.
| same document, same model, same prompt, temperature 0 | records | byte-identical | Jaccard |
|---|---|---|---|
--concurrency 1, run twice |
137 / 137 | yes | 1.0000 |
--concurrency 4, run twice |
160 / 164 | no | 0.705 |
| concurrency 1 vs concurrency 4 | 137 / 160 | no | 0.549 |
Serving one request at a time is exactly reproducible. Serving four at a time is
not, and it is not reproducible against itself either.
Why this happens
Batched inference does not compute the same arithmetic as unbatched inference.
Different batch shapes select different kernels, matrix multiplications get
tiled differently, and floating-point addition is not associative. The logits
come out fractionally different, and anywhere two tokens were nearly tied,
argmax picks the other one. Temperature 0 removes sampling randomness. It does
nothing about the arithmetic underneath.
This is well known in the abstract. What surprised me is the size: 30 percent of
records, not 3.
How I found it, which is the embarrassing part
I was not looking for this. I was testing whether a new filter reduced false
positives, comparing a run with the filter against a run without it.
One name showed up as a false positive only in the filtered run.
A filter cannot add records. It can only remove them. So either my filter was
broken, or the two runs had not seen the same model output at all. It was the
second. My A/B test had been comparing two different underlying extractions and
attributing the difference to the flag.
That experiment was worthless and I nearly published its result.
The claim this retires
I had a determinism check in the repo already. It ran the same corpus five times
and confirmed the final answer was identical each time. It passed. I cited it.
It was measuring almost nothing. The pipeline extracts hundreds of records per
document and then reduces them in code to one answer. A reduction like "which of
these three names appears in the most sections" is extremely tolerant: you can
churn a third of the evidence and still land on the same name, because the
argmax has a margin.
So the final answer was stable while the thing underneath it was not. The check
confirmed the margin was wide, and I read it as confirming the system was
deterministic.
Record-level agreement is the measurement. Answer-level agreement is a
consequence, and a weak one.
What it did and did not change
I re-ran the headline benchmark at concurrency 1, where output is byte-identical.
| metric | batched | deterministic |
|---|---|---|
| the headline metric | 0.980 | 0.980 |
| a positional ordering task | 0.942 | 0.904 |
| overall | 0.899 | 0.870 |
The headline number did not move at all. That is not luck: it counts how many
distinct regions of a document a name appears in, and it does not care which
particular quoted span proves the name was in a region, only that some span
does. Redundant evidence absorbs the churn.
The positional tasks moved by up to 0.04, because those depend on the single
earliest or latest piece of evidence, and the extremes are exactly where losing
30 percent of records bites.
So the honest summary is: the noise is real, it reaches some metrics and not
others, and you cannot know which without measuring.
What to do
Report concurrency with your numbers. It is a experimental condition, not a
performance tuning detail. Two papers can differ on it and disagree for no other
reason.
Measure agreement at the record level, not the answer level. Run the same
input twice, diff the intermediate output, report Jaccard. If your pipeline has
no intermediate output to diff, that is worth fixing on its own.
Run final numbers at concurrency 1. It cost me less than I expected: about
20 minutes to about 23 for a nine-document sweep, because the server was not
parallelising as much as the flag implied. Measure the cost before assuming you
cannot afford it. Iterate batched, publish serial.
Be suspicious of any A/B where the treatment could not have caused the
difference you see. That single impossible data point was the only reason I
found this. If the filtered run had merely looked better, I would have shipped
it.
Top comments (1)
Really strong write-up - the record-level versus answer-level distinction is the part that generalizes furthest. Two additions from the same mechanism family: 1. Continuous batching makes the batch itself a hidden input. With vLLM-style or TGI-style serving, the batch composition changes between runs even at the same nominal concurrency: request arrival order shifts padding and kernel shapes, so you get run-to-run disagreement (your 0.705 Jaccard), not just disagreement against a serial baseline. A fixed seed does not help here - seeds select the RNG stream, not the kernel choice or tiling. Deterministic algorithm flags (PyTorch exposes these) can restore stability, but generally at a throughput cost and only for a fixed batch shape. That is why running the final eval at concurrency 1 is such practical advice for anyone reporting benchmark numbers. 2. The same aggregation blindness shows up in eval metrics. pass@1 is a reduction over many token-level decisions, the way your final answer is a reduction over records - so it can look stable while the underlying output distribution drifts. If a harness only ever reports the top-level number, the states same-behavior and different-behavior-that-reduces-the-same are indistinguishable. Your between-runs Jaccard is the cheaper cousin of what pass@k does for sampling: a view one level down. Curious question: did the churned records concentrate in near-tie regions (small logit gaps) or spread across the whole distribution? If the former, a histogram of logit gaps over your decisions would predict which downstream metrics are fragile before you run the next A/B.