Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
One of the most useful ideas in modern LLM engineering is also one of the least glamorous:
Ask the model the same question 16 times, then keep the best answer.
That sounds almost embarrassingly simple.
But it contains a surprisingly deep idea about how we should think about intelligence in language models.
A language model does not have to produce the right answer on its first attempt. It only has to produce a good answer often enough, and we need a mechanism that can recognize the good attempts.
That turns inference into a search problem:
generate
|
+--------+--------+
| | |
y1 y2 ... yN
| | |
+--------+--------+
|
evaluate
|
choose y*
This is Best-of-N (BoN).
It is cheap conceptually, requires no retraining, is embarrassingly parallel, and has appeared repeatedly in important LLM research under names such as verifier-guided decoding and rejection sampling.
More importantly, it gives developers a useful mental model:
A model's one-shot capability is not the same thing as its capability under search.
That distinction becomes increasingly important as models get expensive to train but inference compute becomes an increasingly important lever.
1. The intuition: don't make the model smarter, give it more shots
Imagine a coding task where your model produces a correct solution 40% of the time.
With one sample:
P(correct) = 0.40
Now generate 8 independent solutions and take the best one using a reliable test suite.
The probability that all eight are wrong is:
P(all wrong) = 0.60^8
~= 0.0168
Therefore:
P(at least one correct)
= 1 - 0.60^8
~= 0.983
So the theoretical success rate has gone from 40% to about 98%.
This is the basic magic of Best-of-N.
The model has not improved.
Its weights have not changed.
Its context window has not changed.
We simply exploited the fact that generation is stochastic.
There is an important conceptual inversion here.
The naive way to improve an LLM is:
better model -> better answer
Best-of-N says:
same model
|
+--> attempt 1
+--> attempt 2
+--> attempt 3
...
+--> attempt N
|
verifier
|
winner
We are turning inference compute into search.
This basic generator/verifier pattern was already demonstrated convincingly by Karl Cobbe and colleagues at OpenAI in 2021. They introduced GSM8K, generated many candidate solutions to math word problems, trained a separate verifier to judge them, and selected the highest-scoring candidate. They found that verification improved performance and scaled effectively with additional data.
That paper is one of the cleanest demonstrations of the principle because the verifier can ultimately be grounded in something objective: is the mathematical answer correct?
2. A piece of LLM history: this was not invented as a clever prompting trick
The idea became particularly concrete with WebGPT.
In 2021, Reiichiro Nakano, Jacob Hilton, John Schulman and colleagues built a version of GPT-3 that could browse the web, gather references, and answer questions.
The interesting part was not merely the browser.
They trained a reward model to predict human preferences and then used rejection sampling: generate several answers, score them, and keep the best one.
Their largest model used best-of-64 at inference time.
The result was striking. On their ELI5 evaluation, the 175B best-of-64 model was preferred by human evaluators 56% of the time over answers written by their human demonstrators, and 69% of the time over the highest-voted Reddit answers.
That is an important historical detail because Best-of-N was not merely a research curiosity.
It became part of the actual engineering strategy for getting more performance out of a fixed model.
And there is an even more interesting lesson in the result.
The improvement did not require changing the model's parameters.
They changed how many chances the model got to answer.
3. The math: why N helps, and why the returns eventually suck
The simple model is:
p = probability one sample is good
N = number of samples
Assuming independence:
P(at least one good sample)
= 1 - (1 - p)^N
This gives a useful back-of-the-envelope calculator.
Suppose your baseline success probability is 20%.
N = 1 -> 20%
N = 2 -> 36%
N = 4 -> 59%
N = 8 -> 83%
N = 16 -> 97%
So the early returns can be spectacular.
But notice what happens after that.
Going from 1 -> 2 buys you 16 percentage points.
Going from 8 -> 16 buys you about 14 points.
Going from 16 -> 32 buys you only about 3 points.
This is the classic shape of test-time scaling:
quality
^
| ______
| ____/
| ____/
| ___/
| __/
|_____/
+---------------------------> N
There is another reason the simple equation is optimistic.
Samples are not actually independent.
Ask GPT to solve the same problem 32 times and you will not get 32 independent minds.
You will often get variations on the same failure mode.
If the model systematically believes a false premise, all 32 samples may share it.
So in practice, what matters is not merely N.
It is closer to:
effective N = number of genuinely different useful attempts
Temperature, prompt variation, structured decomposition, different tool-use trajectories, and different reasoning paths can all increase effective diversity.
This is one reason Best-of-N works dramatically better on some tasks than others.
4. The real trick is not N. It is the selector.
Here is where the subject becomes more interesting.
Generating 32 candidates is easy.
Knowing which one is best is the hard part.
There are three broad possibilities.
A. Ground-truth verifier
This is the ideal case.
For code:
candidate -> compiler -> tests -> pass/fail
For mathematics:
candidate -> symbolic checker -> correct/incorrect
For structured output:
candidate -> JSON parser -> schema validator
For SQL:
candidate -> database -> execution result
When you have a reliable external verifier, Best-of-N becomes extremely powerful.
The model can hallucinate.
The compiler cannot.
That asymmetry is enormously valuable.
B. Learned verifier
Sometimes there is no executable notion of correctness.
"Is this answer useful?"
"Is this explanation clear?"
"Is this response factually accurate?"
You can train another model to score candidates.
This is essentially the reward-model setup used in RLHF-style systems.
But now an ugly problem appears:
You are optimizing against a proxy for quality rather than quality itself.
Leo Gao, John Schulman and Jacob Hilton studied exactly this problem in Scaling Laws for Reward Model Overoptimization.
They showed that as you optimize harder against a learned reward model — including with Best-of-N — you can eventually make the reward-model score rise while the true underlying quality gets worse.
That's a direct manifestation of Goodhart's law.
In abstract form:
true quality: Q(y)
proxy score: R(y)
choose argmax R(y)
as N increases:
max R(y) -> rises
true Q(y) -> eventually may fall
The more candidates you search through, the more aggressively you are probing the weaknesses of your evaluator.
This is perhaps the single most important caveat about Best-of-N.
C. Consistency instead of scoring
There is another beautiful variant: self-consistency.
Instead of asking "which answer does my verifier like most?", ask:
"Which answer keeps recurring across independent reasoning attempts?"
Xuezhi Wang and colleagues introduced this idea in 2022.
Generate multiple reasoning paths and choose the most consistent final answer. On GSM8K, they reported a 17.9 percentage-point improvement from self-consistency over the corresponding chain-of-thought baseline.
So:
Best-of-N:
generate N -> score candidates -> select winner
Self-consistency:
generate N -> extract answers -> vote
The latter is essentially Best-of-N where the selector is a voting mechanism.
5. Why process verification is more interesting than answer verification
Suppose a model produces this:
Step 1: 17 * 24 = 408
Step 2: ...
Step 3: therefore x = 19
An outcome verifier only sees:
x = 19
A process verifier can inspect every intermediate step.
This distinction became especially important in OpenAI's 2023 work by Hunter Lightman and colleagues, Let's Verify Step by Step.
They compared outcome supervision with process supervision, where humans label whether individual reasoning steps are correct.
Their process-supervised model reached 78% accuracy on a representative subset of the MATH test set, and the authors released PRM800K, a dataset containing 800,000 step-level human feedback labels.
Why does that matter for Best-of-N?
Because instead of:
candidate A -> 0.81
candidate B -> 0.79
candidate C -> 0.42
you can evaluate the trajectory:
candidate A:
step 1 -> good
step 2 -> good
step 3 -> suspicious
step 4 -> good
candidate B:
step 1 -> good
step 2 -> good
step 3 -> good
step 4 -> good
That gives your search mechanism much more information.
It also suggests a broader design pattern:
Generation produces possibilities. Verification supplies intelligence about which possibilities deserve more compute.
Once you see LLM systems this way, Best-of-N starts looking less like a decoding hack and more like the simplest member of a family of search algorithms.
6. The developer economics: when should you actually use it?
The great thing about Best-of-N is that its cost structure is unusually transparent.
Suppose one generation costs:
G = generation cost
V = verification cost
N = number of candidates
Then approximately:
total cost ~= N * (G + V)
There is no magic.
If one request produces 2,000 output tokens and you sample 16 candidates, you're asking the system to produce roughly:
16 * 2,000 = 32,000 output tokens
So you should not blindly use N=32 just because it improves benchmark accuracy.
The real engineering question is:
value of additional success
>
cost of another sample
There are, however, two useful economic properties.
First: the samples can be parallel
If you have sufficient serving capacity:
+--> sample 1
+--> sample 2
request -----+--> sample 3
...
+--> sample N
The compute cost scales roughly with N, while wall-clock latency can remain closer to one generation plus verification.
That makes Best-of-N particularly attractive for batch workloads and latency-insensitive tasks.
Second: verification is often much cheaper than generation
For code generation:
LLM generation: expensive GPU tokens
compiler/test: cheap CPU execution
That is an excellent economic asymmetry.
A particularly strong production architecture is therefore:
LLM
|
+----------+----------+
| | |
patch 1 patch 2 patch N
| | |
+----------+----------+
|
test suite
|
winner
Imagine a code-generation system where one attempt has a 30% chance of passing the test suite.
With N=10:
P(success)
= 1 - 0.7^10
~= 97.2%
Even with substantial correlation between attempts, that can be a very attractive trade.
But only if the test suite actually measures what you care about.
A weak test suite turns Best-of-N into an extremely efficient machine for finding code that passes your tests while violating your actual requirements.
That is Goodhart again, only now with software tests.
7. What Best-of-N is really teaching us about LLM architecture
There is a tempting interpretation:
"Best-of-N is a hack we use until models get smarter."
I think that is too narrow.
A deeper interpretation is that modern LLM systems are increasingly becoming generator + evaluator + search systems.
A single language model call looks like:
prompt -> answer
A more capable inference system looks like:
prompt
|
+--> generate candidate
+--> generate candidate
+--> generate candidate
|
v
evaluate
|
v
select
|
v
answer
And more sophisticated systems can recursively extend this:
generate
|
evaluate
|
repair
|
generate again
|
evaluate again
|
...
At that point you are moving toward rejection sampling, verifier-guided search, process reward models, tree search, and eventually RL.
Best-of-N is just the simplest point on that spectrum.
It also gives you a useful way to think about model improvements.
A better base model increases the probability that a randomly sampled candidate is good:
p: 0.20 -> 0.30
A better verifier increases the probability that the good candidate actually gets selected.
And more inference compute increases the number of chances you have to discover one.
So there are three separate scaling axes:
generator quality
+
candidate diversity
+
selector quality
=
inference-time capability
That decomposition is extremely useful when debugging an LLM application.
If Best-of-N doesn't improve your system, there are only a few basic explanations:
- The generator almost never produces the right answer.
- The samples are too correlated.
- The verifier cannot recognize the right answer.
- You have pushed N beyond the useful part of the scaling curve.
- You are optimizing a proxy that is easier to game than the actual objective.
And that last one is the dangerous case.
The beautiful thing about a compiler, a mathematical checker, or a real test suite is that the selector has very little room to lie.
The dangerous thing about a learned reward model is that the model eventually gets to ask:
"What output will make my evaluator happiest?"
rather than:
"What output is actually correct?"
That is why some of the most important progress in test-time scaling has been not simply generating more candidates, but building better verifiers. Cobbe et al. demonstrated the basic generator-verifier loop; Lightman et al. showed why looking inside the reasoning process can make verification substantially more useful; Gao et al. showed the corresponding failure mode when the verifier itself becomes the target.
Conclusion: Intelligence may increasingly look like search
The simplest implementation of Best-of-N is barely ten lines of code:
candidates = [model(prompt, temperature=0.8) for _ in range(N)]
scores = [verifier(x) for x in candidates]
answer = candidates[argmax(scores)]
Yet behind those lines sits a fairly profound idea.
A model does not necessarily need to become uniformly better at producing the answer.
It can become more useful because we give it:
- more attempts,
- better diversity,
- a better judge,
- and more compute to search the space.
That changes the economics of intelligence.
Training gives you a better distribution.
Inference lets you explore that distribution.
Verification tells you where the good parts are.
And Best-of-N is the simplest possible machine for putting those pieces together.
The question I keep coming back to is this:
For the LLM systems you're building, where is the bigger untapped opportunity: making the generator better, or getting much better at judging and searching the outputs it already knows how to produce?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)