Hello, I'm Shrijith Venkatramana. I'm building git-lrc, an AI code reviewer that runs on every commit. Star Us to help devs discover the project. Do give it a try and share your feedback for improving the product.
There is a peculiar thing happening in modern LLMs.
For years, the dominant recipe was:
Make the model bigger. Train it on more data. Spend more compute.
That recipe still matters. But increasingly, another knob is becoming just as interesting:
Give the model more compute when it is actually solving the problem.
Instead of asking a model to produce an answer in one forward pass, we can let it generate several candidate solutions, inspect its own work, backtrack, verify intermediate steps, search over alternatives, call tools, or simply spend more tokens thinking.
This is generally called test-time compute or inference-time compute.
The interesting part is that this changes the economics and architecture of LLM systems. We are no longer thinking of an LLM as:
prompt -> model -> answer
but increasingly as:
-> candidate 1 -
/ \
prompt -> reason/search -> candidate 2 -> verify/select -> answer
\ /
-> candidate 3 -
The model becomes something closer to a reasoning engine whose computational budget can be allocated per problem.
And there is a beautiful historical precedent: this idea is not particularly new.
In 2016, DeepMind's AlphaGo defeated Lee Sedol 4-1. The neural network was important, but AlphaGo did not simply ask the network, "What move should I play?" It combined neural networks with Monte Carlo Tree Search, spending substantial computation at inference time to explore possible futures. ([nature.com][1])
LLMs are now rediscovering a version of the same idea.
1. The basic intuition: intelligence is partly a compute-allocation problem
Suppose I ask you:
What is 17 x 23?
You probably answer immediately.
Now I ask:
Find all integers n such that n^2 + 3n + 2 is divisible by 17, subject to...
You might take out a piece of paper.
The difference isn't necessarily that you suddenly became a more capable mathematician. You allocated more computation to the problem.
An LLM can do something analogous.
A conventional language-model inference looks roughly like:
x -> Transformer(x) -> y
where x is the prompt and y is the generated answer.
But autoregressive generation already contains a primitive form of test-time computation:
x
-> token 1
-> token 2
-> token 3
-> ...
-> token N
Every additional generated token requires another model evaluation.
So if the model has learned that difficult problems benefit from longer reasoning traces, we can simply give it a larger inference budget.
This is the central idea behind the reasoning-model transition that became highly visible with OpenAI's o1. OpenAI reported that o1's performance improved not only with additional training compute, but also with additional time spent thinking at test time. ([openai.com][2])
That distinction is important:
training-time compute:
improve the model itself
test-time compute:
give the model more opportunity to solve this particular problem
The second is potentially much more economically interesting than it initially sounds.
If 99% of your requests are easy, you don't necessarily want to build a model that is permanently expensive enough to solve the hardest 1%.
Instead:
easy problem -> small compute budget
medium problem -> medium budget
hard problem -> large budget
extremely hard -> search / tools / verification / agents
That's adaptive computation.
2. The first trick: just sample more answers
The simplest form of test-time compute is almost embarrassingly straightforward:
Ask the model multiple times.
Suppose the model has probability p of solving a particular problem correctly in one independent attempt.
If we generate N attempts and can identify the correct one, the probability that at least one is correct is:
P(at least one correct) = 1 - (1-p)^N
Suppose:
p = 0.60
N = 1
P(correct) = 60%
With 10 independent attempts:
P(at least one correct)
= 1 - 0.4^10
~= 99.99%
Obviously, there's a catch.
How do you know which answer is correct?
If you simply ask the model for ten answers and pick one arbitrarily, nothing has improved.
This leads to one of the most important ideas in test-time compute:
Generation and evaluation are separate computational problems.
You can spend compute generating possibilities, and then spend additional compute deciding among them.
For example:
generate
|
+------------+------------+
| | |
answer A answer B answer C
| | |
+------------+------------+
|
verifier
|
answer B
This is the same basic reason that best-of-N sampling can outperform a single sample.
It is also why verification becomes such a central problem.
3. Verification: the model doesn't just need to think — it needs to judge its thinking
Consider a programming problem.
You generate:
def solve(x):
...
There are two fundamentally different questions:
- Can the model generate a solution?
- Can we determine whether the solution is good?
For code, we have an unusually powerful verifier:
compile
|
unit tests
|
integration tests
|
correct / incorrect
That makes code one of the most attractive domains for test-time compute.
For mathematics, we might have:
candidate solution
|
symbolic checker
|
numerical checker
|
another LLM
|
final answer
For open-ended reasoning, the verifier becomes much harder.
This distinction motivated substantial research into process supervision. Rather than only rewarding whether the final answer is correct, OpenAI researchers including Hunter Lightman and colleagues studied rewarding correct intermediate reasoning steps. Their 2023 work, Let's Verify Step by Step, showed the value of process-level feedback for mathematical reasoning. ([huggingface.co][3])
The conceptual shift is subtle:
outcome supervision:
problem -> reasoning -> answer
^
reward
process supervision:
problem -> step1 -> step2 -> step3 -> answer
^ ^ ^ ^
reward reward reward reward
Why is that useful at inference time?
Because now the system can potentially ask:
"Which of these reasoning paths looks most promising?"
rather than blindly accepting the first complete answer.
This gives us something much closer to search.
4. From sampling to search: LLM reasoning starts looking like AlphaGo
Here's where things get genuinely interesting.
Imagine the model has generated:
Step 1
|
+-- Step 2A
| |
| +-- Step 3A
|
+-- Step 2B
|
+-- Step 3B
Instead of committing to one path, we can explore several.
Conceptually:
problem
|
+-----+-----+
| |
A1 B1
/ \ / \
A2 A3 B2 B3
| | | |
... ... ... ...
Each node represents a partial reasoning state.
A verifier estimates which branches look promising.
Then the system allocates more compute to promising branches.
That's a search algorithm.
This is precisely the intellectual connection to AlphaGo.
AlphaGo combined:
neural network
+
tree search
The neural network supplied learned intuition about the position, while search spent additional computation exploring possible futures.
Silver and colleagues' 2016 Nature paper described AlphaGo's combination of deep neural networks and Monte Carlo Tree Search, and the system subsequently defeated European Go champion Fan Hui 5-0 before its famous match against Lee Sedol. ([nature.com][1])
The same architectural decomposition makes sense for LLM reasoning:
LLM = learned heuristic
search = computational deliberation
The model doesn't have to encode the entire solution in a single deterministic trajectory.
It can instead provide a policy over possible reasoning trajectories, while inference-time computation explores those possibilities.
That is a much more powerful abstraction.
5. How much compute should we spend?
Now we get to the interesting engineering question.
Suppose you have:
Model A:
70B parameters
1 unit inference cost
70% accuracy
Model B:
200B parameters
3 units inference cost
75% accuracy
You might naturally choose Model B.
But suppose Model A can use test-time search:
Model A + 8x inference compute
-> 85% accuracy
Now the comparison isn't simply:
70B vs 200B
It is:
cheap model + more inference compute
vs
expensive model + less inference compute
This tradeoff was studied systematically by Charlie Snell, Jaehoon Lee, Kelvin Xu and Aviral Kumar in Scaling LLM Test-Time Compute Optimally Can Be More Effective Than Scaling Model Parameters for Reasoning, published at ICLR 2025. They investigated how inference-time compute can be allocated through mechanisms including search with process reward models and adaptive modification of the model's output distribution. ([openreview.net][4])
One of their important observations is that there isn't a universally optimal strategy.
The right allocation depends on things such as:
model size
problem difficulty
available inference budget
quality of the verifier
search strategy
This is intuitive if you think about it economically.
Suppose you have $1 of compute.
You could spend it on:
bigger model
OR
longer reasoning
OR
multiple samples
OR
verification
OR
tool calls
The frontier question is:
Where does the next dollar of compute buy the most probability of getting the answer right?
That's a scaling-law question, but at inference time.
6. The economics get weird: inference becomes an optimization problem
Here's a deliberately crude calculation.
Suppose a model costs:
$0.002 per 1,000 generated tokens
and an ordinary response uses:
500 tokens
Then:
cost = $0.001 / response
Now imagine a reasoning system that uses:
5,000 reasoning tokens
before producing its 500-token answer.
Its cost is roughly:
$0.011 / response
That's approximately 11x the generated-token cost.
For a consumer chatbot, that can be a serious difference.
For a coding agent that saves a developer 20 minutes, it might be trivial.
This produces a very different optimization target:
marginal inference cost
|
v
problem -> compute allocation -> expected success
|
v
business value
Suppose a coding task is worth $5 of engineering time if solved correctly.
Then spending another $0.02 on inference to increase success probability by 2 percentage points has an expected value of:
0.02 * $5 = $0.10
That's a very good trade.
But if the task is:
"Translate this sentence."
then spending $0.02 to move accuracy from 99.5% to 99.9% is probably terrible economics.
This suggests an architecture that looks more like a compute scheduler than a traditional LLM API:
+----------------+
| classify task |
+-------+--------+
|
+------------+-------------+
| | |
easy medium hard
| | |
1 sample 4 samples search
| | |
+------------+-------------+
|
verifier
|
answer
In other words, reasoning becomes a resource-allocation problem.
7. What this means for developers
This is probably the most practical way to think about test-time compute.
Don't treat the model as a function:
answer = model(prompt)
Treat it as a computational substrate:
candidates = generate(prompt, budget=B1)
scores = verify(prompt, candidates, budget=B2)
answer = select(candidates, scores)
And eventually:
state = initialize(problem)
while budget_remaining():
candidates = expand(state)
scores = evaluate(candidates)
state = select_and_expand(candidates, scores)
return best_answer(state)
That abstraction opens up several familiar techniques.
Best-of-N
Generate multiple solutions and select the best.
Useful when:
generation is stochastic
verification is cheap
solutions are relatively independent
Self-consistency
Generate several reasoning trajectories and take the consensus answer.
This works particularly well when multiple independent reasoning paths converge on the same answer.
Verifier-guided search
Generate partial solutions, score them, and expand promising ones.
generate -> score -> prune -> expand -> score -> ...
Tool-assisted verification
For programming:
generate code
-> compile
-> run tests
-> inspect failures
-> modify code
-> repeat
For mathematics:
generate proof
-> symbolic checker
-> identify failed step
-> revise
For research:
hypothesis
-> search
-> retrieve evidence
-> compare sources
-> revise
-> search again
At that point, you've crossed an important conceptual boundary.
You are no longer merely prompting an LLM.
You are building a search procedure around an LLM.
8. The deeper idea: intelligence becomes conditional computation
There is a broader implication here.
Traditional scaling asks:
How capable can we make the model?
Test-time scaling asks:
How much computation should we spend on this particular problem?
Those are very different questions.
Imagine two problems:
Problem A:
"Convert 37°C to Fahrenheit."
Problem B:
"Find a bug in this 20,000-line distributed system."
A fixed-compute model treats them similarly.
A reasoning system shouldn't.
Ideally:
Problem A
-> 1-2 forward passes
-> done
Problem B
-> inspect code
-> formulate hypotheses
-> search
-> run tests
-> revise
-> investigate failures
-> repeat
The computational budget becomes conditional on uncertainty and difficulty.
This is one reason reasoning models are more interesting than merely "LLMs that produce longer answers."
The real development is toward systems where:
model intelligence
+
search
+
verification
+
tools
+
adaptive compute
form a single inference process.
And there is an intriguing symmetry with classical AI.
AlphaGo showed that a learned model could provide intuition while search supplied additional computation. Modern reasoning LLMs are exploring the same basic division of labor in language and code. The difference is that the search space is now made of tokens, programs, proofs, hypotheses, tool calls and actions rather than Go moves. ([nature.com][1])
9. The catch: more compute is not automatically better
There is an easy mistake to make here.
Test-time compute isn't magic.
If the model's samples are highly correlated:
sample 1 -> same mistake
sample 2 -> same mistake
sample 3 -> same mistake
...
then generating 100 samples doesn't buy you much.
Likewise, a bad verifier can confidently select a bad answer.
You can even get a pathological system where:
more reasoning
-> more opportunities for error
-> more elaborate wrong explanations
So the quality of the search landscape matters.
A useful mental model is:
test-time performance
~
generation diversity
x verifier quality
x search efficiency
x compute budget
Not literally as a universal equation, but as an engineering decomposition.
This is why research on process reward models, self-verification and search is so important. A reasoning model needs not merely to generate possibilities, but to distinguish promising trajectories from bad ones. ([huggingface.co][3])
And it explains why the strongest systems increasingly resemble systems engineering problems rather than simply model-training problems.
Conclusion: the model may be the beginning of inference, not the end
The old mental model of an LLM was:
prompt
|
v
neural network
|
v
answer
The emerging one is:
+--> candidate
|
prompt -> model -> search -> verify
|
+--> candidate
|
+--> candidate
|
v
answer
That difference is enormous.
It means that after spending billions of dollars making models more capable during training, we can potentially get another axis of improvement by deciding how much computation to spend while solving each individual problem.
Sometimes that means simply generating more samples.
Sometimes it means thinking longer.
Sometimes it means verifying intermediate steps.
Sometimes it means searching a tree of possible reasoning trajectories.
And for software agents, it can mean compiling code, running tests, inspecting failures and trying again.
The interesting engineering question therefore isn't merely:
"Which model should I call?"
It is increasingly:
"What inference algorithm should I run around the model, and how much compute is this problem worth?"
That is a much more interesting question.
And perhaps the strangest consequence is that "scaling an LLM" may increasingly mean scaling the computation performed after the model has already finished training.
If you were building an LLM-powered coding agent today, where would you spend an extra 10x inference budget: longer chains of thought, parallel samples, verifier-guided search, or actually running more tools/tests?
*AI agents write code fast. They also silently remove logic, change behavior, and introduce bugs -- without telling you. You often find out in production.
git-lrc fixes this. It hooks into git commit and reviews every diff before it lands. 60-second setup. Completely free.*
Any feedback or contributors are welcome! It's online, source-available, and ready for anyone to use.
HexmosTech
/
git-lrc
Free, Micro AI Code Reviews That Run on Git Commit
| 🇩🇰 Dansk | 🇪🇸 Español | 🇮🇷 Farsi | 🇫🇮 Suomi | 🇯🇵 日本語 | 🇳🇴 Norsk | 🇵🇹 Português | 🇷🇺 Русский | 🇦🇱 Shqip | 🇨🇳 中文 | 🇮🇳 हिन्दी |
git-lrc
Free, Micro AI Code Reviews That Run on Commit
GenAI today is a race car without brakes. It accelerates fast -- you describe something, and large blocks of code appear instantly. But AI agents silently break things: they remove logic, relax constraints, introduce expensive cloud calls, leak credentials, and change behavior -- without telling you. You often find out in production.
git-lrc is your braking system. It hooks into git commit and runs an AI review on every diff before it lands. 60-second setup. Completely free.
In short, git-lrc helps Prevent Outages, Breaches, and Technical Debt Before They Happen
At a glance: 10 risk categories · 100+ failure patterns tracked · every commit…
Top comments (0)