Hello, I'm Shrijith Venkatramana, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.
Most LLM applications work like this:
prompt
|
v
LLM
|
v
answer
That architecture has a strange property.
The model can be extremely capable, yet once it commits to a bad intermediate step, everything downstream is built on top of it.
Ask a model to solve a difficult planning problem and you often get something that looks intelligent but fails because of one early mistake:
I think X...
-> therefore Y...
-> therefore Z...
-> final answer is wrong
Humans do something different on difficult problems.
We branch.
"What if I try this approach?"
"That looks bad. Back up."
"What happens if I choose the other interpretation?"
"This branch is promising. Explore it further."
That idea leads to Tree of Thoughts (ToT) and, more ambitiously, to Monte Carlo Tree Search (MCTS)-style reasoning for LLMs.
The interesting part is not really "making the LLM think longer."
It is changing the inference algorithm from:
generate -> continue -> continue -> continue
into:
propose -> evaluate -> branch -> explore -> backtrack -> commit
And that turns out to be a surprisingly important architectural shift.
1. From Chain of Thought to Search
The easiest way to understand Tree of Thoughts is to compare it with ordinary Chain of Thought.
Suppose you ask an LLM:
Use 4, 6, 7, 8 exactly once with +, -, *, / to make 24.
A conventional Chain-of-Thought system might generate:
8 - 6 = 2
7 * 2 = 14
14 + 4 = 18
The problem is obvious: once it walks down this path, it is stuck with its earlier choices.
A Tree-of-Thought system instead treats intermediate reasoning as search states.
Start
|
+------------+------------+
| | |
8 - 6 8 + 6 8 / 6
| | |
+----+----+ ... ...
| |
2 * 7 2 * 4
|
...
The LLM is no longer merely generating a sequence. It is generating candidates for the next state and evaluating which states deserve more exploration.
That distinction is subtle but fundamental.
Chain of Thought is approximately:
x_0 -> x_1 -> x_2 -> x_3 -> ... -> answer
Tree search is:
x_0
/ | \
x_1 x_2 x_3
/ \ / \
x_4 x_5 x_6 x_7
The 2023 Tree-of-Thoughts paper by Shunyu Yao and colleagues made this framing explicit: instead of reasoning token-by-token, the model can operate over coherent intermediate "thoughts", explore alternatives, evaluate them, and backtrack.
The famous result from their Game of 24 experiment illustrates why this matters. Their GPT-4 chain-of-thought baseline solved only about 4% of instances, while their Tree-of-Thoughts procedure reached 74%. The exact numbers are task-specific, but the conceptual point is much bigger: search can compensate for the brittleness of a single sampled reasoning trajectory. (arXiv)
This is very close to what classical AI has been doing for decades.
The twist is that the search tree is no longer a chessboard.
It is made of language.
2. Why AlphaGo Is the Useful Mental Model
If you've seen the 2016 AlphaGo story, you have already seen the basic architecture.
AlphaGo was developed by David Silver, Demis Hassabis and a large DeepMind team. It combined neural networks with Monte Carlo Tree Search to reason over possible Go positions.
And Go was an excellent demonstration of why search is useful.
The game has an enormous branching factor. A purely enumerative search is hopeless.
Instead, AlphaGo used learned models to answer two different questions:
Policy:
Which moves are worth exploring?
Value:
How good does this position look?
Then MCTS concentrated computation on promising parts of the tree.
The result was one of the most memorable moments in AI history: in March 2016, AlphaGo defeated Lee Sedol 4-1. One of the moves from that match, move 37 in game 2, became famous because it was so unlike what human professionals expected.
That is an important detail for LLM engineers.
The breakthrough was not:
"The neural network became infinitely smart."
It was closer to:
"The system learned how to spend computation on the right alternatives."
That distinction becomes extremely relevant for LLM inference.
A language model already contains a huge amount of latent knowledge.
The question becomes:
How should we allocate inference-time compute over possible reasoning paths?
That is where ToT and MCTS-like approaches live.
The original AlphaGo work explicitly described a system that combined policy/value networks with Monte Carlo tree search, and later AlphaZero generalized the idea even further through self-play. (DOI)
3. What a "Thought" Actually Is
A common mistake is to imagine a ToT node as one token.
It isn't.
A better abstraction is:
State = everything needed to continue reasoning
For example, in a math problem:
Node:
equations solved so far
assumptions
candidate result
remaining constraints
In a coding problem:
Node:
current hypothesis
proposed implementation
test results
unresolved bugs
In planning:
Node:
current world state
actions taken
constraints
remaining objective
So a tree might look like:
Problem
|
+-- Strategy A
| |
| +-- A1
| +-- A2
|
+-- Strategy B
|
+-- B1
+-- B2
The search algorithm now needs an evaluator.
Something like:
score(node) = "how promising is this state?"
That evaluator could itself be an LLM.
For example:
Generate candidate thought
|
v
Ask evaluator LLM:
"How likely is this path to reach a correct solution?"
|
v
Keep promising candidates
Notice the recursion.
The same model can play multiple roles:
LLM
| \
| +--> generator
|
+-----> evaluator
This is one of the most interesting properties of LLM reasoning systems.
You do not necessarily need a separately trained symbolic search heuristic.
The language model itself can become both the proposal mechanism and the heuristic.
This is essentially the idea behind RAP, "Reasoning via Planning," from Shibo Hao and colleagues: use the LLM as both a world model and reasoning agent, while MCTS determines where to search. Their experiments applied this to planning, mathematics and logical reasoning. (arXiv)
4. Where MCTS Changes the Game
Tree of Thoughts is a broad framework.
MCTS is a particular search strategy.
The canonical MCTS loop looks roughly like:
1. Select
2. Expand
3. Evaluate / simulate
4. Backpropagate
5. Repeat
Suppose the root has three possible reasoning strategies:
Root
/ | \
A B C
Initially, you don't know which is good.
MCTS might explore them:
A -> score 0.2
B -> score 0.7
C -> score 0.3
So future simulations spend more effort around B.
But there is an important problem.
If you always choose the currently best branch, you can get trapped by a bad estimate.
So MCTS explicitly balances:
exploration
vs
exploitation
A common UCT-style intuition can be written as:
score(child) =
average_value(child)
+ c * sqrt(ln(parent_visits) / child_visits)
The first term says:
"This branch has looked good."
The second says:
"But don't ignore branches you haven't investigated much."
For LLM reasoning, you can imagine an analogous quantity:
search_score =
estimated_solution_quality
+ exploration_bonus
- cost
The exact formula can vary considerably.
The architectural principle is more important:
Use inference-time compute to selectively investigate uncertainty.
That's a much deeper idea than simply increasing the output token budget.
5. A Developer's Version: Build the Search Loop
Suppose you wanted to implement a crude ToT system around an LLM API.
A minimal version might look like:
frontier = [initial_state]
best = None
for depth in range(max_depth):
candidates = []
for state in frontier:
thoughts = generate_thoughts(state, k=4)
for thought in thoughts:
child = apply(state, thought)
score = evaluate(child)
candidates.append((score, child))
candidates.sort(reverse=True)
frontier = [state for score, state in candidates[:beam_width]]
if frontier and is_solution(frontier[0]):
best = frontier[0]
break
That is already a search algorithm.
It is essentially beam search over semantic states.
You can immediately add things such as:
deduplication
state caching
parallel evaluation
early stopping
depth limits
backtracking
tool calls
external verifiers
And suddenly you have something much closer to a reasoning engine.
A practical implementation often looks like this:
+-------------------+
| Problem |
+---------+---------+
|
candidate generation
|
+-------------+-------------+
| | |
v v v
Node A Node B Node C
| | |
evaluate evaluate evaluate
| | |
+------+------+-------------+
|
search policy
|
v
expand best nodes
The particularly powerful part is the ability to insert deterministic verification.
For coding:
LLM proposes patch
|
v
compile + unit tests
|
v
score = test success
For mathematics:
LLM proposes proof
|
v
symbolic checker
|
v
verified / rejected
For SQL:
LLM proposes query
|
v
execute against database
|
v
result consistency
This is where LLM search becomes substantially more interesting than "ask the model to critique itself."
The evaluator does not necessarily have to be another language model.
It can be the environment.
6. The Economics: Why Search Is Expensive
There is a catch.
Search multiplies inference cost.
Suppose a normal request requires:
1 generation
x 3,000 output tokens
and your ToT system explores:
8 candidate thoughts
x 4 reasoning levels
x 2 evaluations
Even with aggressive pruning, you can easily turn one model call into dozens.
For a rough calculation:
baseline:
1 x 3,000 = 3,000 tokens
search:
8 branches
x 4 levels
x 1,000 tokens
= 32,000 tokens
That's roughly a 10x token-compute budget before accounting for evaluator calls.
If an API workload processes 100,000 requests/day:
baseline:
100k requests
search-heavy:
perhaps ~10x inference work
The economics can become the primary engineering constraint.
This suggests a useful design principle:
Don't search everywhere.
Use a routing policy.
For example:
easy problem
-> normal decoding
medium problem
-> self-consistency / small beam
hard problem
-> Tree of Thoughts
very hard + verifiable
-> MCTS + external evaluator
This is economically much more sensible.
There is also another effect: parallelism.
A normal decoding chain is inherently serial:
token 1
-> token 2
-> token 3
-> token 4
Tree search lets you evaluate siblings concurrently:
parent
/ | \
A B C
| | |
model model model
So although total compute increases, latency need not increase proportionally if your serving infrastructure has enough parallel capacity.
That turns the problem into a systems question:
quality gain
vs
tokens
vs
latency
vs
GPU utilization
For production systems, that tradeoff can matter more than the search algorithm itself.
7. The Real Frontier: Search Over Actions, Not Just Text
The most interesting version of this idea is not:
"Let's generate five different explanations."
It is:
"Let's search over possible actions in a partially understood world."
Consider an autonomous coding agent.
Its tree might look like:
Fix bug
|
+------------+------------+
| | |
inspect.py inspect tests inspect logs
|
hypothesis
|
modify code
/ \
patch A patch B
| |
tests tests
| |
pass fail
Now you have something that looks much more like classical planning.
The LLM proposes actions.
The environment changes state.
The environment supplies feedback.
The search algorithm decides where to spend further computation.
That is remarkably close to the architecture that made AlphaGo powerful:
learned model
+
search
+
environment feedback
There is an important conceptual shift here.
The future of capable LLM agents may not be primarily about producing longer chains of prose.
It may be about constructing increasingly effective search procedures over latent actions, tools, hypotheses, programs and world states.
Tree of Thoughts was an early, relatively simple expression of this idea. RAP pushed it toward explicit planning and MCTS. The natural next step is to combine it with real environments, verifiers, tools, memory and learned value functions. (arXiv)
Conclusion: The LLM Becomes the Heuristic, Not the Algorithm
The most useful mental model is probably this:
A plain LLM is a very powerful proposal distribution.
It can suggest:
what to think
what to try
what to inspect
what to change
what to do next
But proposal generation and decision-making are different problems.
Tree of Thoughts inserts a small search algorithm between them:
LLM
|
+--> propose alternatives
|
+--> evaluate states
|
+--> search
|
+--> backtrack
|
+--> commit
MCTS takes that idea further by making the allocation of inference compute itself adaptive.
Instead of spending equal compute on every possibility, the system asks:
Where is another unit of computation most valuable?
That question is arguably more important than simply asking:
"How do I make the model smarter?"
AlphaGo demonstrated this idea dramatically in games. ToT demonstrated that the same basic intuition could improve language reasoning. RAP showed how an LLM itself could participate as both a model of the world and a reasoning agent inside MCTS. (DOI)
And that leaves a particularly interesting engineering question:
For which LLM workloads is an extra 10x inference budget actually worth more than buying a model that is 10x larger?
That tradeoff is likely to become one of the central questions in inference-time scaling.
Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production stable while also shipping at high velocity.
I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.
Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.
Spend code review effort where business risk is highest — not spread evenly across every diff.
Try LiveReview on your codebase:
Top comments (1)
The main wall with tree search in production agents is the intermediate node evaluator. If the value function is just another prompt asking the model to score its own branch, it inherits the same hallucinated assumptions and keeps walking down dead ends. The setups where backtracking actually pays off pair branching with deterministic verifiers like AST parsers, compiler errors, or sandbox execution returns to prune bad branches before spending more compute.