Tree-of-Thoughts works, and it works by externalising the search: one call to propose the next steps, another call per candidate to score it, prune, repeat. A single Game-of-24 hand costs hundreds of API round-trips.
Algorithm of Thoughts (Sel et al., 2023) asks a blunter question. If the model can imitate a worked example, why not show it the whole search — including the branches that failed and the explicit act of backtracking — and let it run the search inside one generation?
The exemplar stops being a solution and becomes a search transcript.
That trade is real and it is not free. ToT spends calls. AoT spends context. Which one wins depends on numbers you can compute before spending a single token.
Live, every number computed in your browser: https://dev48.infy.uk/prompt/day62-algorithm-of-thoughts.html
Exact arithmetic first, or your ground truth is a lie
Game of 24 looks like it lives in integers. It does not.
The canonical case is the hand 3 3 8 8, whose only solution is 8/(3-8/3). In double precision that expression does not equal 24 — it equals 24 plus a few ulps. A strict equality test rejects a correct solution; a tolerance test invites false positives from the other side.
const rEq = (a, b) => a.n === b.n && a.d === b.d; // never ===, never epsilon
Fifteen lines of two-integer rationals with gcd reduction removes an entire class of "my solver disagrees with the internet" bugs. The page shows, per hand, how many correct solutions the float version throws away.
Write the exhaustive solver before you write the prompt
You cannot evaluate a search strategy without knowing the answer. Four cards is about 7,000 complete expression trees — milliseconds of work. Once you have it, every later claim becomes checkable: did the beam keep a branch that leads to 24, did the bounded depth-first walk miss a solution that exists, is this hand solvable at all.
Skip it and you are grading a model against your own guesses.
Canonicalize, or your solution count means nothing
Raw enumeration reports 3+8 and 8+3 as two findings. With four cards that inflates the count by an order of magnitude.
Define the equivalence you actually mean and implement it: flatten runs of the same commutative operator, sort children by canonical key, leave the non-commutative operators alone. Then state it on screen. This page counts solutions modulo commutativity and associativity of + and *, which means (a-b)+c and (a+c)-b are still counted separately even though they are numerically equal.
A count you cannot define is a number you cannot defend.
Price the baseline, do not estimate it
The reason to implement the Tree-of-Thoughts baseline is not to beat it. It is to price it. Wrap the standard expand-score-prune loop in counters that map onto real API traffic: one propose call per expanded state, one value call per candidate per vote.
Moving the beam width from 1 to 10 changes the counted call total by roughly an order of magnitude — and it does so because the counter is attached to the actual loop, not to a formula on a slide.
Keep the backtracks. They are the payload.
The instinct when serialising a trajectory is to tidy it up: drop the dead ends, present the winning path. That destroys the entire method. The failed branches and the explicit act of going back are what teach the model to search rather than to guess.
A hand-written tokenizer measures the real prompt bytes on both sides, and you can slide a context budget to find your own crossover point.
One caveat the page is strict about: it simulates the search, never the LLM. The paper's measured accuracy numbers are cited separately and never mixed into the live figures. Every step here is testable without an LLM, which is the whole point — you should know your ground truth long before you spend a token.
Top comments (0)