DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Buffer of Thoughts: keep a library of reusable thought-templates, not past answers — retrieve, instantiate, distil back

#llm #promptengineering #beginners #ai

Ask a language model a fresh word problem and plain chain-of-thought works it out from zero every single time, re-deriving the same method and re-making the same slips. Few-shot does a little better — it pastes a couple of look-alike question→answer pairs — but it transfers surface form, not method, so a problem that reads differently yet needs the same reasoning slips right past the match. Buffer of Thoughts (Yang et al., 2024) keeps a meta-buffer of reusable thought-templates distilled from problems it has already solved. I built a live demo that runs the real retrieval and buffer updates. Here's the idea.

The buffer holds methods, not answers

Each entry is a thought-template: a signature to retrieve on, a reusable reasoning skeleton (the method), and a usage count. Not a stored answer — a stored how.

{ name: "Combined Work Rate", keywords: ["pipe","fill","rate","hours"],
  skeleton: [ "each worker's RATE = 1 / time",
              "they act at once, so ADD the rates",
              "invert for the combined TIME = 1 / total" ], uses: 5 }
Enter fullscreen mode Exit fullscreen mode

Retrieve the best-matching template

Given a new problem, score every template by similarity and take the top one. Production BoT embeds the problem and each signature and uses cosine similarity; the demo uses a bag-of-words cosine — same idea, no API key. Below a similarity bar, there's simply no good match.

function retrieve(problem, buffer){
  return buffer.map(t => ({ t, sim: similarity(problem, t) }))
               .sort((x, y) => y.sim - x.sim)[0];   // best match
}
Enter fullscreen mode Exit fullscreen mode

Instantiate the skeleton on the specifics

The retrieved skeleton is the scaffold; the model — or, for exact arithmetic, trusted code — fills its slots with this problem's specifics and executes. The hard part (the method) is already decided, so there are far fewer missteps than free-form CoT re-deriving it cold.

const nums = problem.match(/\d+/g).map(Number);     // e.g. times [3, 6]
const rate = nums.reduce((s, t) => s + 1/t, 0);     // 1/3 + 1/6
const time = 1 / rate;                              // = 2 hours
Enter fullscreen mode Exit fullscreen mode

Distil back — the buffer grows

After solving, fold the result back in. If a template was a good match, reinforce it (bump its usage). If nothing cleared the bar, distil a new template from the trace and insert it. This dynamic update is what makes the buffer richer with every solve — the system literally gets better the more it's used.

if (best.sim >= THRESHOLD) best.t.uses += 1;        // reinforce the one we used
else buffer.push(await distil(problem, trace));      // learn one: 4 -> 5
Enter fullscreen mode Exit fullscreen mode

The prompt handed to the model

The loop assembles one prompt — the retrieved template plus the problem, with an explicit instruction to instantiate the template, not start cold. That single reframing is what turns a stored method back into a fresh answer.

const botPrompt = (tpl, problem) => `
You have a distilled thought-template for problems like this.
Instantiate it on the specifics below — do not reason from scratch.
Template — ${tpl.name}:
${tpl.skeleton.map((s, i) => `  ${i+1}. ${s}`).join("\n")}
Problem: ${problem}`;
Enter fullscreen mode Exit fullscreen mode

Why it beats CoT and few-shot

Three things fall out. The reasoning method is derived once and amortised over every future problem that shares its structure, instead of re-paid each time. The template captures the how, so it transfers across surface forms where exemplars break. And because the buffer updates after each solve, it's self-improving. It's prompt/orchestration-only — no fine-tune — and it slots on top of CoT, chain-of-table and tree-of-thoughts as the reusable-reasoning layer above them.

The takeaway I now carry: don't make the model re-derive a solution for every problem, and don't just paste look-alike answers — keep a library of reusable reasoning, retrieve the right skeleton, instantiate it, and distil it back.

Pick a problem and watch BoT retrieve a template, instantiate it, and distil one back:
https://dev48v.infy.uk/prompt/day49-buffer-of-thoughts.html

Top comments (0)