Chain-of-Note: write a relevance note per retrieved passage, answer only from the supportive ones, and honestly abstain

The failure that finally made me stop trusting naive RAG was watching it answer a question the documents couldn't answer — confidently, from a passage that just happened to share keywords. Retrieval ranks by similarity, and similarity isn't truth. A passage can overlap every word in the question and still not answer it (a distractor), and sometimes the answer is in none of the top-K passages at all. Chain-of-Note (Yu et al., 2023) fixes both by inserting a reading step: before answering, write a short note per passage — is it relevant, and what does it actually say? — then answer only from the supportive notes, and say "I don't know" when none support an answer. I built a demo where the retriever similarity, the note logic, the answer extraction and the abstain rule all run live. Here's how it works.

Naive RAG stuffs the blob and answers in one pass

The baseline is what everyone ships first: retrieve the top-K, paste them in, answer.

const docs = await retrieve(question, { k: 4 });   // ranked by similarity
const answer = await llm(`Passages:\n${docs.map((d,i)=>`[${i+1}] ${d.text}`).join("\n")}
Question: ${question}\nAnswer:`);

Fine when retrieval is clean. But the model inherits the retriever's noise and reads everything as one undivided blob, so it follows surface overlap straight into a wrong answer — or, on an unknown question, invents one.

The two ways it breaks

In my demo the answerable scenario asks "what year did Nimbus launch its first drone?" The real passage says the drone shipped in 2017, but a distractor — "held its first launch party in 2012" — scores higher on keyword overlap and steers naive RAG to 2012. The unknown scenario asks for the capital of a fictional country: no passage answers it, yet naive RAG borrows a city name from a nearby passage and hallucinates. Both come from trusting the ranking.

The note is a per-passage verdict

Chain-of-Note is a prompt, not a new pipeline. The instruction is one paragraph: for each passage write a note (relevant? what does it say?), answer only from the relevant ones, and reply "I don't know" if none support an answer. In the demo I made the relevance real — a passage is supportive only if it carries a candidate answer and the required concepts co-occur:

const notes = docs.map(d => {
  const answer   = extractAnswer(d.text, question);
  const relevant = answer != null &&
                   requiredConcepts(question).every(c => c.test(d.text));
  return { relevant, gist: summarize(d.text), answer };
});

So the founding-year passage — full of overlapping words but with no "launch" + "drone" co-occurring — gets marked not relevant and set aside, instead of silently becoming the answer. That's the difference from a plain "abstain if unsure" hint dropped over the blob: the hint has no per-passage verdict to lean on, so it can't cleanly tell a distractor from a real answer.

Filter to the supportive notes, then answer — or abstain

Keep only the notes marked relevant; everything else drops out of the answering context. Then the headline behavior:

function chainOfNote(question, docs) {
  const supportive = docs.map(d => noteFor(question, d)).filter(n => n.relevant);
  if (supportive.length === 0) return { answer: "I don't know", grounded: false };
  const best = supportive.sort((a,b) => b.sim - a.sim)[0];
  return { answer: best.answer, cite: best.i, grounded: true };
}

Zero supportive notes means the honest answer is "I don't know". Abstaining on an unanswerable question isn't a failure — a confident wrong answer is worse than an admitted gap, especially in high-stakes QA. In the demo, toggling the real passage off flips Chain-of-Note straight to an honest abstain while naive RAG keeps grabbing the nearest look-alike.

Where it earns its place

I reach for Chain-of-Note whenever retrieval is noisy or questions can fall outside the corpus. It composes: run it after reranking (fewer, cleaner passages to note) and alongside Self-RAG / CRAG, which decide whether to retrieve and re-retrieve while CoN judges what came back and knows when to abstain. It's prompt-only — no extra retriever, no fine-tune — which is exactly why it's cheap to add and hard to argue with.

Flip the Chain-of-Note switch and toggle passages to make the answer flip:
https://dev48v.infy.uk/prompt/day46-chain-of-note.html