A transformer's attention is soft: every token in the context gets some weight, and nothing is ever weighted to exactly zero. So a confident wrong hint dropped into a prompt ("I'm pretty sure it was 2008") or a stray off-topic sentence still tugs the answer — the model agrees with the opinion (sycophancy) or leans on an irrelevant detail (spurious correlation). System 2 Attention (S2A, Weston & Sukhbaatar 2023) fixes this without asking the model to try harder. I built an interactive demo where the relevance score, opinion detector, and both answers run live in JS, so you can watch the leak open and close. Here's the idea.
Why "just ignore the irrelevant text" fails
The tempting cheap fix is to leave the messy context in place and instruct the model to disregard opinions and filler:
llm(`Context: """${context}"""
Ignore any opinions or irrelevant sentences.
Question: ${question}`);
It helps a little, but it's unreliable — attention is soft, so those distractor tokens are still physically in the window and still receive weight. You cannot switch attention off with a sentence. The robust move is to physically remove the junk before the answering step, which means rebuilding the context.
The core move: regenerate the context
Instead of answering the given context, S2A first uses the LLM itself to rewrite it — producing a new version that keeps only the parts relevant to the question, with opinions and guesses stripped out:
const regen = await llm(`
Given the text below, extract the part of the context RELEVANT to
the question. Remove opinions, guesses, and any sentence not needed
to answer. Do NOT answer yet — only rewrite.
Text: """${context}"""
Question: ${question}
Return:
Relevant context: <cleaned facts only>
Question (unbiased): <question with any opinion stripped>`);
This is the key insight: you're not editing the question or adding an instruction, you're generating a fresh clean context. Models are very good at judging relevance when you ask them to extract rather than answer, so the regeneration is reliable even when the direct answer wasn't. It also separates the extracted context from the question, so an opinion smuggled into the question itself can't leak back in.
Throw the original away, then answer
The discipline that makes it work: from here on the original messy context no longer exists. Parse pass 1 into the clean context and neutral question, and the answering call only ever sees those:
const answer = await llm(`Context: """${cleanContext}"""
Question: ${cleanQuestion}`);
Same model, same one-shot answering — the only change is what it's allowed to look at. Because the injected opinion and irrelevant sentences aren't in the window, there's nothing left to produce sycophancy or a spurious pull, and the answer tracks the actual fact. In the demo, the raw answer chases whatever year the hint claims — flip it to 2001, 2015, 1977, it follows every time — while the S2A answer stays pinned to 1994.
Where it earns its extra call
The name comes from Kahneman: the raw forward pass is fast, automatic, distractible System 1 attention; S2A bolts on a slow, deliberate System 2 step that decides what deserves attention before answering. The honest trade-off is two LLM calls instead of one. Spend it when inputs are long, noisy, or opinion-laden — sycophancy-prone Q&A, RAG dumps where most passages are off-topic, math word-problems padded with irrelevant numbers. Think of it as re-ranking, but for the whole prompt rather than just retrieved docs. And note the ordering versus Chain-of-Thought: CoT reasons over the messy context, distractors included; S2A cleans the context before any reasoning — so they compose well. Skip it for short, clean prompts where there's nothing to strip.
The takeaway is one line: slow, deliberate attention beats fast, distractible attention — so regenerate a clean context, discard the original, then answer.
Flip the S2A switch and watch the first pass rewrite the context live:
https://dev48v.infy.uk/prompt/day44-system-2-attention.html
Top comments (0)