Mayur Agnani

Posted on Nov 5 Originally published at theaicommit.com

How to Stop a Model Picking the Wrong Fact From a Long Prompt

#rag #ai

Engineer's view. This is a search ranking bug. Your site search returns the page that matches the query's words best, even when that page is out of date. A model reading a long prompt does the same thing. A passage that echoes your question can beat the true one. Every extra chunk you send adds rivals.

TL;DR. A model reading a long prompt favors the passage that best matches your question, true or not. Long prompts hurt mostly because they carry more look-alikes, so keep those out and ask questions only the right passage answers.

Explain Like I'm 5

You lost a blue umbrella, so you go to the lost-and-found desk. You say, "I lost a blue umbrella."
The clerk looks in the box and hands you the blue umbrella that best fits your words.

If the box holds one blue umbrella, you get yours. If it holds five, you may get someone else's.
A bigger box full of shoes does not confuse the clerk much. More blue umbrellas do.

So you do two things. You tell the clerk what only yours has: "the one with a broken handle." And
you ask them to set aside the umbrellas that clearly belong to other people.

The Problem

You have shipped this bug before, in a system with no model in it.

A customer types "refund window" into your help center. Search returns last year's refund page,
not the current one. Both pages match the words. The old one matches a little better, because it
says "refund window" twice. Nobody notices until a support ticket quotes a 14-day rule you
dropped in March.

A long prompt has the same bug inside it. Your agent's context holds tool results, retrieved
chunks and old turns. The model has to find the one passage that answers the question. It leans
toward the passage that best matches the question's wording, and it can be fooled by a passage
that only looks right.

Chroma tested 18 models this way in its report on what it calls context rot. Accuracy fell as
input grew, even on easy tasks. Adding one near-miss passage hurt, and four hurt more. On a memory
benchmark, every model scored much higher on a 300-token prompt than on the same task in 113,000
tokens.

I tried to break the length part on a current small model. Length alone did nothing to one easy
fact, out to 110,000 tokens. Four look-alikes cost 4 answers in 10, at 4,000 tokens and at 55,000.

The fix: keep look-alikes out of the prompt, ask questions that name what only the true passage
has, and test your own task with a length and look-alike sweep.

The Fix: Keep Look-Alikes Out and Ask What Only the Answer Has

Isn't the model just running out of room?

No. All of these failures happen far below the context limit, which is the most text a model can
accept. In Chroma's tests, accuracy fell fastest when the answer was worded unlike the question.
A strong match survives a long prompt. A weak match gets lost in it.

My test used gpt-4.1-mini at temperature 0, with this site's old articles as filler. The planted
fact was one sentence: the author's old manager taught them to "write the rollback plan before
writing the migration." With no rivals, it was found in every trial at every length.

Why does a look-alike win?

Then I added four look-alikes, which Chroma calls distractors. Each one echoed the question:
"the most useful thing a former boss taught me was to write the tests first." The question asked
for the most useful thing a former boss taught the author.

The model returned a look-alike in 4 of 10 trials, at both lengths. Length did not make it worse.
More text matters because more text brings in more look-alikes.

What actually fixes it?

Asking about what only the true passage had fixed it: 10 of 10, with the look-alikes still there.
Tagging each passage with its source helped only a little. Removing the look-alikes fixes it by
definition. Ten trials is a small sample, so treat these as directions, not rates.

What This Means for You

When this matters. You put more than one candidate answer into a prompt. That covers a RAG
pipeline that returns ten chunks, an agent that keeps every tool result, and a memory store that
holds old and new versions of a fact. It matters most when documents get revised, because an old
version is the perfect look-alike. It matters little for a single short document.

How it affects you. Your evals probably test the clean case. A test with one right document in
the prompt passes. In production, the prompt also holds the superseded page and a forwarded email
that disagrees. The model then quotes the wrong one some of the time, with full confidence. Adding
more context to "help" makes this worse, because it adds rivals.

What to do about it. First, with no setup: take one real prompt your app sent today and count
the passages that could answer its question. If more than one could, you have the bug.

Next, change how you assemble context. Keep only the current version of each document. Drop
sources outside the question's scope. Drop chunks that repeat each other. Then make the question
name what only the right passage has, such as a date, an ID or an author.

Then add the sweep from ## Implementing It to your eval suite. It tells you how many answers
look-alikes cost on your own task. Passing a clean test only shows the model can read.

If you need a refresher on why a model rereads all of this every turn, see
How Agent Context Gets Re-Read Every Turn.

Implementing It

The change.

Three parts change: the code that assembles context, the code that writes the question, and your
eval suite.

Part 1, the context assembler. This runs after retrieval and before the prompt is built. Filter
on metadata first, because it is exact and cheap. Then filter on similarity, which is fuzzy:

def assemble_context(chunks, scope, budget_tokens):
    newest = {}
    for c in chunks:
        if c["source"] not in scope:          # a forwarded email is not the policy
            continue
        if c["doc_id"] not in newest or c["version"] > newest[c["doc_id"]]["version"]:
            newest[c["doc_id"]] = c           # only the current version survives
    kept, used = [], 0
    for c in newest.values():
        if any(overlap(c["text"], k) > NEAR_DUP for k in kept):
            continue                          # a second copy adds a rival, not information
        cost = len(c["text"]) // 4
        if used + cost > budget_tokens:
            break
        kept.append(c["text"])
        used += cost
    return kept

This needs doc_id, version and source on every chunk in your index. If your index lacks them,
add them at ingest time. That is the real change, and the code above is the easy part. In
code_example.py, six retrieved chunks hold five refund rules. Only the current one survives.

Part 2, the question. Rewrite the user's question before it goes into the prompt. Add the
attribute that only the right passage carries:

# before
q = "What is the refund window?"
# after: name the scope the true passage has, and the look-alikes do not
q = f"What is the refund window in the {policy.name} policy, version {policy.version}?"

If your code cannot tell which policy the user means, ask the user. Do not let the model guess.

Part 3, the eval. Add one sweep per task. It plants the true passage in filler at several
lengths, with and without look-alikes, and calls your real model:

LENGTHS = [500, 4000, 16000, 64000]
for with_la in (False, True):
    for tokens in LENGTHS:
        prompt = build_prompt(needle, lookalikes if with_la else [], tokens, depth, seed=i)
        hits += answer_word in ask(prompt, question).lower()

Write the look-alikes by hand from real near-misses: the old version, a similar product, another
person's note. Random filler does not test this. Chroma's harness is MIT-licensed in
chroma-core/context-rot if you want its needle and distractor setup.

How you know it worked.

Read the sweep as two rows. The clean row should stay flat across lengths. If it falls, your
answers are worded too unlike your questions, so fix retrieval first. The look-alike row should
come close to the clean row once the assembler and the specific question are in place. Log one
production number too: chunks dropped per request by the assembler. If it is always zero, the
metadata filter is not running.

Run at least 20 trials per cell. With 10, a one-answer swing looks like a finding. The reason is
in How to Tell a Real Prompt Improvement From a Lucky Eval Run.

When Filtering Context Is the Wrong Tool

When the look-alikes are the point. A question like "how did the refund window change?" needs the
old version and the new one. Filtering out the old version makes the answer impossible. Label each
version with its date instead, and ask for a comparison.

When the right passage is not known ahead of time. Metadata filters need a scope. An open question
over a pile of email has none, so the filter can only drop exact copies.

When the near-duplicate threshold is too aggressive. Two chunks can share most words and differ in
the one number that matters, such as 14 days versus 30. A high threshold keeps both. A low one can
drop the true one. Check what the filter removed on a sample before you trust it.

My test used one small model, one fact and ten trials per cell. Larger models may resist
look-alikes better, and harder questions may show the length effect Chroma measured.

Three questions before you change your pipeline:

How many passages in today's prompts could answer the same question?
Does every chunk in your index carry a document ID, a version and a source?
Does your eval include the old version of the page, or only the right one?

Glossary

context rot — Chroma's name for model accuracy falling as more text is added to the prompt.
look-alikes — passages that resemble the answer or echo the question but are wrong.
context limit — the most text, in tokens, that a model can accept in one request.
distractors — the research term for look-alikes planted in a test prompt.
context assembler — the code that chooses which retrieved chunks go into the prompt.
sweep — one test run across several prompt lengths, with and without look-alikes.

DEV Community