DEV Community

Cover image for Why Your AI Assistant Answers From the Wrong Document
Gabriel Anhaia
Gabriel Anhaia

Posted on

Why Your AI Assistant Answers From the Wrong Document

#typescript #node #rag #ai

A user asks how to rotate an API key. Your assistant answers confidently, with
a citation, and the answer describes the v1 rotation flow that was
deprecated eighteen months ago.

The retrieval worked. The chunk it found genuinely is about API key rotation.
It is just about a different version of your product, and nothing in the
pipeline had any way to know that mattered.

This is the most common quality failure in a RAG system, and it is almost never
an embedding problem.

1. Near-duplicate documents, one of them stale

Docs sites accumulate versions. v1/auth.md and v2/auth.md say similar
things in similar words, so they embed to nearly the same point. Cosine
similarity cannot prefer one — they are equally "about" the topic.

Whichever wins is decided by noise.

The fix is not better ranking. It is not putting stale content in the index at
all:

const rows = await db.query(
  `SELECT id, text FROM chunks
   WHERE deprecated_at IS NULL
     AND doc_version = $2
   ORDER BY embedding <=> $1
   LIMIT $3`,
  [queryVec, CURRENT_VERSION, k],
);
Enter fullscreen mode Exit fullscreen mode

If old versions must stay searchable, make it explicit rather than accidental:

type Scope = { version: string; includeDeprecated: boolean };
Enter fullscreen mode Exit fullscreen mode

A query without a scope should be a type error. That is the whole fix — the
bug exists because "which version" was never a parameter.

2. Metadata that exists but is not filtered

Most teams store docType, product, locale, tenantId on the chunk row
and then never use them at query time.

// the query that causes this whole post
ORDER BY embedding <=> $1 LIMIT 10
Enter fullscreen mode Exit fullscreen mode

The embedding of "how do I rotate a key" carries no signal about which product
line the user is on. That is not in the text. It is in your session.

export async function search(q: string, ctx: RequestCtx, k = 10) {
  return db.query(
    `SELECT id, text, doc_id FROM chunks
     WHERE tenant_id = $2
       AND product   = $3
       AND locale    = $4
       AND deprecated_at IS NULL
     ORDER BY embedding <=> $1
     LIMIT $5`,
    [await embed(q), ctx.tenantId, ctx.product, ctx.locale, k],
  );
}
Enter fullscreen mode Exit fullscreen mode

Filters first, similarity second. Everything you know for certain should
constrain the candidate set before you ask the vectors to guess.

3. The chunk lost the context that disambiguated it

A chunk reading "Click Rotate, then confirm. The old key stays valid for 24
hours." is genuinely ambiguous. Which product? Which version? The heading two
levels up said so, and chunking threw it away.

Carry the heading path into the embedded text:

const embedText = [
  doc.product,
  doc.version,
  ...chunk.headingPath,     // ["Authentication", "API keys", "Rotation"]
  chunk.text,
].join("\n");

await store(chunk.id, await embed(embedText), chunk.text);
Enter fullscreen mode Exit fullscreen mode

Embed the enriched text; store the original for display. The heading path costs
a few tokens and it is often the only thing distinguishing two otherwise
identical procedures.

Two near-identical chunks disambiguated only by the heading path that chunking discarded.

4. Nothing checks whether the retrieved chunk answers the question

Top-k always returns k results. There is no "no good match" outcome — ask about
something absent from your corpus and you still get ten chunks, and they will
be the ten least-bad ones.

The model then answers from them, because that is what you asked it to do.

Add a floor and a refusal path:

const MIN_RELEVANCE = 0.35;

const scored = await rerank(q, candidates);
const usable = scored.filter((c) => c.relevance >= MIN_RELEVANCE);

if (usable.length === 0) {
  return {
    kind: "no_answer" as const,
    message: "I could not find anything about that in the documentation.",
  };
}
Enter fullscreen mode Exit fullscreen mode

Absolute thresholds on raw cosine are fragile — the distribution shifts with
the embedding model. Threshold on a reranker score, which is calibrated to
"does this answer the question", or on the gap between the top result and the
median.

Then say so in the prompt, and make refusal a real option:

const system = `
Answer only from the provided sources. If they do not contain the answer,
say so plainly. Do not fill gaps with general knowledge — a wrong specific
answer is worse than "not documented".`.trim();
Enter fullscreen mode Exit fullscreen mode

5. The freshest source loses to the most verbose one

Longer chunks often retrieve better only because they contain more words that
overlap the query. A tidy three-line current answer loses to a rambling
outdated page.

Fold recency and authority in as explicit signals rather than hoping:

const score = (c: Scored) =>
  0.65 * c.relevance +
  0.15 * Math.exp(-ageDays(c) / 365) +
  0.10 * AUTHORITY[c.docType] +           // guide > reference > changelog
  0.10 * (c.isCanonical ? 1 : 0);
Enter fullscreen mode Exit fullscreen mode

Weights in one visible place. The alternative — hoping the embedding "knows"
that a 2024 changelog is less useful than the current guide — is not a
mechanism.

The diagnostic to run first

Before changing anything, find out which of the five you have. Take twenty real
questions where the answer was wrong, and record the id of the chunk that
should have been used.

for (const c of GOLDEN) {
  const got = await search(c.question, ctx, 20);
  const rank = got.findIndex((r) => r.id === c.expectedChunkId);
  console.log(c.question, rank === -1 ? "NOT RETRIEVED" : `rank ${rank + 1}`);
}
Enter fullscreen mode Exit fullscreen mode

Two outcomes, two different problems:

Not retrieved at all — a recall problem. Chunking, embedding, or filters
that are too narrow. No amount of reranking fixes it, because the right chunk
never enters the candidate set.

Retrieved at rank 8 — a ranking problem. The right chunk is there and
something else outranked it. Reranking and score composition fix this, and
they are cheap.

Teams routinely spend a week tuning ranking for what turns out to be a recall
failure. Twenty labelled questions tell you which one you have in an hour.

A recall failure versus a ranking failure, distinguished by where the correct chunk lands.

The one-line version

Filter on everything you already know, embed the heading path so chunks
disambiguate themselves, let the system return "not documented", and rank with
recency and authority as explicit terms.

Cosine similarity answers what is this about. Every one of these failures is
your pipeline expecting it to answer which one of these is right, which it
was never able to do.

If this was useful

AI That Reads covers retrieval quality
end to end — metadata filtering, chunk enrichment, reranking, refusal paths,
and the small labelled set that tells you which failure you are actually
looking at.

AI That Reads — RAG in TypeScript

The full series is at
xgabriel.com/ai-in-typescript.

Top comments (0)