Ranking in TypeScript RAG: Why Cosine Similarity Alone Fails

#typescript #node #rag #ai

Book: AI That Reads
The series: AI in TypeScript — 5 books, from your first LLM call to agents in production — all five here
My project: Hermes IDE | GitHub — an IDE for developers who ship with Claude Code and other AI coding tools
Me: xgabriel.com | GitHub

Ask a support assistant "how do I rotate an API key?" and watch
what comes back from a plain vector search. Ten chunks, all of them
genuinely about API keys. The changelog entry announcing key
rotation. The pricing page mentioning key limits. The security
overview describing why rotation matters.

The step-by-step rotation procedure is at rank eleven.

Nothing is broken. Cosine similarity measured topical relatedness and
returned the ten most topically related chunks. It was never asked
which chunk answers the question, because a single embedding
comparison cannot express that difference.

What the geometry actually gives you

An embedding compresses a chunk into one point. Cosine distance
between two points measures how close their overall meaning is.

That is a genuinely useful signal and a lossy one. Compressed away:
whether the chunk is instructional or descriptive, whether it is
current or superseded, whether it names a specific entity in the
query, and whether it contains a procedure at all.

A query is short and interrogative. A good answer chunk is longer and
declarative. They can be semantically close while being different
kinds of text — and "different kind of text" is exactly what
distinguishes an answer from a mention.

So the first stage should not try to be precise. It should be
recall-oriented: fetch generously, then spend effort deciding
which of those actually answer.

Over-fetch, then rerank

export type Candidate = {
  id: string;
  text: string;
  vectorScore: number;
  updatedAt: Date;
  docType: "guide" | "reference" | "changelog" | "marketing";
};

export async function retrieve(q: string, k = 8) {
  const candidates = await vectorSearch(q, { limit: k * 8 });
  const reranked = await rerank(q, candidates);
  return reranked.slice(0, k);
}

k * 8 is the whole trick at the first stage. The rotation procedure
sitting at rank eleven is now inside the candidate set, and a second
stage that reads text rather than comparing points can promote it.

The multiplier is a real dial. Too small and the right answer never
enters the pool. Too large and reranking gets expensive. Between five
and ten times k is a sane starting range; measure it against
labelled queries rather than picking once.

A cross-encoder is a different computation

Bi-encoders — what your vector search uses — embed the query and the
document separately and compare. The two never meet.

A cross-encoder feeds the pair together and scores the relationship
directly. It can attend to the fact that the query says "how do I"
and this chunk begins "To rotate a key:". That interaction is
precisely what the separate embeddings threw away.

export async function crossEncode(
  query: string,
  cands: Candidate[],
): Promise<Scored[]> {
  const scores = await reranker.rank({
    query,
    documents: cands.map((c) => c.text),
  });
  return cands.map((c, i) => ({ ...c, relevance: scores[i] }));
}

It is more expensive per pair, which is why it runs over sixty
candidates rather than the whole corpus. That is the shape of the
whole pattern: cheap and broad, then expensive and narrow.

If you have no reranker available, an LLM does the same job with a
constrained output:

const Rated = z.object({
  ratings: z.array(z.object({
    id: z.string(),
    answers: z.number().min(0).max(3),
  })),
});

async function llmRerank(query: string, cands: Candidate[]) {
  const res = await client.messages.create({
    model: "claude-sonnet-5",
    max_tokens: 1024,
    system:
      "Rate how directly each passage ANSWERS the question. " +
      "3 = contains the answer. 2 = partial. " +
      "1 = related topic only. 0 = irrelevant.",
    messages: [{ role: "user", content: render(query, cands) }],
  });
  return Rated.parse(JSON.parse(textOf(res.content)));
}

The rubric wording carries the weight. "Answers" rather than
"relevant" is the distinction the vector stage could not make, and
stating it explicitly is most of what you are buying.

Compose the rest as pure functions

Relevance is not the only thing you know. Recency and document type
are structured fields sitting right there on the row, and folding
them in as pure functions keeps them testable.

export type Signal = (c: Scored, q: string) => number;

export const recency: Signal = (c) => {
  const days = (Date.now() - c.updatedAt.getTime()) / 86_400_000;
  return Math.exp(-days / 365);          // ~0.37 at one year
};

export const authority: Signal = (c) =>
  ({ guide: 1, reference: 0.9, changelog: 0.5, marketing: 0.2 })[
    c.docType
  ];

export const exactTerm: Signal = (c, q) => {
  const terms = q.toLowerCase().match(/\b[a-z_]{4,}\b/g) ?? [];
  const hits = terms.filter((t) => c.text.toLowerCase().includes(t));
  return terms.length ? hits.length / terms.length : 0;
};

Each is an ordinary function of a candidate. Each can be unit tested
without a model, a database, or a network call — which matters,
because ranking logic is otherwise the least testable part of a RAG
system.

Combine with explicit weights:

const WEIGHTS = [
  [relevanceSignal, 0.6],
  [exactTerm, 0.2],
  [recency, 0.1],
  [authority, 0.1],
] as const;

export function finalScore(c: Scored, q: string): number {
  return WEIGHTS.reduce((sum, [f, w]) => sum + f(c, q) * w, 0);
}

Weights in one array, visible, adjustable, diffable. The alternative
— scoring spread across a long function with magic numbers inline —
is the version nobody dares change later.

exactTerm earns its place more often than people expect. Embeddings
are weak on rare literal tokens: an error code, a config key, a
version string. If the user typed ERR_KEY_ROTATION_FAILED, a chunk
containing that exact string is almost certainly the right chunk, and
the vector stage has no strong opinion about it.

Normalise before you mix

Scores from different sources are on different scales. Cosine
similarity sits roughly in a narrow band near the top; a reranker may
emit logits; your own signals are already in [0, 1].

Adding them raw means whichever has the widest range silently
dominates.

function normalise(xs: number[]): number[] {
  const lo = Math.min(...xs);
  const hi = Math.max(...xs);
  if (hi === lo) return xs.map(() => 0.5);
  return xs.map((x) => (x - lo) / (hi - lo));
}

Min-max within the candidate set, per query. Not global — the point
is relative ordering among these candidates, and a global scale makes
an easy query with ten good answers look the same as a hard one with
none.

Measure before you tune

Thirty real queries with the id of the chunk that answers each. That
is enough to see movement.

it("ranks the answering chunk in the top 3", async () => {
  let hits = 0;
  for (const { query, answerId } of GOLDEN) {
    const top = (await retrieve(query, 3)).map((c) => c.id);
    if (top.includes(answerId)) hits++;
  }
  expect(hits / GOLDEN.length).toBeGreaterThan(BASELINE);
});

Track it as a number that has to go up. Without it, weight tuning is
someone changing 0.6 to 0.7 because a query they personally tried
got better, and quietly making thirty others worse.

The framing that helps

Retrieval is two questions, not one. What is this about — cheap,
approximate, wide. Does this answer the question — expensive,
precise, narrow.

Cosine similarity answers the first well and the second not at all.
Most RAG systems that feel almost-right are missing the second stage
entirely.

If this was useful

AI That Reads covers
retrieval quality end to end — candidate generation, reranking, score
composition, and building the small labelled set that tells you
whether a change helped.