DEV Community

Cover image for Your RAG Finds the Documents. But Which Ones Should Reach the LLM?
Rijul Rajesh
Rijul Rajesh

Posted on

Your RAG Finds the Documents. But Which Ones Should Reach the LLM?

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.


You know that RAG fetches relevant chunks and gives them to the LLM.

So there is a retrieval step involved.

But there may be many chunks that are retrieved.

Which ones should actually be sent to the LLM?

And more importantly, which chunks should get priority?

This is where ranking comes in.

Let's see how it works.


Let's Start With the Core Mechanism

Suppose we have a set of chunks retrieved from our document collection.

At this point, we can use a reranker model to determine which of these chunks are most relevant to the query.

A common approach is to use a cross-encoder reranker.

A cross-encoder is an architecture, not one specific model. Models such as BGE Reranker, MS MARCO cross-encoders, and Cohere Rerank are examples of models that can be used for this job.

The reranker takes the query and one chunk at a time, pairs them together, and evaluates how relevant that chunk is to the query.

It produces a single number called a relevance score.

The same process is repeated for every retrieved chunk.

For example:

Query:
"How do I reset my router?"

Chunk 1:
"To restart your device, hold the power button..."

Score: 0.92

Chunk 2:
"Routers usually have several indicator lights..."

Score: 0.61

Chunk 3:
"The router supports both 2.4 GHz and 5 GHz..."

Score: 0.34
Enter fullscreen mode Exit fullscreen mode

The chunks can then be sorted based on these scores, with the most relevant ones getting higher priority.


So, What Is the Difference Between Fetching and Ranking?

Both steps are looking for relevant information, but they work differently.

Fetching

During the initial retrieval step, the retriever converts the query into an embedding.

The chunks in the database have already been converted into embeddings.

The system then compares the query embedding with the chunk embeddings and measures their similarity.

This allows the retriever to quickly find potentially relevant chunks from a large collection.

These retrieved chunks are then passed to the reranker.

Ranking

Ranking happens after that initial retrieval.

The reranker model takes the query and each retrieved chunk together.

Something like:

[Query: "How do I reset my router?"]

[Chunk: "To restart your device, hold the power button..."]
Enter fullscreen mode Exit fullscreen mode

The reranker looks at the query and the chunk together and produces a relevance score.

This allows it to make a more detailed judgment about how well that particular chunk matches the query.

So the basic flow is:

Query → Retriever finds candidate chunks → Reranker scores them → Sort by relevance → Send the top ones to the LLM

The retriever helps us find candidates quickly.

The reranker then helps us decide which candidates are actually the most relevant.



Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

GitHub logo HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview

gitleaks.yml osv-scanner.yml govulncheck.yml semgrep.yml dependabot-enabled mcp-testcases.yml

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.

blast-radius-demo.mp4

LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.
















The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)

Here's the goal:

  • A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.
  • A 300-line UI change in one file, fully covered by…




Click below to try LiveReview with your codebase:

LiveReview Banner

Top comments (0)