DEV Community: Rijul Rajesh

Your RAG Searches by Meaning. But What About Exact Words? Meet BM25

Rijul Rajesh — Sat, 26 Sep 2026 20:44:55 +0000

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

When you work with RAG systems and come across scenarios where keyword matching is relevant, you might come across a term called BM25.

It sounds like some technical code word, but it is actually quite straightforward.

BM25 = Best Matching 25

It's essentially the name researchers landed on for this particular ranking function. The "25" doesn't have a deeper meaning.

So what exactly is BM25?

BM25 is a ranking formula used to score how relevant a document is to a search query.

Instead of simply asking:

"Does this word appear: yes/no?"

BM25 calculates a relevance score based on three main ideas.

The Three Ingredients of BM25

1. Term Frequency

This basically means: how many times does the word show up?

If you search for:

"dog training"

A document that mentions "dog" 5 times is probably more relevant to dogs than one that mentions it once.

More mentions can mean more relevance.

But there is an important nuance.

The benefit of additional mentions decreases as the word appears more often.

Going from 1 mention to 2 matters more than going from 20 mentions to 21.

BM25 has a built-in diminishing returns effect, so a document can't simply win by repeating the same word hundreds of times.

2. Inverse Document Frequency (IDF)

This basically means:

How rare or special is this word?

Common words like:

"the", "is", "and"

appear in almost every document.

So matching on them tells you very little.

Rare words like:

"gnocchi" or "arrhythmia"

appear in far fewer documents.

If both the query and a document contain a rare word, that can be a much stronger signal of relevance.

BM25 therefore gives more weight to rare terms and less weight to common terms.

3. Document Length Normalization

This basically asks:

Is this document relevant, or is it just really long?

A 5,000-word document will naturally contain more words and potentially more repetitions than a 50-word document.

That doesn't necessarily mean it is more relevant.

BM25 adjusts the score based on document length so that long documents don't get an unfair advantage simply because they contain more words.

It compares each document's length with the average document length in the collection and uses that information when calculating the score.

An Example

Let's say the query is:

"laptop battery life"

We have two documents.

Document A

A 200-word product review that says:

"battery" 4 times
"laptop" 3 times
"life" 2 times

Document B

A 3,000-word laptop manual that says:

"battery" 4 times
"laptop" several times
"life" isn't used in the relevant context

BM25 can rank Document A higher because it matches more of the query terms, while its shorter length also means those occurrences carry more weight relative to the document as a whole.

Document B's greater length can work against it through the length-normalization component.

Why Is BM25 Still Used Today?

Despite embeddings being the newer approach, BM25 is decades old and is still widely used.

It doesn't require model training or a GPU. It can run efficiently on CPUs and is particularly useful when exact terms matter.

For example:

Product IDs
Error codes
Names
Technical terms
Exact phrases

This is why BM25 remains an important part of search systems and is often combined with dense retrieval in hybrid search.

Wrapping Up

BM25 is a way of ranking documents based on how well their terms match a query.

It considers three main things:

Term frequency → How often does the term appear?

Inverse document frequency → How rare is the term?

Document length → Is the document unusually long?

Together, these produce a relevance score that helps the search system decide which documents should appear first.

Even with modern embedding-based retrieval, BM25 remains useful because semantic similarity and exact keyword matching solve different problems.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

Embeddings Turn Text Into Numbers. But There’s More Than One Way to Do It

Rijul Rajesh — Fri, 25 Sep 2026 18:58:09 +0000

If you have read about RAG, you will probably be familiar with the concept of embeddings.

We take a bunch of text and turn it into numbers.

That's it.

But there is one more layer here.

There are different ways to represent text for retrieval.

Two important approaches are sparse representations and dense embeddings.

Let's understand how they work and when each one is useful.

Sparse Representations

Imagine you have a giant dictionary containing every word in your language. Let's say it has 50,000 words.

A sparse representation for a sentence can assign a value to each of those 50,000 words based on how relevant that word is to the sentence.

Most sentences only contain a small number of those words.

So, out of 50,000 possible positions, only a small number will have meaningful values. The rest will be zero.

That's why it's called sparse: most of the vector is empty, with only a few non-zero values.

Here is a simpler example.

Let's say our vocabulary contains just these 10 words, in this order:

[cat, dog, run, fast, food, eat, blue, sky, car, drive]

Now take the sentence:

"The dog can run fast"

The sparse vector could represent it like this:

cat  dog  run  fast  food  eat  blue  sky  car  drive
0    1    1    1     0     0    0     0    0    0

It's essentially a representation of which words are present and how important they are.

Now take a second sentence:

"I like to eat food fast"

cat  dog  run  fast  food  eat  blue  sky  car  drive
0    0    0    1     1    1    0     0    0    0

You can see that both vectors are mostly zeros.

This kind of representation works well when the exact words in the text matter.

But that also creates some limitations.

Where Sparse Representations Can Struggle

Synonyms

"buy" and "purchase" have similar meanings, but a traditional word-based representation can treat them as completely different terms.

Paraphrasing

Two people can ask the same question using completely different words, making it harder to connect them based on literal word overlap.

Conversational queries

Consider:

"Why is my package taking so long?"

and:

"Shipping delays explained"

There is very little literal word overlap, even though the second document could contain exactly the information the user needs.

This is where dense embeddings become useful.

Dense Embeddings

With sparse representations, we are largely looking at which words appear.

With dense embeddings, we are more interested in the semantic meaning of the text.

A dense embedding represents a piece of text as a relatively small vector containing many numerical values.

Instead of having thousands of mostly-zero positions corresponding to vocabulary terms, a dense vector might contain hundreds or thousands of values, with most of them being non-zero.

A simple example:

Sentence 1:

"The dog ran quickly"

Sentence 2:

"The puppy sprinted"

Look closely. These two sentences don't share a single word.

"Dog" ≠ "puppy."

"Ran" ≠ "sprinted."

If we only looked for matching words, these sentences could appear completely different.

Dense embedding models are trained to capture semantic relationships between text.

So these two sentences can end up with embeddings that are relatively close to each other because they express a similar idea.

This is one of the reasons dense embeddings are useful for semantic search.

But dense embeddings have their own limitations.

Where Dense Embeddings Can Struggle

Dense embeddings are designed to capture broader semantic relationships, but they may not be as effective when exact terms are important.

For example:

Product codes such as SKU-4471B
Exact names
Error codes
Rare technical terms
Specific identifiers

A search for an exact product code may be better handled by a method that focuses on literal term matching rather than only semantic similarity.

Wrapping Up

Embeddings turn text into numbers, but there isn't just one way to do that.

Sparse representations preserve explicit lexical signals and are useful when exact terms matter.

Dense embeddings capture broader semantic relationships and are useful when the same idea can be expressed using different words.

Understanding this difference helps explain why RAG systems sometimes use more than one retrieval method instead of relying on dense embeddings alone.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Your RAG Finds the Documents. But Which Ones Should Reach the LLM?

Rijul Rajesh — Tue, 22 Sep 2026 19:12:31 +0000

You know that RAG fetches relevant chunks and gives them to the LLM.

So there is a retrieval step involved.

But there may be many chunks that are retrieved.

Which ones should actually be sent to the LLM?

And more importantly, which chunks should get priority?

This is where ranking comes in.

Let's see how it works.

Let's Start With the Core Mechanism

Suppose we have a set of chunks retrieved from our document collection.

At this point, we can use a reranker model to determine which of these chunks are most relevant to the query.

A common approach is to use a cross-encoder reranker.

A cross-encoder is an architecture, not one specific model. Models such as BGE Reranker, MS MARCO cross-encoders, and Cohere Rerank are examples of models that can be used for this job.

The reranker takes the query and one chunk at a time, pairs them together, and evaluates how relevant that chunk is to the query.

It produces a single number called a relevance score.

The same process is repeated for every retrieved chunk.

For example:

Query:
"How do I reset my router?"

Chunk 1:
"To restart your device, hold the power button..."

Score: 0.92

Chunk 2:
"Routers usually have several indicator lights..."

Score: 0.61

Chunk 3:
"The router supports both 2.4 GHz and 5 GHz..."

Score: 0.34

The chunks can then be sorted based on these scores, with the most relevant ones getting higher priority.

So, What Is the Difference Between Fetching and Ranking?

Both steps are looking for relevant information, but they work differently.

Fetching

During the initial retrieval step, the retriever converts the query into an embedding.

The chunks in the database have already been converted into embeddings.

The system then compares the query embedding with the chunk embeddings and measures their similarity.

This allows the retriever to quickly find potentially relevant chunks from a large collection.

These retrieved chunks are then passed to the reranker.

Ranking

Ranking happens after that initial retrieval.

The reranker model takes the query and each retrieved chunk together.

Something like:

[Query: "How do I reset my router?"]

[Chunk: "To restart your device, hold the power button..."]

The reranker looks at the query and the chunk together and produces a relevance score.

This allows it to make a more detailed judgment about how well that particular chunk matches the query.

So the basic flow is:

Query → Retriever finds candidate chunks → Reranker scores them → Sort by relevance → Send the top ones to the LLM

The retriever helps us find candidates quickly.

The reranker then helps us decide which candidates are actually the most relevant.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Why Does RAG Miss Information That's Clearly in the Document?

Rijul Rajesh — Mon, 21 Sep 2026 18:04:28 +0000

You have the information in your document. You know it's there.

But when you ask your RAG system about it, the model gives an answer that misses it completely.

Why?

In RAG, the model can only answer using the pieces of text that are retrieved.

And those pieces of text depend heavily on how the document was split into chunks.

If the information is split badly, it might never reach the model in the right form, even though it is clearly present in the original document.

Let's first go through a quick overview of how RAG works.

How RAG works

You split your documents into small pieces called chunks.
Each chunk is turned into a number vector called an embedding and stored.
When someone asks a question, the system finds chunks that are relevant to the question.
Those chunks are passed to the LLM, which uses them to generate an answer.

The LLM does not necessarily see your whole document. It usually sees only the chunks that are retrieved.

Chunking determines what information can be retrieved and shown to the model.

Why chunking matters

Retrieval sets the ceiling

If the information you need is not present in the retrieved chunks, the model cannot use it.

This means retrieval puts a ceiling on how accurate the answer can be. Even if the LLM is capable of answering the question, it cannot recover information that was never provided to it.

Chunks that are too big become less focused

Suppose you have a large chunk that covers five different topics.

When this chunk is converted into an embedding, its representation captures information about all of those topics.

Now imagine someone asks about just one specific topic.

The chunk may still be retrieved, but its representation is not focused only on that topic. The other topics can make the match less precise.

So, very large chunks can make retrieval less focused.

Chunks that are too small lose context

Now imagine that a chunk contains only a single sentence.

That sentence might depend on the sentences before it to make sense.

Without that surrounding context, the chunk may not contain enough information to understand what it is talking about.

So, making chunks too small can also hurt retrieval.

Bad cuts can break ideas apart

Suppose a paragraph contains one complete idea, but you split it right in the middle.

Now the two chunks contain only parts of that idea.

If the relevant information is split across chunks, retrieving only one of them may not give the LLM enough context to understand the full idea.

This is why simply cutting text into arbitrary pieces can cause problems.

Because of these issues, there are different ways to decide where one chunk should end and another should begin.

Let's look at some common strategies.

Common chunking strategies

Fixed-size chunking

This splits the document into chunks containing a fixed number of tokens.

For example, you might create a new chunk every 500 tokens.

It is simple, but it does not care about the meaning of the text. It can cut a sentence, paragraph, or idea in the middle.

Fixed-size chunking with overlap

This is similar to fixed-size chunking, but consecutive chunks share some text.

For example:

Chunk 1: A B C D E F
Chunk 2:       E F G H I J

The overlap helps preserve some context when an idea happens to cross a chunk boundary.

Recursive or structure-aware chunking

Instead of immediately cutting at an arbitrary token count, the system tries to preserve the document's structure.

It might first split by headings, then paragraphs, then sentences, using smaller units only when necessary.

This helps keep related content together.

Semantic chunking

Semantic chunking tries to identify where the topic or meaning changes.

Instead of asking only:

"Have we reached 500 tokens?"

it asks something closer to:

"Has the topic changed enough that this should become a new chunk?"

This can produce more meaningful chunks, but it generally requires additional processing.

Parent-child chunking

Here, retrieval happens using smaller child chunks, but when a relevant child is found, the system can provide the larger parent section to the LLM.

This gives retrieval a focused unit to search while still providing more surrounding context to the model.

Contextual chunking

Here, additional context is added to each chunk to explain where it came from or what it represents.

For example, a chunk might be accompanied by information about its document, section, or surrounding context.

This can help the retrieval system and the LLM interpret the chunk more accurately.

Wrapping up

There is no single way to split a document into chunks.

The goal is to create chunks that are focused enough to retrieve accurately, while containing enough context to preserve their meaning.

If the chunks are too large, retrieval can become less focused.

If they are too small, important context can be lost.

And if the boundaries break apart ideas, the LLM may never receive the information it needs in a usable form.

So before blaming the LLM for a bad RAG answer, it is worth looking at something much earlier in the pipeline:

How did you split the document in the first place?

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Your AI Knows How to Answer. But Who Teaches It What a Good Answer Is?

Rijul Rajesh — Sun, 20 Sep 2026 18:53:16 +0000

You use ChatGPT, Gemini, and all these AI bots.

Whatever you ask, they usually give you a good answer, often something we like to hear.

Ever wondered how they are tuned to our tastes?

We humans have a hand in that too. We teach these models to produce responses that people prefer.

There are different ways to teach a language model to produce responses that people prefer.

Two important approaches are DPO and RLHF.

Both use human preferences to guide a model, but they do it differently.

Let's understand how each one works.

DPO

DPO stands for Direct Preference Optimization.

It is a way to teach a language model which kinds of answers people prefer.

When a model learns to generate text, you usually want it to produce responses that are helpful, follow instructions, and avoid undesirable behavior.

But describing exactly what makes an answer "good" can be difficult.

Instead, we can ask humans to compare answers.

For example, given the same question, the model might produce two responses:

Answer A: Clear, helpful, and directly answers the question.

Answer B: Long and confusing, and doesn't really answer the question.

A human can simply say:

A is better than B.

DPO uses many of these preferences to adjust the model so that it becomes more likely to produce responses similar to the preferred answers.

How DPO works

You provide training examples containing:

A prompt
A chosen answer
A rejected answer

For example:

Prompt:
How do I reset my password?

Chosen:
Go to Settings → Security → Reset Password and follow the instructions.

Rejected:
Passwords can be changed in many different ways depending on the situation.

The model learns from many such comparisons.

A simple way to think about it is:

"When you see situations like this, produce something more like A and less like B."

After seeing enough comparisons, the model learns patterns in the preferences.

The problem with DPO

The model learns from the preference data it receives.

If the human judgments are noisy, inconsistent, or biased, those problems can make their way into the model.

For example, if humans consistently prefer overly long answers, the model may learn that longer answers are better even when they are not.

So DPO is only as good as the preference data used to train it.

Now let's look at another approach.

RLHF

RLHF stands for Reinforcement Learning from Human Feedback.

Like DPO, RLHF uses human preferences to teach a model which responses people prefer.

But the training process is different.

A classic RLHF pipeline has three major stages.

1. Start with a base model

First, you start with a pretrained language model.

It already knows how to generate text, but it may not reliably follow instructions.

So it is usually fine-tuned on examples of good conversations and instructions.

This gives you a model that can follow instructions reasonably well.

2. Train a reward model

Now humans compare different answers from the model.

For example:

Question:
Explain photosynthesis.

Answer A:
Photosynthesis is the process plants use to convert light energy into chemical energy.

Answer B:
Plants use sunlight, water, and carbon dioxide in a biological process.

Humans indicate which answer they prefer.

A separate model, called a reward model, is then trained on many of these preferences.

Its job is to predict how much a human would prefer a particular response.

Instead of simply saying:

A is better than B

the reward model can assign scores to responses.

For example:

Answer A → 0.82
Answer B → 0.54

The reward model has learned to act as a rough approximation of the human preferences represented in its training data.

3. Practice against the reward model

Now the language model generates new responses.

The reward model scores those responses.

The language model is then updated using reinforcement learning to produce responses that receive higher rewards.

You can think of the process like this:

Language model
      ↓
Generates an answer
      ↓
Reward model
      ↓
Gives a score
      ↓
Reinforcement learning
      ↓
Updates the language model

There is also a constraint that keeps the updated model from moving too far away from its original behavior. In classic RLHF, this is commonly implemented using a KL-divergence penalty.

Without such constraints, the model could find strange ways to increase its reward without actually producing better responses.

A simple analogy

Imagine a student learning to write essays.

First, a group of teachers evaluates a collection of essays and decides which ones are better.

Then, a grading assistant is trained to imitate those teachers' judgments.

Now the student can write a new essay, receive a score from the grading assistant, and use that feedback to improve.

The teachers don't need to grade every essay themselves.

The grading assistant provides feedback repeatedly while the student practices.

That is roughly the idea behind the reward-model stage of RLHF.

DPO vs RLHF

The biggest difference is what happens after humans provide their preferences.

With RLHF, those preferences are first used to train a separate reward model. The language model then uses reinforcement learning to optimize against that reward model.

With DPO, there is no separate reward-model-and-RL loop in the standard DPO procedure. The preference pairs are used directly to optimize the language model.

You can think of the difference like this:

RLHF

Human preferences
       ↓
Reward model
       ↓
Reinforcement learning
       ↓
Language model

DPO

Human preferences
       ↓
Direct preference optimization
       ↓
Language model

This makes DPO's training pipeline simpler than the classic RLHF setup.

The trade-off

RLHF gives you an explicit reward model and an RL optimization process, but that also makes the training pipeline more complicated.

There are more moving parts, and reinforcement learning introduces its own training challenges.

DPO removes the separate reward-model training and RL optimization loop, making the preference-training process simpler.

But DPO still depends heavily on the quality of the preference data.

In both approaches, there is a fundamental limitation:

The model can only learn the preferences that are represented in the feedback it receives.

If the human preferences are inconsistent, biased, or poorly defined, the resulting model can inherit those problems.

Conclusion

DPO and RLHF are two different ways of using human preferences to shape the behavior of a language model.

RLHF uses a reward model and reinforcement learning to turn human preferences into a training signal.

DPO uses preference comparisons directly to optimize the model.

The important difference is not that one uses human preferences and the other doesn't.

Both do.

The difference is how those preferences are turned into updates to the language model.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Vector Search Isn't Enough for Everything. Meet Hybrid Retrieval

Rijul Rajesh — Fri, 18 Sep 2026 18:51:23 +0000

When building a RAG system, one of the most common ways to retrieve information is through embeddings.

The documents are converted into vectors, and when a user asks a question, the question is also converted into a vector.

The system then looks for documents whose vectors are most similar to the question.

This works well when the user and the document are talking about the same idea in different words.

But vector search has some limitations.

For example, imagine a user asks:

What is the error code ERR-1042?

A vector search system is looking for semantic similarity. It may find documents that talk about errors, failures, or troubleshooting, but an exact match for ERR-1042 is not necessarily what semantic similarity is best at.

This becomes even more important for things like:

Product IDs
Error codes
Part numbers
Names
File names
Exact technical terms
Code snippets

There is also another problem.

Some questions are too complicated to answer with a single search.

For example:

Why did our payment service start failing after the latest deployment?

Finding the answer may require looking at deployment documentation, error logs, configuration changes, and troubleshooting guides.

A single similarity search may not be enough.

This brings us to hybrid retrieval.

Hybrid retrieval brings different ways of finding information together.

It can combine exact matching, semantic similarity, and, in some systems, reasoning-based retrieval.

What is Hybrid Retrieval?

Hybrid retrieval means searching in more than one way and then combining the results.

Think about a librarian.

You could describe what a book is about:

"I'm looking for a book about the history of computers."

The librarian can use that description to find something relevant.

But you could also give the librarian the exact title:

"I'm looking for The Innovators."

Or perhaps you give them the exact book ID.

The librarian can use different pieces of information to find the book.

Hybrid retrieval works in a similar way.

Instead of relying on just one retrieval method, it can use multiple methods to find relevant information.

There are three useful retrieval approaches to understand.

1. Keyword Search

Keyword search looks for specific words or terms in the documents.

For example:

ERR-1042

If a document contains ERR-1042, keyword search can find it directly.

This is particularly useful for things where the exact text matters, such as:

Error codes
Product numbers
IDs
Names
Technical terms
Code

Keyword search doesn't need to understand the meaning of the query. It looks for matching terms.

2. Meaning Search

This is the vector-based search that is commonly used in RAG.

The system converts the query and documents into embeddings and looks for vectors that are semantically similar.

For example, the user might ask:

How do I fix a service that keeps crashing?

A document might say:

Troubleshooting repeated application failures

The words are different, but the meaning is similar.

A semantic search can identify this relationship.

This makes vector search useful when the user doesn't use the exact words that appear in the documents.

3. Reasoning-Based Retrieval

Some questions are more complicated and may require more than one search.

For example:

Why did our payment service start failing after the latest deployment?

An AI system could break this into smaller questions:

1. What changed in the latest deployment?
2. What errors are associated with the payment service?
3. Do the deployment changes relate to those errors?

It can then search for information related to each step.

This is sometimes called agentic or multi-step retrieval, where the system decides what to search for next based on what it has already found.

This approach is more useful for complicated questions than for simple lookups.

Bringing Them Together

The interesting part is that these approaches don't have to compete with each other.

A hybrid retrieval system can use them together.

For example, it could:

User question
      ↓
Keyword search + Meaning search
      ↓
Combine the results
      ↓
Rank the results
      ↓
Smarter model re-ranks the best candidates
      ↓
Relevant context
      ↓
LLM

The keyword search can catch exact terms.

The vector search can catch related meanings.

The results from both searches can then be merged and ranked.

If a document appears highly relevant in both searches, the system can give it more importance.

For more complicated questions, the system can also perform additional searches based on what it has learned from the earlier results.

This gives the RAG system more than one way to find the information it needs.

Why Does This Matter?

The main idea is simple.

Different search methods are good at finding different kinds of information.

Keyword search is good when the exact words matter.

Vector search is good when the meaning matters.

Reasoning-based retrieval can help when the question itself requires multiple searches.

By combining them, a RAG system doesn't have to depend entirely on one retrieval method.

This is why hybrid retrieval can be useful when building RAG systems that need to handle a wide variety of questions.

Wrapping Up

Vector search is powerful, but it isn't perfect for every type of query.

Sometimes you need an exact match.

Sometimes you need to find something based on meaning.

And sometimes the question is complicated enough that you need to search multiple times.

Hybrid retrieval brings these different approaches together.

The goal isn't necessarily to replace vector search.

It is to give the retrieval system more than one way to find the right information.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Knowledge Poisoning in RAG: Attacking AI Through Its Knowledge Base

Rijul Rajesh — Thu, 17 Sep 2026 19:14:43 +0000

If you are familiar with RAG (Retrieval-Augmented Generation), you know that it allows an LLM to answer questions using information from your documents.

If you are new to RAG, you can check out this article first.

At a high level, RAG embeds your documents, retrieves the most relevant pieces for a question, and then passes them to the model as context.

Now, with any emerging technology, there will always be a security angle.

RAG is no exception.

The model relies on the information that gets retrieved.

So one potential weak point is the knowledge base itself.

If the documents contain bad or malicious information, the model may use that information when generating its answer.

This means an attacker may be able to influence the model's output simply by modifying the documents it retrieves, without directly attacking the model through techniques like jailbreaking.

This is known as knowledge poisoning.

There are different types of attacks.

Let's look at some of them.

Different Types of Attacks

1. Direct Injection

Here, we plant a false document that looks like a legitimate document.

When the document is retrieved, the model may treat the false information as real and use it in its answer.

2. Prompt Injection

Here, we hide an instruction for the model inside the data.

When that data is retrieved and included as context, the model may interpret the hidden instruction as something it should follow.

Depending on the instruction, this could cause the model to produce unintended or potentially harmful output.

3. Embedding Hijacking

Here, we add irrelevant text that is designed to be similar to the target query.

Because the retrieval system looks for similar content, the malicious text may be retrieved instead of the information that actually answers the question.

The model then receives the wrong context and may produce the wrong answer.

4. Gradual Drift

This one is more subtle.

Instead of making one obvious malicious change, an attacker makes small changes over time.

Each individual change may look harmless, but together they can gradually push the knowledge base toward incorrect information.

The problem is that there may be no single edit that clearly explains where things went wrong.

The incorrect information has accumulated gradually.

Understanding It With a Demo

Let's see one of these attacks in action.

I built a small project that demonstrates knowledge poisoning in a RAG system.

There are a few different attacks in the project, but I'll demonstrate one of them here.

You can clone the project here:

https://github.com/RijulTP/rag-demo

You can set your Gemini API key in the environment and try commands such as:

make demo-attack1

When you run it, you first see the original state before poisoning, where it correctly answers the questions.

This shows the state before the knowledge base is poisoned.

Now let's demonstrate the attack.

A file called poison_pricing.md is added to the knowledge base.

The file contains modified pricing information.

As a result, when the poisoned document is retrieved, the model uses the incorrect information and gives faulty pricing data.

Now let's look at the defense.

Two defenses are put in place.

The first is trusted sources.

The second is a pattern-based check.

As you can see, poison_pricing.md is not included in the trusted sources.

So the poisoned chunk is dropped before it reaches the model.

As a result, the poisoned information no longer alters the model's answer.

So this is one demo, there are few more in the repository, you can try it out.

Wrapping Up

RAG doesn't just introduce a retrieval problem. It also introduces a new place where an attacker can try to influence the model: the knowledge base.

Understanding these attack possibilities is important when building RAG systems.

Some attacks are easy to spot, while others can be much harder to detect.

Gradual drift is especially difficult to catch because there may be no single obvious change responsible for the final incorrect answer.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Autoregressive vs Diffusion: A Different Way AI Could Generate Text

Rijul Rajesh — Wed, 16 Sep 2026 18:13:26 +0000

When you are using AI tools, it is useful to know that there are different ways an AI model can generate text.

Today, most deployed language models use one main approach, but other approaches are emerging as well.

Here, I'll introduce you to two different approaches to generating text.

The first is autoregressive generation.

Autoregressive

The way autoregressive generation works is quite simple.

The model generates one token at a time, moving from left to right.

For example, if we ask the model:

The cat is sitting on the ___

The model might first predict:

mat

Then it uses everything generated so far to predict what comes next.

So the process looks something like:

The
   ↓
The cat
   ↓
The cat is
   ↓
The cat is sitting
   ↓
The cat is sitting on
   ↓
The cat is sitting on the mat

This is the approach used by most deployed language models today.

The important thing to notice is that the process is sequential.

The model needs to generate one part before it can move on to the next.

Once a token has been generated, the model doesn't normally go back and revise it as part of that same generation process.

This also limits parallelism because the next token depends on the tokens that came before it.

Now, let's look at an emerging approach that works quite differently.

Diffusion

Diffusion is more like sculpting.

Imagine you are making a statue. You start with a rough shape and then keep refining different parts until you get the final result.

Diffusion-based text generation follows a somewhat similar idea.

Instead of starting from the first token and generating one token after another, some diffusion language models can start with a sequence containing multiple masked or uncertain positions.

For example:

The [MASK] is [MASK] on the [MASK].

The model can work on multiple positions rather than being forced to complete them strictly from left to right.

For example, multiple positions could be refined in the same step:

The [MASK] is [MASK] on the [MASK].
      ↓           ↓           ↓
The cat is sleeping on the mat.

But the process doesn't necessarily stop there.

The model can revisit and refine parts of the sequence based on the surrounding context.

For example:

The cat is sleeping on the mat.
             ↓
          revisit
             ↓
The cat is sitting on the mat.

Here, the model could change an earlier prediction:

sleeping → sitting

The exact way this refinement works depends on the diffusion model, but the key idea is that the model can iteratively refine the sequence rather than only moving forward from left to right.

So the process is less like:

Token 1 → Token 2 → Token 3 → Token 4 → ...

and more like:

Multiple uncertain positions
          ↓
   Fill / refine them
          ↓
 Revisit uncertain parts
          ↓
      Refine again
          ↓
     Final sentence

This also creates opportunities for more parallel processing, since multiple positions can potentially be worked on at the same time.

Some models exploring this approach include LLaDA, Dream-7B, and Mercury.

Since this is still an emerging approach, performance varies across models and tasks, while autoregressive generation remains the dominant approach today.

The Potential Edge of Diffusion

So why are people interested in this approach?

One potential advantage is parallelism.

If multiple parts of the sequence can be processed at the same time, diffusion-based generation could potentially produce responses faster.

It could also potentially reduce the cost of generation if that parallelism can be used effectively.

Of course, this doesn't mean diffusion is automatically faster or cheaper in every situation. The actual performance depends on how the model and inference system are designed.

A Simple Way to Remember It

You can think of the two approaches like this:

Autoregressive: Write the sentence one word at a time, from left to right.

Diffusion: Rough out different parts of the sentence and keep refining them until the final version emerges.

That's the fundamental difference in how they approach text generation.

Wrapping Up

There are still many new developments happening in how AI generates text.

Autoregressive generation is the dominant approach today, but diffusion-based language models are exploring a different way of generating text.

And the interesting part is that this area is still evolving.

We are already seeing newer ideas emerge, such as soft masking, which takes the idea of masking and makes it more flexible.

So, even something as fundamental as how an AI writes a sentence is still being explored.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Learning to Build with LLMs the Framework-Free Way

Rijul Rajesh — Mon, 14 Sep 2026 19:01:31 +0000

Usually, when learning to build with LLMs, we start with a framework, follow a quickstart guide, and build a demo.

The problem is that once everything is set up, you may end up not fully understanding what is actually happening under the hood.

This can become a problem down the line when you eventually need to tune or modify things.

Because of this, I found a repository that is quite handy for learning:

AI Engineer Notebooks

You can find it here:

https://github.com/calmrocks/ai-engineer-notebooks

These are a set of clean, self-contained notebooks that promote more framework-free learning and let you focus on the actual concepts.

Checking the First Notebook

Frameworks may change from time to time, but the underlying concepts stay mostly the same.

So, if you want to try it out, you can go through each Colab notebook one by one, as mentioned in the README.

Here, I'll be going through one of the notebooks mentioned in the beginning.

It covers the fundamentals.

You can find the notebook here:

https://colab.research.google.com/github/calmrocks/ai-engineer-notebooks/blob/main/01-model-apis/00-prompting-basics.ipynb

You can first go through the setup.

You can create a free API key at Groq and add it using the key icon on the right.

The notebook then shows some of the basics, such as prompts and the difference they can make.

It also shows how specifying the output format can affect the response.

Going a Little Further

Now, let's look at another notebook that goes a bit further:

https://colab.research.google.com/github/calmrocks/ai-engineer-notebooks/blob/main/02-evals-basics/01-measuring-outputs.ipynb#scrollTo=K6anGbvMJwxm

This one is more about measurements.

Imagine you built a little tool.

Someone emails support, and your tool reads that email and outputs three things:

What the problem is
How urgent it is
What category it falls into, such as billing or a bug

That's it. That's the whole system we're testing.

Now, how do you know if it's actually good?

Most people just try a few emails, look at the output, think "Yeah, that looks right," and move on.

That's what we tend to do by default.

It feels fine until it isn't.

The notebook teaches you to do something slightly annoying but much more honest:

Write down the right answers first, before you even start improving anything.

Step 1: Golden Set

First, they create 6 fake emails and, next to each one, write down what the correct priority and category should be.

Think of it as an answer key.

For example:

"Someone got double-charged" → "urgent" and "billing"

"Someone is asking how to change their email" → "low" priority and "account" category

These are simple examples, but now the correct answers are written down instead of just living in your head.

Step 2: Run the Tests

Now you feed those same 6 emails into your AI tool and check each answer against what you wrote down.

Did it match?

Did it not match?

You add everything up and get a real number, rather than relying on a feeling.

In this case, the result is 83%.

That's your starting point.

Step 3: Change One Thing

Now, you change the prompt a little.

Step 4: Run the Same Test Again

Then you run the exact same test again.

Initially, the score was 83%.

Now, it has improved to 100%.

That's how you can systematically improve things.

Wrapping Up

These are some simple and practical ways of learning what goes into building with LLMs.

Following this kind of framework-free learning can give you a base-level understanding of the concepts, which can make it easier to handle more complex systems later.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

The 3 Scaling Laws of AI: From Training More to Thinking More

Rijul Rajesh — Sun, 13 Sep 2026 19:43:24 +0000

We see the pace of AI development speeding up. One way to view this is through the 3 scaling laws of AI.

It originally started with one scaling law, but as AI has advanced, it has grown into 3 different ways of scaling AI capabilities.

Pre-training Scaling

This is the scaling involved in the pre-training of a model.

To help understand it better, imagine a student preparing for a huge general knowledge exam.

If they read 10 books, they might do okay.

But if we scale that to 100 books, they will do much better.

So, the more they study, the more their score climbs.

Now, coming to a real example, GPT-2, which was released in 2019, had 1.5 billion parameters and was trained on a modest slice of internet text.

GPT-4-class models were trained on trillions of tokens.

The difference in knowledge between them is a direct result of pre-training scaling.

Post-training Scaling

Now imagine that same well-read student takes practice tests, and a tutor gives them feedback:

"That answer was too long."

"You should have shown your reasoning."

"That tone was rude."

The student's knowledge doesn't grow, but their ability to give the answer the grader actually wants improves enormously.

Coming to a real example, the raw pretrained GPT-3 model had issues with going off-topic.

But after reinforcement learning with human feedback, it became what we know as ChatGPT.

The underlying knowledge is largely the same, but its ability to follow instructions and present answers in an ideal way has improved.

Inference Time Scaling

Picture 2 students solving a tricky math problem.

One student instantly blurts out the first answer that comes to mind.

But the second student works it out on scratch paper, checks their steps, tries different approaches, notices their previous mistakes, corrects them, and eventually reaches a solution.

So, the second student gets the correct answer because they spent more time thinking, not because they studied more beforehand.

Now, this is what we see with Agentic AI. We can see different effort levels, and the system can determine how much thinking goes into generating a response.

So, a simple analogy to understand this is:

Pre-training makes the student smarter, post-training makes the student better at giving good answers, and inference-time scaling makes the student take their time on hard questions.

Wrapping Up

It's interesting to see how AI is evolving and how fast-paced this evolution is.

When these 3 scaling laws come together, it's even more impressive to see how AI capabilities are continuing to improve, with GPT-6 Astra coming up and the capabilities it is offering.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

RAG Without Vectors? Meet Vectorless RAG

Rijul Rajesh — Sat, 12 Sep 2026 20:45:01 +0000

If you are familiar with RAGs, you know its usual flow.

Split the document into chunks, convert each chunk into a vector embedding, store them in a vector database, and retrieve the chunk most similar to the query.

But this approach has some blind spots.

So, to address some of these issues, there is a concept called Vectorless RAG.

Problem with Traditional RAG

The RAG we see works well for many use cases.

But basic chunk-based RAG can break down on long, structured, or reasoning-heavy documents.

A few pain points are:

Fragmented context
- Sometimes during chunking, related information can be sliced across different chunks.
- This can separate a number from the sentence that explains it.
Similarity isn't relevance
- Vector search finds text that is semantically close to a query but not necessarily the text that actually answers it.
Opaque retrieval
- The similarity score doesn't tell you why a chunk was chosen. This can be a problem in fields like finance, where traceability matters.

What Does Vectorless RAG Do Differently?

Vectorless RAG can replace similarity-based retrieval with structure and reasoning. Rather than flattening a document into chunks and vectors, some approaches treat the document as a hierarchy, closer to a textbook with a table of contents than a bag of disconnected paragraphs.

The Building Blocks: What Makes It Up

The Document Tree. Instead of chopping a document into random chunks, it's mapped out like a table of contents, with chapters, sections, and subsections, the same way you'd flip through a book.
Section Summaries. Every branch of that tree gets a short summary describing what's inside it, so the system knows what each section is "about" without reading the whole thing.
The Navigator (LLM Reasoning). Instead of a math formula comparing vectors, an LLM reads the tree and reasons through it step by step: "Is the answer likely in Section 2 or Section 5?" Then it drills into the right branch.
Page/Section Pointers. Once the right section is found, the system knows exactly which page or part of the document it came from, so answers can be traced back to their exact source.

Issues with Vectorless RAG

The cost of infrastructure can shift toward the cost of inference.
Structure-dependent approaches can have difficulty with unstructured data.
It's best understood as a complementary strategy.

Wrapping Up

Traditional RAG works well for many use cases, but it can have limitations when dealing with long and structured documents.

Vectorless RAG takes a different approach by using the structure of the document to navigate and retrieve relevant information.

It doesn't necessarily mean that vector search needs to be completely replaced. Depending on the use case, both approaches can work together, with each being useful for different types of data and retrieval needs.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

You Thought Training an AI Model Was Enough. Then There's Post-Training

Rijul Rajesh — Thu, 10 Sep 2026 20:07:43 +0000

When you think of AI models being trained, it's not merely about having a large set of data and training the model on it.

When you are putting the model as the brain for a chatbot, there is additional training required so that it can respond appropriately to questions.

This process is called post-training.

Let me walk you through it and give you a basic idea of how this works.

You Have a Raw Pretrained Model: What Next?

You have successfully pre-trained an AI model, and the model is good at predicting the next token based on the internet data it has learned from.

Now, this is a knowledgeable model.

But it's not necessarily helpful.

When asked a question, it could refuse to answer, continue the text in an unexpected way, or not follow the instruction properly.

So what we have here is essentially a text predictor.

But we need to turn it into an assistant.

First, let's understand Supervised Fine-Tuning.

Supervised Fine-Tuning

Think of SFT as showing the model thousands of worked examples:

"Here's a question, and here's exactly how a helpful assistant should answer it."

Humans, or sometimes strong existing models, create these example responses, and the model is trained on them using a process similar to pretraining.

This is a relatively direct way of teaching the model how we want it to behave.

It teaches things like following instructions, producing the expected format, and responding in a helpful way.

The main limitation is that the model is limited by the quality of the responses it is given.

And when you are thinking at a large scale, creating perfect answers for every possible question doesn't always scale.

Now, in this technique, we are giving the model examples of the responses we want.

There is another technique where, instead of giving just one ideal answer, we give the model preferences between different responses.

For example:

Question
   ↓
Response A: Better
Response B: Worse

The model can then learn which type of response is preferred.

For this, we will explore DPO and PPO.

Proximal Policy Optimization (PPO)

PPO is a reinforcement learning technique that can be used for post-training language models.

A simplified view looks like this:

Train a separate reward model using examples of human preferences, so that it can predict how good a response is.
Use the reward model to provide a reward signal while training the language model.

The language model is then optimized to produce responses that receive higher rewards.

Direct Preference Optimization

DPO was designed as a simpler alternative to the traditional reward-model-based approach.
There is no separate reward model in the DPO training process.
Instead, we directly tune the language model using preference data, teaching it to prefer one response over another.

DPO is generally easier to implement and can be cheaper than a PPO-based approach because it avoids training and maintaining a separate reward model during the preference optimization process.

Wrapping Up

It's important to understand that simply training a model on a large amount of data won't necessarily make it ready for the real world.

We need to perform the appropriate post-training steps so that the model becomes more useful, follows instructions, and responds in the way we expect from an assistant.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.