DEV Community: Ramya Perumal

AI Agents - Introduction to LLM and AI Terminologies

Ramya Perumal — Tue, 01 Sep 2026 18:38:49 +0000

LLM

LLM is a model, which means an equation.

Example:

y = mx + c

y = m1x^3 + m2x^2 + m3x + m4

A model is actually made up of weights. In any model, e.g., ChatGPT model or Gemini model, they would have used a large amount of input to train the model.

Input means a large amount of text/image data that is available on the internet. The input would have been fed into the Transformer architecture to get the output, which is the model.

Weights are floating-point numbers that represent the model's learned parameters. A 10B or 100B parameter model means how many parameters (weights) are present inside the model.

We cannot store a large-parameter model on our computer due to inadequate storage and computational power. Storage and CPU/GPU power decide what size of model can be run on a computer.

To run a model locally, we can use one of the following tools:

Llama.cpp
Ollama
LM Studio

Open Weight Model vs Open Source Model

An open-weight model shares its model weights. So, we can run them, fine-tune them, and host them on a local system.

Here, the training code, data, and full methodology are not shared.

Whereas, in an open-source model, the weights, training code, data, and sometimes the dataset are shared.

Why Do We Need to Use LLMs?

LLM is a next-word predictor.

Suppose we ask:

"Hi, how..."

The answer can be:

How are you?
How do you do?
How is your life?
etc.

These are possibilities.

Here, most of the time, the answer will be "How are you?" because if a word has more presence, it has a higher possibility of occurring.

Each possibility will have a score between 0 and 1.

We have 3 controlling parameters to control the output generated by the LLM.

1. Temperature

Usually set from 0–1. It controls the randomness of the model.

If the value is 0–0.3, which is low, it means generating the most likely words, i.e., facts or commonly occurring words.

If the value is high, the model will choose less likely words.

We use this high value in storytelling and creative writing.

2. Top K

Controls the number of possibilities.

For example, K = 3 means choosing only the 3 most likely possibilities.

Top K will be used along with Temperature.

For example, if we set Top K = 5 and Temperature = 0.5, the LLM will take possibilities from the selected Top K values based on the temperature.

Top K is used to limit the number of possibilities.

3. Top P

It is also called a sampling method.

The possibility scores are added up until they reach the defined value in Top P.

Top P is also used along with Temperature.

Tokens

Tokens are the process of splitting words into small pieces. A small piece may not be complete.

For example, a token can be split into "to" and "kens".

These are called tokens, and the process is called tokenization.

A tokenizer, such as those used by OpenAI and GPT, can be used.

We can specify the size of the token that we want to split.

Each token will be represented by a number, which in turn is converted into an embedding.

Context Window

It is a short-term memory.

The context window is the maximum number of tokens that a model can see at a given time.

Short context window and long context window are the types.

AI Agents - Data Structures in Python

Ramya Perumal — Sun, 30 Aug 2026 22:07:35 +0000

Data Structure means data stored in some structured format. How will we store strings, integers, and booleans in a data structure? Here we will see List, Dictionary, Tuple, and Set data structures in details.

Data structures can be treated as containers.

List

It is a dynamic array. An array is a collection of data that is stored consecutively. In Java or other languages, for a normal array, we have to specify the size to refer to the data. Every array has an index that points to the value of the array. The index starts from 0.

num[0] = 1
num[1] = 10
etc.

An array will maintain the data insertion order.

Dynamic array means allocating more space at runtime.

We can store heterogeneous data in the list, which means storing different data types or structures inside a single sequential collection.

E.g.
nums = [2, 3, 6, 8, 0, 1]

nums[0] - access the value at index 0
len(nums) - find the length of the list
nums[-1] - access the last item

nums.append(5) - add the new value at the end
nums.clear() - empty the list
nums.count(5) - count the number of occurrences of the particular value in the list
nums.reverse() - reverse the entire list
nums.sort() - sort the values in the list
nums.insert(0, 10) - insert the value 10 at index 0

These are the functions that can be used with a list.

Dictionary

Dictionary means a key-value pair.

Syntax:

dict_name = {key: value, key: value}

student1 = {"name": "some name", "age": 20}

Here, the key is immutable. The key can be either a string, boolean, integer, or tuple.

Generally, to search for a particular value, we need to search all the values one by one. But in a dictionary, by using the key, fetching time will be faster compared to other data structures. We can retrieve the value in O(1) time.

dict.get() - get the value of a key
dict.keys() - list the keys in the dictionary
dict.values() - list the values in the dictionary
dict.items() - list the pairs in the dictionary

These are the functions associated with a dictionary.

Tuple

An analogy for a tuple is taking a screenshot. Once we take the screenshot, we cannot change anything inside it. The same is applicable to a tuple. Once a tuple is created, we cannot change its values. We can delete the tuple by using the del keyword.

Syntax:

tuple_name = (value1, value2, value3)

A tuple is created mainly to preserve values as original and to maintain their integrity. So, it is immutable. Lists and dictionaries are mutable, where we can change the values in them.

Functions can always return a tuple.

Set

Set is a collection of unique elements. Here, duplication of elements is not allowed. To implement this, hashing methodology is implemented internally. It is the same as set operations in mathematics.

Syntax:

set_name = {element1, element2, element3}

Set does not maintain the user-specified order. Even if we try to add duplicates, it will not allow them.

Set will add the new element at any index. That is why it does not have an index.

Some of the methods in Set:

set_name1.union(set_name2) - union between two sets  
set_name1.intersection(set_name2)  - common elements between two sets 
set_name1.issubset(set_name2)  - check set1 is subset of set2 
set_name1.issuperset(set_name2) - check set1 is superset of set2
set_name1.pop() - remove random element
set_name1.remove(element) - remove specified elements

AI Agents - Python basics

Ramya Perumal — Tue, 25 Aug 2026 22:03:25 +0000

AI Agents - Python Basics

Why do we need to use Python to create AI agents?
Because the number of users in the community is bigger, and the number of libraries is more compared to other languages.
Why not use popular languages like Java or Go? Because every language is used for different purposes.

Python is called an interpreted language, which means it executes the code line by line and does not compile it first. Other languages like Java and C++ compile the code (i.e., convert it into binaries, 0s and 1s) and execute it from the compiled file.

Print Function

Prints the content in the console that is inside the function.

A function is simply a plan to do a task repeatedly. Here, we create a function when we do not want to repeatedly do the same work again and again.

Syntax:

def function_name(parameter_list):
return

E.g.

print("Hello World")

print - is the function_name, and "Hello World" is printed in the console.

print("Hello World", "Hi")

Identifier

name = "some name"

The value will be stored in RAM.

There are two types of memory in a computer.

RAM - Random Access Memory - Temporary memory used to access data faster for computational purposes.

ROM - Read Only Memory - Hard drive.

Initially, the OS will be stored in the ROM or hard drive. The OS will not run directly from the ROM. When the computer starts, the OS files will be moved to RAM because fetching data from RAM will be much faster. Since RAM is costly, we cannot store all the data in it.

"some name" will be stored somewhere in RAM. We cannot remember the memory address for the process all the time, so we assign a label called "name". In Python, there is no concept called a "Variable". The same label can be replaced with a different memory address. So, the "name" identifier will point to the address where the value is stored.

E.g.

age = 30
Flag = True
name = "Peter"

These are string/boolean/integer/None types of values that can be stored in the identifier.

In Python, everything will be treated as an object. Each object has attributes and functions. Each object may or may not have attributes and functions. To know whether a particular object has attributes and functions, use the method called 'dir'.

dir(age)

It will list down the attributes and functions associated with the object.

Acknowledgment

A special thanks to my mentor Syed Jafer for the continuous guidance and support throughout these sessions. I truly appreciate the effort taken to explain each concept clearly and make the learning process easier.

RAG - Hallucination Detection

Ramya Perumal — Fri, 21 Aug 2026 00:54:08 +0000

Hallucination

Hallucination means making an assumption or making up something when the LLM does not know the answer.

Hallucination in RAG

Example:

Suppose we have a PDF file that contains information about Python, but it does not have any details about Decorators.

Here, the document is split into chunks, and the chunks are stored in the vector database.

Suppose we ask the query:

"What is a decorator?"

The LLM should not give any response because the information about decorators is not available in the database.

But the LLM may give a response from its own knowledge, which may be correct, but it is not from the database.

This is called hallucination in RAG.

Methods to Measure Hallucination

Knowledge Groundedness

The response given by the LLM should be supported by the information given in the context.

If the response is not supported by the context, it is considered hallucinated.

Ways to Detect Hallucination

1. Embedding

Compare the embedding of the context with the embedding of the LLM response to check how closely they are related.

2. LettuceDetect

LettuceDetect is a BERT-based model used to find where hallucinations could occur at the character level.

It checks the response word by word to identify possible hallucinations.

3. LLM as a Judge

Give the response to another LLM and ask it to detect hallucinations.

The drawback here is that the LLM may hallucinate again, and it can also be expensive unless we use a local model.

4. RAGAS Framework

RAGAS is also called an evaluation framework. It provides different metrics that can be used to evaluate RAG systems and identify potential hallucinations.

Some of the important metrics are:

Faithfulness – Checks whether the results are backed by the provided context. This can be used to check hallucination.
Answer Relevancy – Suppose the context and answer are relevant. This metric checks how relevant the answer is to the query.
Context Precision – Checks how many of the retrieved documents are useful and relevant.
Context Recall – Checks whether the necessary documents have been retrieved for the query.
Answer Correctness – Checks the response against the ground truth.
Answer Similarity – Checks the semantic similarity between the expected answer and the generated answer.

RAGAS can be used in a CI/CD pipeline.

We can write unit test cases and compare the actual answer with the expected answer using the above metrics.

DeepEval can also be used to evaluate RAG systems and detect hallucinations.

RAG - Async Pipelines, MCP

Ramya Perumal — Tue, 18 Aug 2026 19:47:08 +0000

What is Synchronous?

Everything goes sequentially.

Example:

P1, P2, and P3 are the three processes.

Synchronous means P2 will start when P1 starts and finishes its job. P3 will start when P2 starts and finishes its job.

It is not required all the time.

During retrieval, in the case of hybrid search, first we will do vector search, then text search, and then any work related to semantic caching / context search.

The goal here is to give more relevant context to the LLM.

Suppose:

Vector search takes 10 seconds
Text search takes 5 seconds
Semantic caching / context search takes 2 seconds

In the case of a sequential process, we need to wait 17 seconds to get the context.

So, is there any way we can reduce the waiting time?

The answer is Async Pipelines.

Here, we will start 3 threads at the same time.

One thread will do vector search.
Another thread will do text search.
The third thread will do semantic caching / context search.

Their finishing times may be different.

At most, we have to wait only 10 seconds to get the context.

Through this async pipeline, we are reducing latency.

Context Switching

A single thread switches between multiple tasks when needed. It is also called parallel processing.

Multithreading

Each task will be allotted a separate thread. So, it can finish on its own time.

These threads are limited to the available CPU cores.

We can see the real difference in latency when we do this at the production level.

MCP - Model Context Protocol

If we ask the question "What happened today?", an LLM cannot answer.

If we ask the same question to ChatGPT or Gemini, it gives a response. How?

Because of tool calling, which means attaching some tools or functionality to the LLM. The result of the tool will be fed into the LLM.

The application developer will be responsible for writing the code for the tool/functionality.

Example

Suppose the idea of the application is to fetch weather details.

Every developer may write their own code to fetch the weather details. At the end, all users/developers will get more or less the same result.

This is not the right approach.

So, here comes the concept of MCP.

The responsible body/owner that provides the weather details will write the common tool or protocol (a set of rules).

The consumer/developer will use the common tool/protocol to fetch the result.

Here, the developer does not need to write their own code.

The common tool/protocol is called MCP.

MCP will have functions and their descriptions. The LLM will decide which function to call.

We can call the tool either by:

1. StdIO (Standard Input/Output)

MCP is within the same machine.

2. HTTP

We can also use the HTTP method to call the MCP, such as calling an API.

However, we are introducing latency when using HTTP.

What is the Relation Between MCP and RAG?

Create an MCP for a RAG system.

For example, suppose we have built a RAG system using documents that we gathered.

Since you have the database and RAG system, you are the only person who can invoke the RAG system directly.

Instead of keeping the RAG functionality restricted to your application, we can expose the RAG functionality through MCP.

We can segregate the functionality according to different responsibilities or divisions.

For example:

RAG system → Responsible for retrieving relevant information from the documents
MCP → Exposes the RAG functionality as a common tool
LLM/Application → Can invoke the MCP tool when it needs information

Now, anyone who has access to the MCP can use the RAG functionality without directly accessing the underlying database or writing their own retrieval code.

In this way, MCP acts as a common interface between the RAG system and different applications or LLMs.

The RAG system continues to handle document retrieval, while MCP provides a standardized way for other applications or agents to access that functionality.

RAG - Memory Systems

Ramya Perumal — Thu, 13 Aug 2026 18:08:59 +0000

We need memory to store the previous conversational history.

For example:

Previous question is:

User: File handling in Python

Assistant: Explain about file handling.

Next time, the user asks:

User: What are the modes in it?

The LLM should understand the context and then respond.

Below are the details that can be stored in the memory.

Past Conversations
User Preferences that we specify in the system prompts, e.g., JSON format
Past Decisions
Previous Tasks

Based on the details we are storing, we will choose between long-term or short-term memory.

Short-Term Memory

Redis, Valkey, Memcached, and Cosmos, which are cached databases, can be used for short-term memory to store the last few conversations or a summary.

We can set a general data invalidation rule to erase the content or use an LRU cache eviction policy, where the least recently used data will be erased from the database.

Long-Term Memory

Postgres, Pinecone, and MongoDB can be used for long-term memory to store long conversation histories.

How It Is Functioning

A summary of the entire conversation history will be stored in short-term memory to reduce latency whenever needed.

Episodic Memory

Episodic memory is a type of memory that stores specific events or experiences that happened in the past, usually together with information about what happened, when it happened, and the context surrounding it.

We can use either a short-term or long-term memory database depending on the use case. It is a kind of combination of short-term and long-term memory.

User:

I am planning a trip to Paris.

Agent:

How many days will you stay?

User:

5 days.

Later,

User:

Can you suggest an itinerary?

Episode 1

User wants to travel to Paris.

Trip duration: 5 days.

User previously mentioned:

Destination = Paris

Duration = 5 days

This information can be used to provide a more relevant response.

This helps the LLM understand what happened previously.

Semantic Memory

Semantic memory contains facts extracted from previous conversational history. Semantic memory is generally considered long-term memory.

Example

Paris is the capital of France.

The Louvre is a museum in Paris.

France uses the Euro.

That's general knowledge.

Sliding Window Memory

It is a short-term memory. Here, we store the last 3 to 4 conversations.

Redis or Valkey, like any cache memory, can be used.

Summarized Memory

Each and every time, the conversation, which includes the user query and response, will be summarized.

Even though token consumption during summarization is more, overall token consumption will be less.

It is a long-term memory.

Entity Fact Memory

This memory is used to store facts about a particular entity.

The difference between entity fact memory and semantic memory is that semantic memory is the broader category. Entity fact memory is one way of organizing and storing semantic knowledge about specific entities.

It can be used as either long-term or short-term memory.

Semantic Memory

Entity: Python

Python → is a programming language

Python → is used for AI

Python → supports object-oriented programming

Python → was created by Guido van Rossum

Entity Fact Memory

Entity: Alice

Alice → works at ABC Company

Alice → prefers Python

Alice → is working on Project X

It is not a good practice to store the entire conversation. We can make decisions based on the conversation and then store the relevant information. This is a good practice.

RAG - Multivector Retrievel, Multi Hop, Conversational RAG

Ramya Perumal — Wed, 12 Aug 2026 17:51:51 +0000

Multi-Vector Retrieval

In a typical RAG pipeline, every chunk is converted into an embedding and stored in the vector database.

In Multi-Vector Retrieval, instead of creating a single embedding for a chunk, multiple embeddings are created to represent different aspects of the original chunk.

Each embedding captures a different perspective of the same content. The metadata of each embedding contains a reference to the original chunk.

When a user query is received, the retriever searches across all these embeddings. Since each embedding represents a different aspect of the content, the chances of retrieving more relevant information are higher.

This approach generally provides better retrieval performance.

However, it is a costly approach because:

Multiple embeddings are generated for every chunk.
More storage is required in the vector database.
More tokens are consumed during embedding generation.

If no other optimization technique is able to improve the RAG performance, Multi-Vector Retrieval can be considered as a final optimization step.

Multi-Hop

Multi-Hop Retrieval is used when the answer cannot be obtained from a single piece of context. Instead, the LLM has to retrieve multiple related contexts and connect them to generate the final answer.

Example

Suppose the knowledge base contains the following information:

Biryani contains spices.
The spices used in biryani are cardamom, cinnamon, cloves, etc.

Now suppose the user asks:

"What spices need to be added to biryani?"

The LLM first retrieves the information:

"Biryani contains spices."

It then hops to the next related context:

"The spices used in biryani are cardamom, cinnamon, cloves, etc."

Finally, it combines both pieces of information to generate the complete answer.

In this process, the LLM hops from one retrieved context to another until it gathers enough information to answer the user's query.

Each retrieved context should have a meaningful relationship with the next one so that the LLM can follow the chain of information and produce the correct response.

Conversational RAG

A user may not restrict themselves to asking only one query. They may ask a sequence of queries. Here, we are going to look at how to build Conversational RAG.

Traditional Flow

User Query
Retriever
Related Documents
LLM
Answer

A user may ask questions that are indirect or related to the previous question. So, we need to implement Query Transformation and Expansion to get the related queries and their responses.

Therefore, we need to have memory to store the previous conversation history.

Conversational RAG will be:

Conversation History + Current Question
Intent
Query Transformation and Expansion
Retriever
Relevant Context
LLM
Context-Aware Answer

Here, we are going to see how we can store conversation history in memory.

We can use persistent memory such as PostgreSQL or SQLite, or short-term memory using Redis, Valkey, Memcached, etc. We can also use a pickle file to store the history.

If we are using a file, make sure that the application we are using is single-threaded.

To store the history, we can use the session ID as the primary key.

Always store the entire history in long-term memory. However, we cannot store the entire history in short-term memory.

The purpose of storing history in short-term memory is to reduce latency.

To solve this problem, we can store the entire history in long-term memory and summarize the conversation history and store the summary in short-term memory whenever needed.

RAG - Parent Retriever

Ramya Perumal — Wed, 05 Aug 2026 01:52:04 +0000

RAG Pipeline Optimization Techniques

The following techniques are commonly used to optimize a RAG pipeline:

Semantic Caching
Query Transformation and Expansion
Context Compression
Parent Retrieval
Multi-Vector Retrieval

These techniques are used to improve the quality of the responses generated by the LLM.

Parent Retrieval

Today, we are going to focus on Parent Retrieval.

A chunk is a small portion of a document or paragraph, typically consisting of 500–1000 words. When these chunks are converted into embeddings and stored in a vector database, semantically related chunks are positioned close to one another.

When a user submits a query, the system retrieves the chunks that are closest to the query. However, not all the retrieved chunks may be the most relevant. Sometimes, we may miss other chunks that provide better context for the user's query.

Example

Suppose we have the following paragraphs:

Paragraph 1 (P1)

Each line is stored as an individual chunk:

P1C1
P1C2
P1C3
P1C4

Paragraph 2 (P2)

Each line is also stored as individual chunks:

P2C1
P2C2
P2C3

Suppose the expected answer to the user query is P1C2.

The vector database retrieves:

P1C2
P2C3

Although P1C2 is correctly retrieved, the related chunks P1C1, P1C3, and P1C4 may provide much better context than P2C3. Since these chunks are not retrieved, we may lose important context that could improve the final LLM response.

If we store the document paragraph-wise instead of using smaller chunks, we may retrieve unnecessary context, which increases token consumption.

Solution – Parent Retrieval

Suppose Paragraph 1 is divided into four chunks:

P1C1
P1C2
P1C3
P1C4

Whenever a specific chunk is retrieved from the vector database, it also contains a reference to its parent paragraph, which is stored in the chunk's metadata.

For example, if P1C2 is retrieved, the metadata also contains a reference to Paragraph 1.

Using this reference, the retriever can also fetch the remaining chunks belonging to the same parent paragraph:

P1C1
P1C3
P1C4

This ensures that we do not miss important context that is closely related to the user's query.

Example

Suppose the user query is:

"Create a dictionary"

The retrieved chunks are:

P1C2
P2C2
P2C3

Instead of sending only these chunks to the LLM, Parent Retrieval also includes the remaining chunks from the same parent paragraph:

P1C1
P1C3
P1C4

Providing this additional context enables the LLM to generate a more accurate response.

When Should Parent Retrieval Be Used?

Parent Retrieval is useful when we feel that important context is missing from the retrieved chunks and we want to include additional context that is closely related to the query.

The parent paragraph is stored similarly to other chunks. However, it is not used directly for vector search. Instead, its reference is stored in each chunk's metadata, allowing the retriever to fetch the parent context whenever required.

RAG - Context Compression

Ramya Perumal — Thu, 30 Jul 2026 01:58:52 +0000

In a RAG pipeline, documents are first split into chunks, converted into embeddings, and stored in a vector database.

When a user query arrives, it is converted into an embedding and used to retrieve relevant results from the vector database. Although the retrieved results may be relevant, they may not always be directly related to the user's query.

Example

User Query

"FastAPI dependency injection"

The retrieved results may include:

"Use FastAPI in deployment"
"How to dockerize a FastAPI application"

These results may be semantically related to FastAPI, but they are not necessarily the most accurate results for the user's query. This depends on how the documents were embedded and stored in the vector database.

To improve the quality of the retrieved context, the fetched results can be compressed into one or two chunks before they are passed to the augmentation phase. This reduces the number of tokens used during augmentation and acts as an optimization technique.

From the above example, the chunks retrieved from the vector database may not be completely relevant to the user query. Therefore, we compress the retrieved chunks without changing their original context and create one or two chunks that are more relevant to the query.

Context compression is not a mandatory step in every RAG pipeline. It is generally adopted based on trial and error and depends on the application's requirements.

This compression technique is mainly useful when:

There are token constraints.
Higher-quality retrieval results are expected.

Context Compression Techniques

There are different approaches for context compression:

LLM-based Compression
Embedding-based Compression
Keyword-based Compression

1. LLM-based Compression

In this approach, the relevant chunks retrieved from the vector database are given to an LLM for compression.

For example, suppose four chunks are retrieved from the vector database. Instead of sending all four chunks to the main LLM, they are first passed to another LLM whose responsibility is only to compress them into one or two meaningful chunks.

Although using an LLM for compression may appear expensive, a locally deployed LLM or a low-cost LLM is typically used for this task.

The purpose of this LLM is not to generate the final response. Its responsibility is only to merge and compress the retrieved chunks while preserving their context.

The compressed chunks are then sent to a more powerful or application-specific LLM to generate the final response.

This creates a two-stage pipeline, which is more token-efficient while still producing high-quality output.

2. Embedding-based Compression

In embedding-based compression, the embeddings of the top retrieved chunks are compared with the embedding of the user query using cosine similarity.

The chunks that are most relevant to the user query are selected and used as the compressed context.

3. Keyword-based Compression

Keyword-based compression uses techniques such as TF-IDF or the BM25 algorithm.

The keywords present in the user query are compared with the retrieved chunks.

The chunks that have the highest keyword relevance are selected and used as the compressed context.

Combining Compression Techniques

A single application is not limited to using only one compression technique.

Depending on the application requirements, two or more context compression techniques can be combined to improve retrieval quality while reducing token usage.

RAG - Query Transformation and Expansion

Ramya Perumal — Wed, 29 Jul 2026 17:27:33 +0000

Query Transformation

When a user asks a blended or incomplete query, such as:

"How do I deploy it?"

the LLM uses the background context it already has to transform the query into a more meaningful one.

For example:

"How do I deploy a FastAPI application?"

By transforming the query into a more specific one, the system is able to retrieve more relevant documents from the vector database.

For query transformation to work effectively, the LLM should have some background information about the conversation.

Query Expansion

When a user query is converted into an embedding, the point obtained in the vector database may not be close to the most relevant documents.

By expanding the query into different variations, we can retrieve more relevant documents.

Example

Original User Query

"How do I deploy a FastAPI application?"

Expanded Queries

Deploy FastAPI using Docker
FastAPI deployment guide
FastAPI deployment on AWS
FastAPI with Gunicorn
FastAPI with Uvicorn

These are different variations of the original user query.

How Does Query Expansion Work?

For each variation of the user query, the system retrieves a set of relevant contexts from the vector database.

During the augmentation phase, the retrieved contexts, along with the original user query, are sent to the LLM so that it can generate more accurate results.

Query Expansion and Query Transformation

Both query expansion and query transformation are performed by the LLM.

Query Transformation rewrites an incomplete or ambiguous query into a more meaningful query.
Query Expansion generates multiple variations of the user query to improve document retrieval.

Alternative Approach

Instead of calling the LLM for query expansion, we can use a rule-based approach by storing related queries for a user query as a cluster in the vector database.

When a user submits a query, the system retrieves the related queries from the cluster and uses them to improve document retrieval.

RAG - Semantic Caching

Ramya Perumal — Sat, 18 Jul 2026 17:12:20 +0000

When a user submits a query, the query is converted into an embedding and searched against the vector database to retrieve the relevant documents.

But what happens if the user asks the same or a very similar query again?

This is where semantic caching comes into the picture.

Instead of searching the vector database again, the system stores the previous search result in a cache. A cache is a temporary storage where frequently accessed or recently queried results are stored. When the user asks the same or a semantically similar query again, the system can retrieve the result directly from the cache instead of querying the vector database again.

Benefits

Saves retrieval time
Reduces token consumption
Reduces the number of calls to the vector database
Reduces the number of calls to the LLM

How Do We Store Results in the Cache?

We can use Redis or Valkey for semantic caching.

These are in-memory databases, which means they store data in RAM instead of disk. Since data is stored in memory, retrieval is much faster compared to traditional databases.

Typically, we store:

User query
Related answer
Metadata
Embeddings

Example

Suppose a user asks:

"What is today's gold price?"

The query and its corresponding answer are stored in Redis.

Later, another user asks:

"Gold price today?"

Although both queries have the same meaning, Redis cannot directly retrieve the previous answer because it expects the key to match exactly.

This is one of the limitations of using Redis as a simple key-value store.

How Can We Solve This?

One approach is:

Retrieve all the keys stored in Redis (for example, using KEYS *).
Generate or retrieve the embedding for each stored query.
Convert the current user query into an embedding.
Compare the current query embedding with the stored query embeddings using cosine similarity.
If the similarity score is above a predefined threshold, retrieve the corresponding answer from Redis.

This allows semantically similar queries to reuse cached results even when the text is different.

Ways to Implement Semantic Caching

Semantic caching can be implemented in two ways:

Using frameworks such as LangChain
Using in-memory databases such as Redis, Valkey, or other similar databases

Cache Invalidation

One of the most important aspects of semantic caching is cache invalidation, which determines how long cached data should remain valid before it is automatically removed or refreshed.

For example, suppose a user asks:

"What is today's gold price?"

The answer should only be valid for a limited period. If the application returns yesterday's gold price, the information becomes incorrect.

There is no single solution for cache invalidation. The appropriate strategy depends on the application and the type of data being cached.

Different scenarios need to be considered before deciding when cached data should expire.

When Should In-Memory Databases Be Used?

In-memory databases are well suited for:

Temporary queries
Frequently asked questions
Data that is accessed repeatedly

By understanding the meaning of the query, we can define guardrails to determine which queries should be cached and when the cache should be invalidated.

The main objective is to optimize the RAG pipeline by reducing unnecessary calls to both the vector database and the LLM.

Although it is not possible to eliminate duplicate requests completely, semantic caching can significantly reduce them.

Important Consideration

We should not store every query in an in-memory database.

Only queries that are valuable for caching should be stored because RAM has limited storage capacity. Therefore, an effective caching strategy should carefully decide which queries are worth storing and for how long.

RAG - Meta Filtering and Reranking

Ramya Perumal — Sun, 12 Jul 2026 21:43:55 +0000

Generally, when a user asks a query, the system searches for the relevant chunks stored in the vector database using cosine similarity. The better we can filter the data, the smaller the search space becomes, resulting in faster and more efficient retrieval.

Suppose we have a book with 10 chapters. If we want to search for a particular topic, all the points in the vector database are compared with the user query, and only the closest points are retrieved. This process is called KNN (K-Nearest Neighbors).

Another algorithm is ANN (Approximate Nearest Neighbors). Instead of checking all the points in the vector database, ANN searches only within a smaller region based on the proximity of the data. As the name suggests, it does not always return the exact result, but it provides the most preferred or approximate results much faster.

Is there any other method we can use to make the search more effective?

Metadata Filtering

Metadata means data about the data.

Metadata is stored along with each chunk. It can contain information related to the chunk, such as the chapter name, topic description, author, or any other relevant details.

When the user query contains information related to the metadata (for example, a chapter name or topic), the system can directly filter the relevant chunks before performing vector similarity search. This technique is called metadata filtering.

Metadata filtering is supported by:

Pinecone
ChromaDB
Qdrant

FAISS does not provide built-in support for metadata filtering.

Reranking

Documents are first split into chunks, and each chunk is converted into vectors and stored in the vector database.

When a user query arrives, it is converted into a vector and searched against the vector database to retrieve the closest chunks. However, we do not know whether the retrieved documents are actually the most relevant to the query. It is not always true that the closest vectors represent the most relevant documents.

How Reranking Works

The documents retrieved from the vector database are passed to a cross-encoder along with the user query.

The cross-encoder assigns a relevance score that indicates how closely each document matches the query. The documents are then displayed in ascending or descending order based on these scores.

The results produced by the cross-encoder are called reranked results.

The retrieved documents remain the same as those returned by the vector database, but their order changes. Documents with higher relevance scores appear before those with lower scores.

A cross-encoder is a neural ranking model. Instead of encoding the query and documents separately, it takes both the query and the document together as input to a transformer model and generates a relevance score for each document.

There are transformer models specifically designed for reranking tasks. The encoder understands the meaning of both the query and the document and reranks the documents accordingly.

Why Use Reranking?

Reranking is an important step in the RAG pipeline.

It is especially useful when working with documents that contain images or other multimodal content.

Example

Suppose the user asks:

"Show me the front view of the truck."

The vector database may retrieve multiple images related to trucks because they are semantically similar.

The reranker analyzes both the query and the retrieved images (or their associated text descriptions) and assigns relevance scores.

As a result, the image showing the front view of the truck receives a higher score than the other truck images, making it appear first in the final results.