DEV Community: Ramya Perumal

AI Agents - Tool Calling And Its Types

Ramya Perumal — Mon, 05 Oct 2026 01:49:49 +0000

Tool means a function. The Agent will execute the function when it is needed. The tool should be written with a proper description. Without a proper description, the Agent cannot identify the tool properly.

Below are the following types of tool calling.

1. Function Call - Manual Loop

Here, tools will be bound with the LLM using a function called bind_tools. When this is done, the LLM will get to know the tool's name, description, and JSON Schema.

When the user asks a query, we check whether the response has an attribute called tool_calls. If the tool_calls list is empty, we take it as the final response. But if it is not empty, it means there are some tools that need to be called.

Our code will execute the tool and return the result to the LLM. This process will be repeated until the entire query is processed. Finally, the LLM generates the answer.

Here, the description of the tool should be clear so that the LLM can identify the tools carefully.

2. Custom Tools

We can build the tool in 3 ways:

Using decorators, which is a function call.
Using Pydantic - we can specify the specification, i.e., validating the input used in the tool. Basically, passing the arguments to the tool that need to be validated.
Structured tool - use the existing function and make it a tool.

It will be used in legacy code that we want to use as a tool along with parameters.

3. External API Tools

Here, tools will be calling an API URL to get the result and process the query according to the result.

Example:

User asks about the current weather of Chennai.

To get the weather, first we need to get the coordinates and then the weather. To get the coordinates, we will use a tool that calls an API to get the coordinates of the particular place. Here, it is Chennai.

Then the Agent will initiate the get_weather tool call to get the weather of the place. Using the result of the tool call, the LLM will generate the result in natural language.

DB Tools

We are using tools to get the result from the database.

For example,

User asks the query: Which customer from Chennai has spent the most in total, and how much?

We are using 2 tools called list_tables and run_sql_query.

list_tables will get all the table schemas from the database in a read-only mode and form a SQL query that will be used to fetch the result from the tables.

run_sql_query will take the SQL query and fetch the result.

Using the result of the tool call, the LLM will generate the result in natural language.

We should be more cautious about what type of query we give permission to execute. Suppose we give permission to execute a DELETE query; we will end up losing the data.

We need to define the schema properly.

AI Agents - Async Programming and Pytantic Data Validation

Ramya Perumal — Thu, 01 Oct 2026 18:35:32 +0000

Async

Sync means one after another. For example, calling 3 functions named f1, f2, and f3. Assume f1 performs some I/O operations like a DB query, API request, or file operation. After f1 is completed, f2 and f3 will be executed.

There are some operations that are running on top of the CPU.

E.g.

def add(a, b):
    return a + b

When we call this function, it will be executed by the CPU.

And some operations do not utilize the CPU, i.e., I/O operations.

E.g.

def getweatherdetails():
    return request to HTTP API

Here, to send a request, the CPU will be utilized. After sending the request, the CPU will not be used until we get the response from the third party. We need to wait for some time.

In this case, if the main function is calling all 3 functions one by one, we have to wait for f1 to get the response before moving to f2, and so on.

def main():

    f1()
    f2()
    f3()

Here, in Async, after sending the request to the third party, f2 will be performed without waiting for the response from the third party. Async means no waiting (context switching) and will be used only in I/O operations.

Async will be implemented in Python using a package called asyncio.

Here, make_toast() will be called first, and during the waiting time, make_tea() will be called.

Pydantic

By default, the output that comes from the LLM will not be structured. It will be in text format. Either we should structure it by writing code or use a prompt that is fed into the LLM so that the output can be used to take further action.

Python is a dynamically typed language.

age = 12
age = "12"

Here, the data type of the variable age is converted from int to string. Even though there are many benefits to dynamic typing, some problems may arise.

The response from the LLM will be in text format. So, we have to parse it using a regular expression or get the output in the desired format by creating a data structure. So, the LLM must adhere to the format to generate the output.

Example:

Data structure called:

{
    "name": "string",
    "age": "int"
}

So, we can access the name by using user.name.

Before accessing it, we have to do validation to check the correctness. Data validation will not be applied implicitly to control the data. It will increase the dumb code, which affects the readability and maintainability of the code.

To solve this problem, there is a package called Pydantic. This package will control the data validation. So, we can totally rely on the objects that are produced by Pydantic.

from pydantic import BaseModel, Field

class User(BaseModel):
    name: str
    age: int = Field(gt=0)

user = User(name="somename", age=30)
print(user)

To add more implicit conditions, we have to import the Field class. Here, we are saying that age should be greater than 0.

AI Agents - Tool Calling

Ramya Perumal — Wed, 09 Sep 2026 00:13:59 +0000

When a user asks the LLM to perform an action, e.g., get the current weather or current stock market details, the LLM cannot get this information on its own. It needs some functionality or tools along with a description of when to call these tools/functionality.

How is it working?

The LLM analyzes the query to find out whether it is related to any of the tools and their descriptions.

If so, the LLM will suggest the tool along with the parameters to the agent/external logic. The external logic will perform the action and send the result back to the LLM.

The LLM reads the result and generates the structured output.

Suppose the tool call is not matched; the LLM will derive the answer from its own knowledge. If we give an ambiguous description, the LLM may not generate the output properly.

The description for the tool calls must be proper so that the LLM can semantically identify the tool and match the arguments.

example description

"""
Add up two integer numbers.

This function simply wraps the `+` operator, and does not
do anything interesting, except for illustrating what is
the docstring of a very simple function.

Parameters
----------
num1 : int
    First number to add
num2 : int
    Second number to add

Returns
-------
int
    The sum of `num1` and `num2`

See Also
--------
subtract : Subtract one integer from another

Examples
--------
>>> add(2, 2)
4
>>> add(25, 0)
25
>>> add(10, -10)
0
"""

AI Agents - Introduction to LLM and AI Terminologies

Ramya Perumal — Tue, 01 Sep 2026 18:38:49 +0000

LLM

LLM is a model, which means an equation.

Example:

y = mx + c

y = m1x^3 + m2x^2 + m3x + m4

A model is actually made up of weights. In any model, e.g., ChatGPT model or Gemini model, they would have used a large amount of input to train the model.

Input means a large amount of text/image data that is available on the internet. The input would have been fed into the Transformer architecture to get the output, which is the model.

Weights are floating-point numbers that represent the model's learned parameters. A 10B or 100B parameter model means how many parameters (weights) are present inside the model.

We cannot store a large-parameter model on our computer due to inadequate storage and computational power. Storage and CPU/GPU power decide what size of model can be run on a computer.

To run a model locally, we can use one of the following tools:

Llama.cpp
Ollama
LM Studio

Open Weight Model vs Open Source Model

An open-weight model shares its model weights. So, we can run them, fine-tune them, and host them on a local system.

Here, the training code, data, and full methodology are not shared.

Whereas, in an open-source model, the weights, training code, data, and sometimes the dataset are shared.

Why Do We Need to Use LLMs?

LLM is a next-word predictor.

Suppose we ask:

"Hi, how..."

The answer can be:

How are you?
How do you do?
How is your life?
etc.

These are possibilities.

Here, most of the time, the answer will be "How are you?" because if a word has more presence, it has a higher possibility of occurring.

Each possibility will have a score between 0 and 1.

We have 3 controlling parameters to control the output generated by the LLM.

1. Temperature

Usually set from 0–1. It controls the randomness of the model.

If the value is 0–0.3, which is low, it means generating the most likely words, i.e., facts or commonly occurring words.

If the value is high, the model will choose less likely words.

We use this high value in storytelling and creative writing.

2. Top K

Controls the number of possibilities.

For example, K = 3 means choosing only the 3 most likely possibilities.

Top K will be used along with Temperature.

For example, if we set Top K = 5 and Temperature = 0.5, the LLM will take possibilities from the selected Top K values based on the temperature.

Top K is used to limit the number of possibilities.

3. Top P

It is also called a sampling method.

The possibility scores are added up until they reach the defined value in Top P.

Top P is also used along with Temperature.

Tokens

Tokens are the process of splitting words into small pieces. A small piece may not be complete.

For example, a token can be split into "to" and "kens".

These are called tokens, and the process is called tokenization.

A tokenizer, such as those used by OpenAI and GPT, can be used.

We can specify the size of the token that we want to split.

Each token will be represented by a number, which in turn is converted into an embedding.

Context Window

It is a short-term memory.

The context window is the maximum number of tokens that a model can see at a given time.

Short context window and long context window are the types.

AI Agents - Data Structures in Python

Ramya Perumal — Sun, 30 Aug 2026 22:07:35 +0000

Data Structure means data stored in some structured format. How will we store strings, integers, and booleans in a data structure? Here we will see List, Dictionary, Tuple, and Set data structures in details.

Data structures can be treated as containers.

List

It is a dynamic array. An array is a collection of data that is stored consecutively. In Java or other languages, for a normal array, we have to specify the size to refer to the data. Every array has an index that points to the value of the array. The index starts from 0.

num[0] = 1
num[1] = 10
etc.

An array will maintain the data insertion order.

Dynamic array means allocating more space at runtime.

We can store heterogeneous data in the list, which means storing different data types or structures inside a single sequential collection.

E.g.
nums = [2, 3, 6, 8, 0, 1]

nums[0] - access the value at index 0
len(nums) - find the length of the list
nums[-1] - access the last item

nums.append(5) - add the new value at the end
nums.clear() - empty the list
nums.count(5) - count the number of occurrences of the particular value in the list
nums.reverse() - reverse the entire list
nums.sort() - sort the values in the list
nums.insert(0, 10) - insert the value 10 at index 0

These are the functions that can be used with a list.

Dictionary

Dictionary means a key-value pair.

Syntax:

dict_name = {key: value, key: value}

student1 = {"name": "some name", "age": 20}

Here, the key is immutable. The key can be either a string, boolean, integer, or tuple.

Generally, to search for a particular value, we need to search all the values one by one. But in a dictionary, by using the key, fetching time will be faster compared to other data structures. We can retrieve the value in O(1) time.

dict.get() - get the value of a key
dict.keys() - list the keys in the dictionary
dict.values() - list the values in the dictionary
dict.items() - list the pairs in the dictionary

These are the functions associated with a dictionary.

Tuple

An analogy for a tuple is taking a screenshot. Once we take the screenshot, we cannot change anything inside it. The same is applicable to a tuple. Once a tuple is created, we cannot change its values. We can delete the tuple by using the del keyword.

Syntax:

tuple_name = (value1, value2, value3)

A tuple is created mainly to preserve values as original and to maintain their integrity. So, it is immutable. Lists and dictionaries are mutable, where we can change the values in them.

Functions can always return a tuple.

Set

Set is a collection of unique elements. Here, duplication of elements is not allowed. To implement this, hashing methodology is implemented internally. It is the same as set operations in mathematics.

Syntax:

set_name = {element1, element2, element3}

Set does not maintain the user-specified order. Even if we try to add duplicates, it will not allow them.

Set will add the new element at any index. That is why it does not have an index.

Some of the methods in Set:

set_name1.union(set_name2) - union between two sets  
set_name1.intersection(set_name2)  - common elements between two sets 
set_name1.issubset(set_name2)  - check set1 is subset of set2 
set_name1.issuperset(set_name2) - check set1 is superset of set2
set_name1.pop() - remove random element
set_name1.remove(element) - remove specified elements

AI Agents - Python basics

Ramya Perumal — Tue, 25 Aug 2026 22:03:25 +0000

AI Agents - Python Basics

Why do we need to use Python to create AI agents?
Because the number of users in the community is bigger, and the number of libraries is more compared to other languages.
Why not use popular languages like Java or Go? Because every language is used for different purposes.

Python is called an interpreted language, which means it executes the code line by line and does not compile it first. Other languages like Java and C++ compile the code (i.e., convert it into binaries, 0s and 1s) and execute it from the compiled file.

Print Function

Prints the content in the console that is inside the function.

A function is simply a plan to do a task repeatedly. Here, we create a function when we do not want to repeatedly do the same work again and again.

Syntax:

def function_name(parameter_list):
return

E.g.

print("Hello World")

print - is the function_name, and "Hello World" is printed in the console.

print("Hello World", "Hi")

Identifier

name = "some name"

The value will be stored in RAM.

There are two types of memory in a computer.

RAM - Random Access Memory - Temporary memory used to access data faster for computational purposes.

ROM - Read Only Memory - Hard drive.

Initially, the OS will be stored in the ROM or hard drive. The OS will not run directly from the ROM. When the computer starts, the OS files will be moved to RAM because fetching data from RAM will be much faster. Since RAM is costly, we cannot store all the data in it.

"some name" will be stored somewhere in RAM. We cannot remember the memory address for the process all the time, so we assign a label called "name". In Python, there is no concept called a "Variable". The same label can be replaced with a different memory address. So, the "name" identifier will point to the address where the value is stored.

E.g.

age = 30
Flag = True
name = "Peter"

These are string/boolean/integer/None types of values that can be stored in the identifier.

In Python, everything will be treated as an object. Each object has attributes and functions. Each object may or may not have attributes and functions. To know whether a particular object has attributes and functions, use the method called 'dir'.

dir(age)

It will list down the attributes and functions associated with the object.

Acknowledgment

A special thanks to my mentor Syed Jafer for the continuous guidance and support throughout these sessions. I truly appreciate the effort taken to explain each concept clearly and make the learning process easier.

RAG - Hallucination Detection

Ramya Perumal — Fri, 21 Aug 2026 00:54:08 +0000

Hallucination

Hallucination means making an assumption or making up something when the LLM does not know the answer.

Hallucination in RAG

Example:

Suppose we have a PDF file that contains information about Python, but it does not have any details about Decorators.

Here, the document is split into chunks, and the chunks are stored in the vector database.

Suppose we ask the query:

"What is a decorator?"

The LLM should not give any response because the information about decorators is not available in the database.

But the LLM may give a response from its own knowledge, which may be correct, but it is not from the database.

This is called hallucination in RAG.

Methods to Measure Hallucination

Knowledge Groundedness

The response given by the LLM should be supported by the information given in the context.

If the response is not supported by the context, it is considered hallucinated.

Ways to Detect Hallucination

1. Embedding

Compare the embedding of the context with the embedding of the LLM response to check how closely they are related.

2. LettuceDetect

LettuceDetect is a BERT-based model used to find where hallucinations could occur at the character level.

It checks the response word by word to identify possible hallucinations.

3. LLM as a Judge

Give the response to another LLM and ask it to detect hallucinations.

The drawback here is that the LLM may hallucinate again, and it can also be expensive unless we use a local model.

4. RAGAS Framework

RAGAS is also called an evaluation framework. It provides different metrics that can be used to evaluate RAG systems and identify potential hallucinations.

Some of the important metrics are:

Faithfulness – Checks whether the results are backed by the provided context. This can be used to check hallucination.
Answer Relevancy – Suppose the context and answer are relevant. This metric checks how relevant the answer is to the query.
Context Precision – Checks how many of the retrieved documents are useful and relevant.
Context Recall – Checks whether the necessary documents have been retrieved for the query.
Answer Correctness – Checks the response against the ground truth.
Answer Similarity – Checks the semantic similarity between the expected answer and the generated answer.

RAGAS can be used in a CI/CD pipeline.

We can write unit test cases and compare the actual answer with the expected answer using the above metrics.

DeepEval can also be used to evaluate RAG systems and detect hallucinations.

RAG - Async Pipelines, MCP

Ramya Perumal — Tue, 18 Aug 2026 19:47:08 +0000

What is Synchronous?

Everything goes sequentially.

Example:

P1, P2, and P3 are the three processes.

Synchronous means P2 will start when P1 starts and finishes its job. P3 will start when P2 starts and finishes its job.

It is not required all the time.

During retrieval, in the case of hybrid search, first we will do vector search, then text search, and then any work related to semantic caching / context search.

The goal here is to give more relevant context to the LLM.

Suppose:

Vector search takes 10 seconds
Text search takes 5 seconds
Semantic caching / context search takes 2 seconds

In the case of a sequential process, we need to wait 17 seconds to get the context.

So, is there any way we can reduce the waiting time?

The answer is Async Pipelines.

Here, we will start 3 threads at the same time.

One thread will do vector search.
Another thread will do text search.
The third thread will do semantic caching / context search.

Their finishing times may be different.

At most, we have to wait only 10 seconds to get the context.

Through this async pipeline, we are reducing latency.

Context Switching

A single thread switches between multiple tasks when needed. It is also called parallel processing.

Multithreading

Each task will be allotted a separate thread. So, it can finish on its own time.

These threads are limited to the available CPU cores.

We can see the real difference in latency when we do this at the production level.

MCP - Model Context Protocol

If we ask the question "What happened today?", an LLM cannot answer.

If we ask the same question to ChatGPT or Gemini, it gives a response. How?

Because of tool calling, which means attaching some tools or functionality to the LLM. The result of the tool will be fed into the LLM.

The application developer will be responsible for writing the code for the tool/functionality.

Example

Suppose the idea of the application is to fetch weather details.

Every developer may write their own code to fetch the weather details. At the end, all users/developers will get more or less the same result.

This is not the right approach.

So, here comes the concept of MCP.

The responsible body/owner that provides the weather details will write the common tool or protocol (a set of rules).

The consumer/developer will use the common tool/protocol to fetch the result.

Here, the developer does not need to write their own code.

The common tool/protocol is called MCP.

MCP will have functions and their descriptions. The LLM will decide which function to call.

We can call the tool either by:

1. StdIO (Standard Input/Output)

MCP is within the same machine.

2. HTTP

We can also use the HTTP method to call the MCP, such as calling an API.

However, we are introducing latency when using HTTP.

What is the Relation Between MCP and RAG?

Create an MCP for a RAG system.

For example, suppose we have built a RAG system using documents that we gathered.

Since you have the database and RAG system, you are the only person who can invoke the RAG system directly.

Instead of keeping the RAG functionality restricted to your application, we can expose the RAG functionality through MCP.

We can segregate the functionality according to different responsibilities or divisions.

For example:

RAG system → Responsible for retrieving relevant information from the documents
MCP → Exposes the RAG functionality as a common tool
LLM/Application → Can invoke the MCP tool when it needs information

Now, anyone who has access to the MCP can use the RAG functionality without directly accessing the underlying database or writing their own retrieval code.

In this way, MCP acts as a common interface between the RAG system and different applications or LLMs.

The RAG system continues to handle document retrieval, while MCP provides a standardized way for other applications or agents to access that functionality.

RAG - Memory Systems

Ramya Perumal — Thu, 13 Aug 2026 18:08:59 +0000

We need memory to store the previous conversational history.

For example:

Previous question is:

User: File handling in Python

Assistant: Explain about file handling.

Next time, the user asks:

User: What are the modes in it?

The LLM should understand the context and then respond.

Below are the details that can be stored in the memory.

Past Conversations
User Preferences that we specify in the system prompts, e.g., JSON format
Past Decisions
Previous Tasks

Based on the details we are storing, we will choose between long-term or short-term memory.

Short-Term Memory

Redis, Valkey, Memcached, and Cosmos, which are cached databases, can be used for short-term memory to store the last few conversations or a summary.

We can set a general data invalidation rule to erase the content or use an LRU cache eviction policy, where the least recently used data will be erased from the database.

Long-Term Memory

Postgres, Pinecone, and MongoDB can be used for long-term memory to store long conversation histories.

How It Is Functioning

A summary of the entire conversation history will be stored in short-term memory to reduce latency whenever needed.

Episodic Memory

Episodic memory is a type of memory that stores specific events or experiences that happened in the past, usually together with information about what happened, when it happened, and the context surrounding it.

We can use either a short-term or long-term memory database depending on the use case. It is a kind of combination of short-term and long-term memory.

User:

I am planning a trip to Paris.

Agent:

How many days will you stay?

User:

5 days.

Later,

User:

Can you suggest an itinerary?

Episode 1

User wants to travel to Paris.

Trip duration: 5 days.

User previously mentioned:

Destination = Paris

Duration = 5 days

This information can be used to provide a more relevant response.

This helps the LLM understand what happened previously.

Semantic Memory

Semantic memory contains facts extracted from previous conversational history. Semantic memory is generally considered long-term memory.

Example

Paris is the capital of France.

The Louvre is a museum in Paris.

France uses the Euro.

That's general knowledge.

Sliding Window Memory

It is a short-term memory. Here, we store the last 3 to 4 conversations.

Redis or Valkey, like any cache memory, can be used.

Summarized Memory

Each and every time, the conversation, which includes the user query and response, will be summarized.

Even though token consumption during summarization is more, overall token consumption will be less.

It is a long-term memory.

Entity Fact Memory

This memory is used to store facts about a particular entity.

The difference between entity fact memory and semantic memory is that semantic memory is the broader category. Entity fact memory is one way of organizing and storing semantic knowledge about specific entities.

It can be used as either long-term or short-term memory.

Semantic Memory

Entity: Python

Python → is a programming language

Python → is used for AI

Python → supports object-oriented programming

Python → was created by Guido van Rossum

Entity Fact Memory

Entity: Alice

Alice → works at ABC Company

Alice → prefers Python

Alice → is working on Project X

It is not a good practice to store the entire conversation. We can make decisions based on the conversation and then store the relevant information. This is a good practice.

RAG - Multivector Retrievel, Multi Hop, Conversational RAG

Ramya Perumal — Wed, 12 Aug 2026 17:51:51 +0000

Multi-Vector Retrieval

In a typical RAG pipeline, every chunk is converted into an embedding and stored in the vector database.

In Multi-Vector Retrieval, instead of creating a single embedding for a chunk, multiple embeddings are created to represent different aspects of the original chunk.

Each embedding captures a different perspective of the same content. The metadata of each embedding contains a reference to the original chunk.

When a user query is received, the retriever searches across all these embeddings. Since each embedding represents a different aspect of the content, the chances of retrieving more relevant information are higher.

This approach generally provides better retrieval performance.

However, it is a costly approach because:

Multiple embeddings are generated for every chunk.
More storage is required in the vector database.
More tokens are consumed during embedding generation.

If no other optimization technique is able to improve the RAG performance, Multi-Vector Retrieval can be considered as a final optimization step.

Multi-Hop

Multi-Hop Retrieval is used when the answer cannot be obtained from a single piece of context. Instead, the LLM has to retrieve multiple related contexts and connect them to generate the final answer.

Example

Suppose the knowledge base contains the following information:

Biryani contains spices.
The spices used in biryani are cardamom, cinnamon, cloves, etc.

Now suppose the user asks:

"What spices need to be added to biryani?"

The LLM first retrieves the information:

"Biryani contains spices."

It then hops to the next related context:

"The spices used in biryani are cardamom, cinnamon, cloves, etc."

Finally, it combines both pieces of information to generate the complete answer.

In this process, the LLM hops from one retrieved context to another until it gathers enough information to answer the user's query.

Each retrieved context should have a meaningful relationship with the next one so that the LLM can follow the chain of information and produce the correct response.

Conversational RAG

A user may not restrict themselves to asking only one query. They may ask a sequence of queries. Here, we are going to look at how to build Conversational RAG.

Traditional Flow

User Query
Retriever
Related Documents
LLM
Answer

A user may ask questions that are indirect or related to the previous question. So, we need to implement Query Transformation and Expansion to get the related queries and their responses.

Therefore, we need to have memory to store the previous conversation history.

Conversational RAG will be:

Conversation History + Current Question
Intent
Query Transformation and Expansion
Retriever
Relevant Context
LLM
Context-Aware Answer

Here, we are going to see how we can store conversation history in memory.

We can use persistent memory such as PostgreSQL or SQLite, or short-term memory using Redis, Valkey, Memcached, etc. We can also use a pickle file to store the history.

If we are using a file, make sure that the application we are using is single-threaded.

To store the history, we can use the session ID as the primary key.

Always store the entire history in long-term memory. However, we cannot store the entire history in short-term memory.

The purpose of storing history in short-term memory is to reduce latency.

To solve this problem, we can store the entire history in long-term memory and summarize the conversation history and store the summary in short-term memory whenever needed.

RAG - Parent Retriever

Ramya Perumal — Wed, 05 Aug 2026 01:52:04 +0000

RAG Pipeline Optimization Techniques

The following techniques are commonly used to optimize a RAG pipeline:

Semantic Caching
Query Transformation and Expansion
Context Compression
Parent Retrieval
Multi-Vector Retrieval

These techniques are used to improve the quality of the responses generated by the LLM.

Parent Retrieval

Today, we are going to focus on Parent Retrieval.

A chunk is a small portion of a document or paragraph, typically consisting of 500–1000 words. When these chunks are converted into embeddings and stored in a vector database, semantically related chunks are positioned close to one another.

When a user submits a query, the system retrieves the chunks that are closest to the query. However, not all the retrieved chunks may be the most relevant. Sometimes, we may miss other chunks that provide better context for the user's query.

Example

Suppose we have the following paragraphs:

Paragraph 1 (P1)

Each line is stored as an individual chunk:

P1C1
P1C2
P1C3
P1C4

Paragraph 2 (P2)

Each line is also stored as individual chunks:

P2C1
P2C2
P2C3

Suppose the expected answer to the user query is P1C2.

The vector database retrieves:

P1C2
P2C3

Although P1C2 is correctly retrieved, the related chunks P1C1, P1C3, and P1C4 may provide much better context than P2C3. Since these chunks are not retrieved, we may lose important context that could improve the final LLM response.

If we store the document paragraph-wise instead of using smaller chunks, we may retrieve unnecessary context, which increases token consumption.

Solution – Parent Retrieval

Suppose Paragraph 1 is divided into four chunks:

P1C1
P1C2
P1C3
P1C4

Whenever a specific chunk is retrieved from the vector database, it also contains a reference to its parent paragraph, which is stored in the chunk's metadata.

For example, if P1C2 is retrieved, the metadata also contains a reference to Paragraph 1.

Using this reference, the retriever can also fetch the remaining chunks belonging to the same parent paragraph:

P1C1
P1C3
P1C4

This ensures that we do not miss important context that is closely related to the user's query.

Example

Suppose the user query is:

"Create a dictionary"

The retrieved chunks are:

P1C2
P2C2
P2C3

Instead of sending only these chunks to the LLM, Parent Retrieval also includes the remaining chunks from the same parent paragraph:

P1C1
P1C3
P1C4

Providing this additional context enables the LLM to generate a more accurate response.

When Should Parent Retrieval Be Used?

Parent Retrieval is useful when we feel that important context is missing from the retrieved chunks and we want to include additional context that is closely related to the query.

The parent paragraph is stored similarly to other chunks. However, it is not used directly for vector search. Instead, its reference is stored in each chunk's metadata, allowing the retriever to fetch the parent context whenever required.

RAG - Context Compression

Ramya Perumal — Thu, 30 Jul 2026 01:58:52 +0000

In a RAG pipeline, documents are first split into chunks, converted into embeddings, and stored in a vector database.

When a user query arrives, it is converted into an embedding and used to retrieve relevant results from the vector database. Although the retrieved results may be relevant, they may not always be directly related to the user's query.

Example

User Query

"FastAPI dependency injection"

The retrieved results may include:

"Use FastAPI in deployment"
"How to dockerize a FastAPI application"

These results may be semantically related to FastAPI, but they are not necessarily the most accurate results for the user's query. This depends on how the documents were embedded and stored in the vector database.

To improve the quality of the retrieved context, the fetched results can be compressed into one or two chunks before they are passed to the augmentation phase. This reduces the number of tokens used during augmentation and acts as an optimization technique.

From the above example, the chunks retrieved from the vector database may not be completely relevant to the user query. Therefore, we compress the retrieved chunks without changing their original context and create one or two chunks that are more relevant to the query.

Context compression is not a mandatory step in every RAG pipeline. It is generally adopted based on trial and error and depends on the application's requirements.

This compression technique is mainly useful when:

There are token constraints.
Higher-quality retrieval results are expected.

Context Compression Techniques

There are different approaches for context compression:

LLM-based Compression
Embedding-based Compression
Keyword-based Compression

1. LLM-based Compression

In this approach, the relevant chunks retrieved from the vector database are given to an LLM for compression.

For example, suppose four chunks are retrieved from the vector database. Instead of sending all four chunks to the main LLM, they are first passed to another LLM whose responsibility is only to compress them into one or two meaningful chunks.

Although using an LLM for compression may appear expensive, a locally deployed LLM or a low-cost LLM is typically used for this task.

The purpose of this LLM is not to generate the final response. Its responsibility is only to merge and compress the retrieved chunks while preserving their context.

The compressed chunks are then sent to a more powerful or application-specific LLM to generate the final response.

This creates a two-stage pipeline, which is more token-efficient while still producing high-quality output.

2. Embedding-based Compression

In embedding-based compression, the embeddings of the top retrieved chunks are compared with the embedding of the user query using cosine similarity.

The chunks that are most relevant to the user query are selected and used as the compressed context.

3. Keyword-based Compression

Keyword-based compression uses techniques such as TF-IDF or the BM25 algorithm.

The keywords present in the user query are compared with the retrieved chunks.

The chunks that have the highest keyword relevance are selected and used as the compressed context.

Combining Compression Techniques

A single application is not limited to using only one compression technique.

Depending on the application requirements, two or more context compression techniques can be combined to improve retrieval quality while reducing token usage.