In a RAG pipeline, documents are first split into chunks, converted into embeddings, and stored in a vector database.
When a user query arrives, it is converted into an embedding and used to retrieve relevant results from the vector database. Although the retrieved results may be relevant, they may not always be directly related to the user's query.
Example
User Query
"FastAPI dependency injection"
The retrieved results may include:
- "Use FastAPI in deployment"
- "How to dockerize a FastAPI application"
These results may be semantically related to FastAPI, but they are not necessarily the most accurate results for the user's query. This depends on how the documents were embedded and stored in the vector database.
To improve the quality of the retrieved context, the fetched results can be compressed into one or two chunks before they are passed to the augmentation phase. This reduces the number of tokens used during augmentation and acts as an optimization technique.
From the above example, the chunks retrieved from the vector database may not be completely relevant to the user query. Therefore, we compress the retrieved chunks without changing their original context and create one or two chunks that are more relevant to the query.
Context compression is not a mandatory step in every RAG pipeline. It is generally adopted based on trial and error and depends on the application's requirements.
This compression technique is mainly useful when:
- There are token constraints.
- Higher-quality retrieval results are expected.
Context Compression Techniques
There are different approaches for context compression:
- LLM-based Compression
- Embedding-based Compression
- Keyword-based Compression
1. LLM-based Compression
In this approach, the relevant chunks retrieved from the vector database are given to an LLM for compression.
For example, suppose four chunks are retrieved from the vector database. Instead of sending all four chunks to the main LLM, they are first passed to another LLM whose responsibility is only to compress them into one or two meaningful chunks.
Although using an LLM for compression may appear expensive, a locally deployed LLM or a low-cost LLM is typically used for this task.
The purpose of this LLM is not to generate the final response. Its responsibility is only to merge and compress the retrieved chunks while preserving their context.
The compressed chunks are then sent to a more powerful or application-specific LLM to generate the final response.
This creates a two-stage pipeline, which is more token-efficient while still producing high-quality output.
2. Embedding-based Compression
In embedding-based compression, the embeddings of the top retrieved chunks are compared with the embedding of the user query using cosine similarity.
The chunks that are most relevant to the user query are selected and used as the compressed context.
3. Keyword-based Compression
Keyword-based compression uses techniques such as TF-IDF or the BM25 algorithm.
The keywords present in the user query are compared with the retrieved chunks.
The chunks that have the highest keyword relevance are selected and used as the compressed context.
Combining Compression Techniques
A single application is not limited to using only one compression technique.
Depending on the application requirements, two or more context compression techniques can be combined to improve retrieval quality while reducing token usage.
Top comments (0)