Context Compression: Fitting More Useful Information Into Your LLM's Context Window

There is a tension at the heart of every enterprise RAG system. Better retrieval recall means more documents in the context. More documents in the context means longer prompts. Longer prompts mean higher inference cost, higher latency, and, past a certain length, degraded generation quality as the model's effective attention dilutes across too much content.

The way most teams resolve this tension is by limiting the number of retrieved documents passed to the LLM. Pull five chunks, pass five chunks. The limit is arbitrary but practically reasonable.

Context compression offers a different resolution. Instead of limiting how many documents you include, you reduce how many tokens each document contribution requires while preserving the information that is actually relevant to the specific query. The goal is to make the information denser rather than making the selection narrower.

This is not summarization. Summarization loses information indiscriminately. Context compression preserves information that is relevant to the query and discards information that is not. The output is a more efficient representation of the documents for that specific query, not a shorter version of the documents in general.

Why this matters in practice

Consider a typical enterprise retrieval scenario. An employee asks about the approval process for a specific expense category. The retrieval system pulls five chunks. One chunk is from the expense policy document and contains exactly the answer, embedded in two paragraphs about the approval process surrounded by three paragraphs about submission timelines, receipt requirements, and currency conversion for international expenses. The other four chunks are from adjacent policy sections that contain some relevant context but also significant irrelevant content.

Without compression, the LLM receives roughly 2,500 tokens across these five chunks, of which maybe 600 tokens are directly relevant to the question. The other 1,900 tokens are noise relative to this query, and at sufficient volume they degrade generation quality by diluting the model's attention on the relevant content.

With compression applied, the LLM receives the relevant portions of each chunk, perhaps 700 tokens total, representing a 72% token reduction with no loss of query-relevant information. The answer it generates is not just cheaper and faster. It is more accurate because the relevant content is not competing with irrelevant content for the model's attention.

Implementing LLM-based context compression

The most straightforward compression approach uses a small, fast LLM to extract the query-relevant portions of each retrieved chunk before passing the results to the main generation model.

from langchain.retrievers.document_compressors import LLMChainExtractor
from langchain.retrievers import ContextualCompressionRetriever
from langchain_openai import ChatOpenAI

# Or replace with your self-hosted inference endpoint
compression_llm = ChatOpenAI(
    model="gpt-4o-mini",  # small/fast model for compression
    temperature=0
)

compressor = LLMChainExtractor.from_llm(compression_llm)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=compressor,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 8})
)

# Usage - retrieves 8 chunks, compresses each to query-relevant portions,
# returns compressed versions to generation model
compressed_docs = compression_retriever.get_relevant_documents(query)

The compressor calls the compression LLM once per retrieved document, passing the document content and the original query, and asking the model to extract only the portions relevant to answering the query. Documents where no portion is relevant return an empty extraction and are dropped entirely.

The cost model

Adding a compression step adds latency and LLM cost. Whether this is net positive depends on your specific situation.

The compression adds API calls to a fast/cheap model (gpt-4o-mini, haiku, or a self-hosted small model). These calls are parallelizable and each one is processing a relatively small document chunk. At typical chunk sizes of 400 to 800 tokens, the compression call for each chunk is fast and inexpensive.

The savings come from reducing tokens in the main generation call, which uses a larger, more expensive model. If your main generation model costs 10x more per token than your compression model, and compression reduces the generation context by 60%, the net token cost is lower even accounting for the compression overhead.

For organizations with high query volumes, the economics are usually favorable. For low-volume deployments where latency matters more than cost, the additional latency from the compression step may not be justified.

def estimate_compression_economics(
    avg_retrieved_chunks: int,
    avg_chunk_tokens: int,
    expected_compression_ratio: float,  # e.g., 0.35 means 35% of tokens retained
    compression_model_cost_per_1k_tokens: float,
    generation_model_cost_per_1k_tokens: float,
    daily_queries: int
) -> dict:
    tokens_before_compression = avg_retrieved_chunks * avg_chunk_tokens
    tokens_after_compression = tokens_before_compression * expected_compression_ratio

    daily_compression_cost = (
        daily_queries * tokens_before_compression / 1000 * compression_model_cost_per_1k_tokens
    )
    daily_generation_savings = (
        daily_queries * (tokens_before_compression - tokens_after_compression) / 1000 *
        generation_model_cost_per_1k_tokens
    )

    return {
        "daily_compression_cost": daily_compression_cost,
        "daily_generation_savings": daily_generation_savings,
        "net_daily_savings": daily_generation_savings - daily_compression_cost,
        "break_even_compression_ratio": compression_model_cost_per_1k_tokens / generation_model_cost_per_1k_tokens
    }

Run this with your actual model costs and expected compression ratio to determine whether the economics make sense for your deployment.

Embedding-based compression as a faster alternative

For deployments where the latency of LLM-based compression is a constraint, embedding-based compression provides a lighter alternative. Rather than using an LLM to extract relevant sentences, it uses embedding similarity to filter individual sentences within each chunk.

from langchain.retrievers.document_compressors import EmbeddingsFilter
from langchain_openai import OpenAIEmbeddings

# Or replace with your self-hosted embedding model
embeddings = OpenAIEmbeddings()

embeddings_filter = EmbeddingsFilter(
    embeddings=embeddings,
    similarity_threshold=0.75  # only retain sentences with similarity above this threshold
)

compression_retriever = ContextualCompressionRetriever(
    base_compressor=embeddings_filter,
    base_retriever=vectorstore.as_retriever(search_kwargs={"k": 8})
)

This approach filters individual sentences within each document based on their embedding similarity to the query. Sentences below the similarity threshold are dropped. It is faster than LLM-based compression because it uses embedding inference rather than generation, but it is less accurate because it cannot use natural language understanding to determine relevance.

The appropriate choice between LLM-based and embedding-based compression depends on your latency requirements and your tolerance for compression accuracy. For high-stakes queries where answer accuracy matters, LLM-based compression is worth the additional latency. For high-volume, latency-sensitive applications, embedding-based compression provides meaningful context reduction at lower cost.

Combining compression with reranking

In a mature retrieval pipeline, compression and reranking address complementary problems and are most effective when combined. The retrieval pipeline order that I use in production:

1. Initial retrieval: embedding similarity search, k=20 candidates
2. Reranking: cross-encoder reranks to top 8 by relevance to query
3. Compression: LLM-based extraction of query-relevant portions from each of the top 8
4. Generation: main LLM receives compressed, reranked content

This pipeline produces a context that is both more relevant (from reranking) and more dense (from compression) than the baseline retrieve-and-generate approach. The cost is added latency from the additional processing steps. In my experience, the quality improvement justifies the latency cost for most enterprise knowledge retrieval applications, though the specific tradeoff should be measured against your actual query distribution and user requirements.

The goal is to give the generation model the most useful possible context for each specific query, not the most content. Compression is one of the underused techniques for achieving that goal.