I run a lot of RAG pipelines. And for a while, I was doing what most people do: retrieve the top-k documents, concatenate them into the context, and ship the whole thing to the LLM.
It worked. But when I started looking closely at what was actually in those contexts, I noticed something uncomfortable: a huge fraction of every prompt was filler. Background sentences that restated the obvious. Transition paragraphs that connected ideas the model didn't need connected. Prose padding that existed because technical documents are written for humans who need continuity, not for models that need information density.
And I was paying for every token of it.
The Problem Is Worse Than You Think
Here's what makes this particularly painful: it's not just a billing issue. Long-context prompts have real performance implications.
Quadratic attention cost. Transformer attention scales as O(n^2) with sequence length. Adding 2,000 filler tokens to a 4,000-token context doesn't increase cost by 50% - it increases the prefill computation by nearly 225%.
Lost in the Middle. Research consistently shows that LLMs underweight information in the middle of long contexts. If your most relevant retrieved chunk lands in position 12 of 20, the model may not give it the attention it deserves.
Instruction drift. This is the one that bit me hardest. When system directives are diluted across thousands of tokens of prose, models sometimes drop low-frequency rules. A JSON schema requirement buried after 3,000 tokens of context is easier to forget than one sitting at token 200.
What I Built
I spent about three months building LLMSlim, a Python library for surgical prompt compression. The goal was simple: reduce the token count of prompts significantly without losing any critical information and without touching system instructions.
The core of the library is a 6-step deterministic pipeline:
from llmslim import compress
# One line to compress your context
result = compress(
your_massive_rag_context,
target_ratio=0.5, # Keep 50% of tokens
strategy="extractive" # 100% offline
)
print(result.compressed_text) # Compressed context ready to use
print(result.token_reduction) # e.g., 0.52 (52% reduction)
How the Pipeline Works
Step 1: Protected Sentence Splitting
The text is split into sentences using regex patterns that avoid breaking on code fences (`), URLs, or Markdown title markers. AST fence content is preserved as atomic units.
Step 2: TF-IDF Vector Graph Construction
Each sentence becomes a vector in TF-IDF space. We compute pairwise cosine similarities to build a weighted graph where edges represent semantic similarity.
Step 3: LexRank Centrality Scoring
We run power iteration over the stochastic transition matrix derived from the similarity graph. Sentences with higher stationary probability scores are more informationally central to the document.
Step 4: Priority Tier Hard Locking
Before any pruning happens, a deterministic rule pass identifies and hard-locks:
- Tier 4 (highest): Sentences containing MUST, NEVER, ALWAYS, REQUIRED, or system role markers
- Tier 3: Sentences with numerical entities, proper nouns, URLs, and technical identifiers
These sentences are immune to the scoring algorithm. They survive regardless of centrality.
Step 5: Two-Pass Budget Allocation
Pass 1 allocates token budgets proportionally across semantic chunks. Pass 2 rebalances globally to hit the target ratio, using a priority-aware knapsack allocation.
Step 6: Ordered Reassembly
Selected sentences are reassembled in their original document order. This preserves logical flow and causal reasoning chains.
The Hybrid Strategy (v0.3.0)
The extractive pipeline is fast (sub-30ms) and fully offline. But for contexts where you want even higher compression quality, v0.3.0 introduces hybrid mode:
`python
from llmslim import compress, CallableProvider, RewriteRequest
Plug in any LLM you already have
def my_rewriter(req: RewriteRequest) -> str:
return openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": req.user_prompt}]
).choices[0].message.content
provider = CallableProvider(my_rewriter)
result = compress(
context,
target_ratio=0.45,
strategy="hybrid",
provider=provider
)
`
The hybrid mode runs extractive pre-pruning first (removing obvious noise), then passes the reduced context to the LLM rewriter for semantic compression. This achieves higher information density than pure extraction while keeping the LLM call focused on a smaller input.
Benchmarks
All benchmarks are reproducible and run on N=500 prompts per dataset:
| Dataset | Token Reduction | Latency | Directive Retention |
|---|---|---|---|
| System Directives | 51.4% | 24.8ms | 100.0% |
| 50k Long Context (GPT-5) | 55.0% | 26.0ms | 100.0% |
| XML Mode (Claude 3.5) | 50.0% | 24.0ms | 100.0% |
| 100k Megabyte RAG (Gemini) | 65.0% | 38.0ms | 100.0% |
The 100% directive retention across all test domains is the number I'm most proud of. It's not coincidental - it's the Priority Tier system doing exactly what it was designed to do.
What I Learned Building This
A few things surprised me:
TF-IDF is surprisingly competitive with embeddings for this task. I initially assumed I'd need neural embeddings for good centrality scoring. TF-IDF vector space turned out to work remarkably well for identifying genuinely redundant prose, largely because redundant sentences tend to share vocabulary.
The two-pass budget allocator matters more than I expected. A naive single-pass selection consistently underperforms because it doesn't account for how information clusters within documents. The second pass global rebalancing step is responsible for maybe 8-10% of the quality improvement over simpler approaches.
Instruction fidelity is a hard constraint, not a soft one. Early versions of the pipeline treated all sentences equally by score. The moment I realized I needed a hard exclusion layer (not just upweighting) for directives, the results became dramatically more reliable.
Try It
`bash
pip install llmslim
`
Full documentation, integration guides for LangChain, LlamaIndex, FastAPI, Ollama, and more are at https://www.llmslim.app
The source is on GitHub and the benchmark methodology is fully open if you want to reproduce or challenge the numbers.
I'd genuinely love to hear from anyone running production RAG pipelines about what compression approaches you've tried and what's worked. This is still a largely unsolved problem at the edges.
Top comments (2)
nice breakdown, especially the quadratic point most people skip over. one cheap complement to compressing: since lost-in-the-middle is roughly u-shaped, reordering buys you a lot. put your strongest chunks at the very top and bottom and let the weak ones sit in the sag, and you recover most of the 'buried at position 12' loss without dropping a single token. compression and reordering stack, and reordering is basically free.
The quadratic prefill cost is the one that changed how I approached this — at 4k context you're already paying 4x per useful token versus 1k, so filler compounds fast. The fix I've had most luck with is running a cross-encoder on sentence-level spans, not paragraph-level chunks, before final assembly: it cuts context by 40-50% without degrading recall because most of the prose glue you described scores below the sentence threshold. Graph-structured retrieval is another lever — instead of pulling whole document sections, you extract the entity-relationship subgraph relevant to the query and materialize only that, skipping transition sentences structurally rather than filtering them post-hoc. The lost-in-the-middle point is underrated; even after tightening context I now sort retrieved spans by relevance score descending rather than document order, which keeps high-signal content at positions the model actually attends to.