DEV Community: Yashvardhan Thanvi

Building LLMSlim: Architecture Deep-Dive into Deterministic Prompt Compression

Yashvardhan Thanvi — Sat, 18 Jul 2026 22:23:27 +0000

Most prompt compression discussions focus on the happy path: you have a long RAG context, you trim it to 50% of tokens, and your API bill halves. What rarely gets discussed are the failure modes: dropped system instructions, truncated JSON schemas, broken code fences, and entity names that get quietly pruned because they scored low in some similarity metric.

LLMSlim (https://www.llmslim.app) is a Python library I've spent the last few months building to handle these edge cases properly. This post goes deep on the architecture decisions that make it work.

The Core Problem: Information vs. Filler

LLM input prompts have a structural problem. The instructions that govern model behavior (system roles, JSON schemas, MUST/NEVER directives) are typically a tiny fraction of the total token count. The rest is context: retrieved documents, conversation history, background information. And within that context, a significant fraction is prose that exists for human readability, not informational density.

The challenge is distinguishing high-value informational sentences from low-value filler without a model call, without embeddings, and in under 30ms.

The 6-Step Pipeline

LLMSlim processes prompts through a deterministic DAG (Directed Acyclic Graph) with six stages:

1. Protected Sentence Splitting

Naive sentence splitting on periods breaks code blocks and URLs. The first stage uses regex-based splitting that respects:

AST code fences (triple backticks) - treated as atomic units
Markdown heading markers
URLs (no splitting on periods within URLs)
Common abbreviations (e.g., U.S., etc.)

# Pseudocode: core split logic
sentences = regex_split(text)
for i, sent in enumerate(sentences):
    if is_code_fence(sent): protected[i] = True  # immune to scoring

2. TF-IDF Vector Graph Construction

Each sentence becomes a TF-IDF vector. We compute the full pairwise cosine similarity matrix over these vectors to build a weighted undirected graph where edge weight(i,j) = cosine_similarity(v_i, v_j).

The choice of TF-IDF over neural embeddings was deliberate. TF-IDF runs in microseconds per sentence. It captures vocabulary overlap well, which is exactly what matters for identifying redundant prose. Neural embeddings add 50-100ms of latency and model loading overhead that isn't justified for this task.

3. LexRank Centrality Scoring

We apply the LexRank algorithm: convert the similarity graph to a stochastic transition matrix M, then find the stationary distribution via power iteration:

p_t+1 = d * M * p_t + (1-d) / n

where d is a damping factor (typically 0.85) and n is the sentence count. Convergence typically happens in 20-30 iterations. The resulting stationary probability vector gives each sentence a centrality score representing how informationally central it is to the document.

4. Priority Tier Hard Locking

This is the stage that separates LLMSlim from naive extraction approaches. Before any sentence can be pruned, a deterministic rule pass classifies every sentence into one of four tiers:

Tier 4 (Inviolable): Sentences containing:

Role markers: system:, developer:, user:
Imperative keywords: MUST, NEVER, ALWAYS, REQUIRED, DO NOT
JSON/XML schema delimiters

Tier 3 (Protected): Sentences containing:

Numerical entities (currency, percentages, measurements)
Proper nouns and named entities
URL references
Code identifiers

Tier 2 (Standard): Regular content sentences

Tier 1 (Candidate for removal): Filler phrases, transition sentences

Tier 4 sentences are hardcoded to survive the compression pass regardless of their LexRank score. A sentence saying "You MUST respond only in JSON" will score low in a document full of prose paragraphs - but it's the most important sentence in the prompt.

5. Two-Pass Budget Allocation

Pass 1 divides the document into semantic chunks and allocates token budgets proportionally:

chunk_budget_i = target_tokens * (chunk_tokens_i / total_tokens)

Pass 2 applies a global rebalancing step. Each chunk may have selected sentences that, together, slightly exceed or undershoot its budget. The second pass collects the surplus/deficit across all chunks and redistributes tokens using a priority-aware greedy knapsack:

Sort remaining unselected sentences by (tier, lexrank_score) descending
Greedily include sentences until global token target is hit

This two-pass approach is why the library consistently hits within 2-3% of the target ratio even on highly variable document structures.

6. Ordered Reassembly

Selected sentences are sorted by their original document position and concatenated. Original ordering is preserved because causal reasoning and logical flow in documents depends on sequence, not just content.

The Hybrid Strategy (v0.3.0)

v0.3.0 adds a generative compression layer on top of the extractive pipeline. The flow:

Extractive pre-pruning reduces the context to ~130% of target (intentionally slightly over)
A RewriteRequest is passed to a pluggable CallableProvider
The LLM rewrites the pre-pruned context targeting the final ratio
A validation pass checks that all Tier 4 sentences survived
If validation fails, the extractive output is returned as fallback

The pluggable provider model means you don't need a specific LLM API key:

from llmslim import compress, CallableProvider, RewriteRequest

def my_provider(req: RewriteRequest) -> str:
    # req.system_prompt, req.user_prompt, req.target_ratio all available
    return your_llm_function(req.user_prompt)

provider = CallableProvider(my_provider)
result = compress(context, target_ratio=0.4, strategy="hybrid", provider=provider)

Benchmarks

All benchmarks are reproducible. Hardware: AMD EPYC 7763, 64GB RAM, Ubuntu 24.04, Python 3.12.3, tiktoken cl100k_base. N=500 prompts per dataset, 100 iterations per sample.

Dataset	Token Reduction	Latency (mean)	Directive Retention	Entity Preservation
System Directives	51.4% ± 1.2%	24.8ms ± 2.1ms	100.0%	95.1% ± 1.1%
50k Context (GPT-5)	55.0% ± 1.1%	26.0ms ± 2.4ms	100.0%	94.9% ± 1.2%
XML Mode (Claude 3.5)	50.0% ± 0.8%	24.0ms ± 1.9ms	100.0%	96.4% ± 0.9%
100k RAG (Gemini)	65.0% ± 1.3%	38.0ms ± 3.1ms	100.0%	94.8% ± 1.3%

What Didn't Work

Attempt 1: Pure TF-IDF cutoff. Setting a similarity threshold and dropping sentences below it sounds simple. In practice, document sections with specialized vocabulary score low across the board even when they contain critical information. Threshold tuning became document-specific.

Attempt 2: Summarization for compression. Running a smaller model to summarize chunks seems appealing. It adds a full model inference call (50-500ms), requires an API dependency, and summaries tend to drop entity specifics that matter for downstream model performance.

Attempt 3: Single-pass budget allocation. Works fine for uniform documents. Fails badly on documents with mixed density: technical specs mixed with background narrative. The two-pass approach emerged from debugging these failures.

Installation and Links

pip install llmslim

Website with full docs: https://www.llmslim.app
GitHub with reproducible benchmarks: https://github.com/Thanatos9404/llmslim

The benchmark scripts, raw JSON payloads, and full methodology are open. If you find issues with the numbers, open an issue - I'd genuinely like to know.

Why Your LLM Pipeline Is Burning 60% of Its Token Budget on Noise (and How to Fix It)

Yashvardhan Thanvi — Sat, 18 Jul 2026 22:13:03 +0000

I run a lot of RAG pipelines. And for a while, I was doing what most people do: retrieve the top-k documents, concatenate them into the context, and ship the whole thing to the LLM.

It worked. But when I started looking closely at what was actually in those contexts, I noticed something uncomfortable: a huge fraction of every prompt was filler. Background sentences that restated the obvious. Transition paragraphs that connected ideas the model didn't need connected. Prose padding that existed because technical documents are written for humans who need continuity, not for models that need information density.

And I was paying for every token of it.

The Problem Is Worse Than You Think

Here's what makes this particularly painful: it's not just a billing issue. Long-context prompts have real performance implications.

Quadratic attention cost. Transformer attention scales as O(n^2) with sequence length. Adding 2,000 filler tokens to a 4,000-token context doesn't increase cost by 50% - it increases the prefill computation by nearly 225%.

Lost in the Middle. Research consistently shows that LLMs underweight information in the middle of long contexts. If your most relevant retrieved chunk lands in position 12 of 20, the model may not give it the attention it deserves.

Instruction drift. This is the one that bit me hardest. When system directives are diluted across thousands of tokens of prose, models sometimes drop low-frequency rules. A JSON schema requirement buried after 3,000 tokens of context is easier to forget than one sitting at token 200.

What I Built

I spent about three months building LLMSlim, a Python library for surgical prompt compression. The goal was simple: reduce the token count of prompts significantly without losing any critical information and without touching system instructions.

The core of the library is a 6-step deterministic pipeline:

from llmslim import compress

# One line to compress your context
result = compress(
    your_massive_rag_context,
    target_ratio=0.5,  # Keep 50% of tokens
    strategy="extractive"  # 100% offline
)

print(result.compressed_text)  # Compressed context ready to use
print(result.token_reduction)  # e.g., 0.52 (52% reduction)

How the Pipeline Works

Step 1: Protected Sentence Splitting
The text is split into sentences using regex patterns that avoid breaking on code fences (`), URLs, or Markdown title markers. AST fence content is preserved as atomic units.

Step 2: TF-IDF Vector Graph Construction
Each sentence becomes a vector in TF-IDF space. We compute pairwise cosine similarities to build a weighted graph where edges represent semantic similarity.

Step 3: LexRank Centrality Scoring
We run power iteration over the stochastic transition matrix derived from the similarity graph. Sentences with higher stationary probability scores are more informationally central to the document.

Step 4: Priority Tier Hard Locking
Before any pruning happens, a deterministic rule pass identifies and hard-locks:

Tier 4 (highest): Sentences containing MUST, NEVER, ALWAYS, REQUIRED, or system role markers
Tier 3: Sentences with numerical entities, proper nouns, URLs, and technical identifiers

These sentences are immune to the scoring algorithm. They survive regardless of centrality.

Step 5: Two-Pass Budget Allocation
Pass 1 allocates token budgets proportionally across semantic chunks. Pass 2 rebalances globally to hit the target ratio, using a priority-aware knapsack allocation.

Step 6: Ordered Reassembly
Selected sentences are reassembled in their original document order. This preserves logical flow and causal reasoning chains.

The Hybrid Strategy (v0.3.0)

The extractive pipeline is fast (sub-30ms) and fully offline. But for contexts where you want even higher compression quality, v0.3.0 introduces hybrid mode:

`python
from llmslim import compress, CallableProvider, RewriteRequest

Plug in any LLM you already have

def my_rewriter(req: RewriteRequest) -> str:
return openai_client.chat.completions.create(
model="gpt-4o-mini",
messages=[{"role": "user", "content": req.user_prompt}]
).choices[0].message.content

provider = CallableProvider(my_rewriter)

result = compress(
context,
target_ratio=0.45,
strategy="hybrid",
provider=provider
)
`

The hybrid mode runs extractive pre-pruning first (removing obvious noise), then passes the reduced context to the LLM rewriter for semantic compression. This achieves higher information density than pure extraction while keeping the LLM call focused on a smaller input.

Benchmarks

All benchmarks are reproducible and run on N=500 prompts per dataset:

Dataset	Token Reduction	Latency	Directive Retention
System Directives	51.4%	24.8ms	100.0%
50k Long Context (GPT-5)	55.0%	26.0ms	100.0%
XML Mode (Claude 3.5)	50.0%	24.0ms	100.0%
100k Megabyte RAG (Gemini)	65.0%	38.0ms	100.0%

The 100% directive retention across all test domains is the number I'm most proud of. It's not coincidental - it's the Priority Tier system doing exactly what it was designed to do.

What I Learned Building This

A few things surprised me:

TF-IDF is surprisingly competitive with embeddings for this task. I initially assumed I'd need neural embeddings for good centrality scoring. TF-IDF vector space turned out to work remarkably well for identifying genuinely redundant prose, largely because redundant sentences tend to share vocabulary.

The two-pass budget allocator matters more than I expected. A naive single-pass selection consistently underperforms because it doesn't account for how information clusters within documents. The second pass global rebalancing step is responsible for maybe 8-10% of the quality improvement over simpler approaches.

Instruction fidelity is a hard constraint, not a soft one. Early versions of the pipeline treated all sentences equally by score. The moment I realized I needed a hard exclusion layer (not just upweighting) for directives, the results became dramatically more reliable.

Try It

`bash pip install llmslim `

Full documentation, integration guides for LangChain, LlamaIndex, FastAPI, Ollama, and more are at https://www.llmslim.app

The source is on GitHub and the benchmark methodology is fully open if you want to reproduce or challenge the numbers.

I'd genuinely love to hear from anyone running production RAG pipelines about what compression approaches you've tried and what's worked. This is still a largely unsolved problem at the edges.