DEV Community

Devanshu Biswas
Devanshu Biswas

Posted on

Prompt Compression: Drop the Filler, Keep the Numbers, Negations and Entities

#ai #llm #rag #machinelearning

Most of your prompt is filler. A long system prompt, a stack of retrieved chunks, a block of few-shot examples — a large fraction of those tokens carry almost no information: articles, connectives, hedges, boilerplate. You pay for every one of them on every call, and prefill latency grows with prompt length. Prompt compression, the idea behind Microsoft's LLMLingua family, shrinks the text by scoring each token's information value and dropping the lowest until it hits a target ratio — keeping the meaning intact.

Surprise equals information

A token's value is its self-information, −log₂ p(token): how surprising it is. A rare token — an error code, a name, a number — is high-information because the model can't guess it back. A stopword like "the" is almost fully predictable from context, so its information is near zero and it's cheap to drop. LLMLingua measures this with a small language model's perplexity; the deterministic stand-in below uses inverse frequency.

freq = Counter(t.lower().strip(".,") for t in toks)
N = len(toks)
STOP = {"a", "an", "the", "of", "to", "and", "or", "is", "for", "on", "in", "you", "it"}

def info(t):
    key = t.lower().strip(".,")
    surprisal = -math.log2(freq[key] / N)   # rare -> high, common -> low
    if key in STOP:  surprisal *= 0.12       # filler the model can guess back
    return surprisal
Enter fullscreen mode Exit fullscreen mode

The one rule that keeps it safe

Low information is not the same as safe to remove. A lone "not" is common (low surprisal) but load-bearing — drop it and "Do not issue a refund" becomes "Do issue a refund". Three classes are pinned regardless of score: numbers, negations, and named entities. Lose a dollar amount, a "never", or an entity and the instruction silently flips.

NEG = {"not", "no", "never", "without", "none", "neither", "nor", "cannot"}

def protected(t):
    key = t.lower().strip(".,")
    is_number   = bool(re.search(r"\d", key))                # 250, $250, INV-4471
    is_negation = key in NEG                                 # not / never / without
    is_entity   = bool(re.match(r"[A-Z]", t)) and t.lower() not in FIRST_WORDS
    return is_number or is_negation or is_entity
Enter fullscreen mode Exit fullscreen mode

Prune to a budget

The compression ratio r is a budget, not a fixed threshold. Give protected tokens infinite value so they always survive, then drop the round(r·N) lowest-value non-protected tokens. Order is preserved, so you keep a readable subset rather than a reordering.

def compress(toks, r=0.5, protect=True):
    def value(t):
        if protect and protected(t): return float("inf")   # pinned
        return info(t)
    n_remove = round(r * len(toks))
    order = sorted(range(len(toks)), key=lambda i: (value(toks[i]), i))
    drop = {i for i in order[:n_remove] if value(toks[i]) != float("inf")}
    return " ".join(t for i, t in enumerate(toks) if i not in drop)
Enter fullscreen mode Exit fullscreen mode

Then prove nothing critical was lost — diff the protected set against the survivors, and if any dropped, the compression is unsafe at that ratio: raise the budget or lower r.

Why not just truncate — or summarize?

The lazy way to hit a budget is to cut the end (or middle) of the prompt. It's free and it's blind: it amputates whatever lives there, and prompts often put their hardest constraints last ("Never share the card number", "Under GDPR..."). Salience pruning removes low-value tokens from everywhere, so the guardrails survive.

Compression is also not summarization. Summarization is abstractive — a second LLM call that costs tokens, adds latency, and can hallucinate. Compression is extractive: it only ever deletes existing tokens, so it's cheap, deterministic, and can't invent facts. The compressed text reads like a telegram, and LLMs handle it fine. Summarize for humans; compress for the model.

In production you rarely hand-roll the scorer. LLMLingua and LLMLingua-2 ship a PromptCompressor that ranks tokens with a small model, honours a target rate, and force-keeps tokens you name — routinely 2×–5× compression with answer quality nearly unchanged. Compress the big context once, cache it, then send the shorter prompt to your model. It slots in right after retrieval and alongside long-context management, before the model ever reads a token.

Drag the compression-ratio slider and watch the meaning-retained check flip when a number or negation falls: https://dev48v.infy.uk/ai/days/day58-prompt-compression.html

Top comments (0)