ContextPack: shrink your prompts, hand off your chats, keep the meaning

This is a submission for the Hacktoberfest Open-Source AI Challenge Week 2: Side Hustle

What I Built

ContextPack shrinks the wordy parts of an LLM prompt, and it refuses to ship a shorter prompt that means something different. Also, it helps in context handoff to the next LLM the developer is going to use rather than giving it the whole context again , manually.

That second half is the whole point. Most prompt compressors chase the biggest saving. But a prompt that loses one "not" is worse than a long one. If you tell a model "do NOT delete any files" and the compressed version quietly drops the NOT, you saved ten tokens and made the prompt dangerous. ContextPack is built the other way around: it would rather save nothing than change your meaning.

Who would pay for it: teams whose LLM bill is mostly input tokens. A support bot that wraps every customer email in a long instruction block. A copilot where users type long, polite, rambling requests. An agency with bloated prompt templates. As a made-up example, not my measurement: if a team spends $2,000 a month on input tokens and cuts 30% of them, that is $600 a month back. The side-hustle idea is a pay-per-request API with a free tier.

There is a second tool in the same app, Context Handoff. You paste a long chat or bug log, and it gives you a clean summary (stack, decisions, constraints, failed approaches, next step) that you can paste into a new model, so you don't have to re-explain a whole project every time you switch.

What it saves, measured on my own small set of prompts (not a benchmark):

• Wordy, polite prompts: about 45% with the AI rewrite (94 to 52 tokens in the demo below), and about 35% with the rule engine alone.
• Prompts with protected data (customer emails, IDs, numbers): about 22% with the AI rewrite, 6.7% with rules alone.
• Raw, unfenced code: left untouched.

Tight, dense prompts barely shrink, and the tool tells you that instead of pretending.

Demo

Live app: https://contextpack-sandy.vercel.app/

Every example below is a button in the app. Click one, press Compress, and you get a result in about a second, and a better one a few seconds later if the AI step succeeds.

Hybrid mode: 94 to 52 tokens (44.7%) in 688 ms on this run. NOT, "reports" and 50 survive.

Before: "Hello! I would like you to please help me write a Python script. Could you kindly make sure that the script does NOT delete any files, and that it only reads files from the folder "reports". I want you to process at most 50 files at a time. Basically, the script should really just print the total number of lines in each file. Thank you so much, I really appreciate your help!"

After: "Write a Python script. The script does NOT delete any files, and only reads files from the folder "reports". Process at most 50 files at a time. The script should print the total number of lines in each file."

The app also lists every word it removed, so you can check the work yourself.

105 to 82 tokens (21.9%). The <data> block, JIRA-4521, 120 and both "Do not" sentences are exactly as I typed them.

This is the case I care about most. Wrap customer text in <data> tags and it is never touched, because you should never let a compressor rewrite what your customer wrote.

Context Handoff on a short chat. (The heading still says "ContextSwitch" because that is what this project used to be called.)

Code

aaryanideas28 / Hacktoberfest-DevWeek-2

Made for Vibe Coding / Dev Community

📦 ContextPack

Compress your prompts. Carry your context.

ContextPack is an API-first service built with FastAPI, featuring a web dashboard to streamline developer workflows through two core capabilities:

1. Prompt Compression — Shortens prompts while preserving critical information, including instructions, numbers, negations, quoted strings, URLs, and code.
2. Context Handoff — Transforms long, unstructured chat transcripts into compact, structured Markdown artifacts that can be transferred to a new conversation without losing essential context.

Both engines follow a strict zero-trust principle: model-generated outputs are never inherently trusted. Outputs are validated against the original input, and the service falls back to deterministic rules whenever validation fails.

---

🗺️ System Architecture

flowchart TD
    UI["Web UI<br/>app/static"] --> API["FastAPI<br/>main.py"]
    API --> COMP["POST /api/v1/compress"]
    API --> HAND["POST /api/v1/handoff"]
    COMP --> CP["compressor.py<br/>Rule-based pre-pass"]
    CP --> REWRITE["LLM Rewrite<br/>Generative pass"]
    REWRITE --> VALIDATE["protect.py<br/>Validate output"]
    VALIDATE --> FALLBACK["Fallback Layer"]

    HAND --> EXTRACT["extractor.py<br/>Rule-based extraction"]
    EXTRACT --> GEMMA["gemma.py<br/>Optional LLM extraction"]
    GEMMA --> VERIFY["Verify against

…

View on GitHub

The heart of the project is one small gate. Every rewrite from a model has to pass it before the user ever sees it (simplified):

How I Built It

Two ways to build this, and why I picked one

Before writing any code, I sketched two completely different designs. I'd read a lot about prompt compression, and these were the two roads that kept coming up.

Architecture A: let a model rewrite the prompt.

You send the prompt to an LLM with an instruction like "say this in fewer words, keep the meaning." The model hands back a tidy, readable, shorter prompt. It's flexible, it understands context, and you don't need any special hardware, because someone else's server does the thinking.

Architecture B: score every token and delete the boring ones.

This is the LLMLingua idea. You run a small open model (I used Gemma 2 2B) over your prompt once. It gives you a "surprise score" for every token. A token the model could have guessed anyway carries little information, so you drop the most predictable ones. There is no text generation at all, just one forward pass. On a GPU, with something like vLLM or Hugging Face serving the model, that should be fast, and it can cut a prompt aggressively. It also never invents new words, because it only deletes.

	A: rewrite with a model	B: token scoring on a GPU
Speed	Waits on a model call (seconds, sometimes slow)	One forward pass, should be much faster on a GPU
Output	Clean sentences a human can read	Choppy, with words removed
Risk	The model can reword or drop things	Only deletes, but "predictable" is not "unimportant"
Needs	An API key	A GPU (or lots of RAM), and access to the weights
Compression	Moderate	Can be aggressive
Privacy	Your prompt goes to an API	Can run fully on your own machine

I went after B first, because lower latency is a real selling point. I built the whole thing: loading Gemma 2 2B, getting a surprise score per token, merging tokens into words, and deleting the lowest scorers. It's in the repo as gemma_entropy.py.

Then reality arrived. The Gemma weights are gated, so my first run died with a 401 error. After I sorted that out, a laptop CPU would need roughly 10 GB of RAM and was far too slow, because B really wants a GPU. And a free serverless host like Vercel can't run a GPU model at all. So B would have been a demo that only worked on a machine I didn't have.

I shipped A, and kept B in the repo as a safe fallback path. I didn't get to benchmark B on a GPU, so its speed advantage is what I expect from the design, not something I measured.

The lesson B taught me

Building B taught me the most important thing in this project: "predictable" does not mean "unimportant." The word "not" after "do" is almost perfectly predictable, and it flips the whole meaning of the sentence. A score-and-delete system would happily drop it.

That's why protection comes first in everything I built, including A. Negations, numbers, quotes, URLs, code and <data> blocks are hidden behind placeholders before any rule touches them, and they have to come out the other side exactly as they went in.

How the shipped version (Architecture A) works

Browser
  |-- 1) instant:  POST /compress  mode=rule    --> shows a result in milliseconds
  '-- 2) upgrade:  POST /compress  mode=hybrid  --> swaps in a better one if it is safe

FastAPI backend
  rules  -->  AI rewrite  -->  safety gate  -->  result
                  |                 |
        (slow / rate limited)   (anything doubtful)
                  '---> rule-only result ---'

1. Rules first. A rule engine strips filler ("I would like you to", "thank you so much", chat slang like "can u pls"). It runs while the protected pieces are hidden.
2. Then the model rewrites the prose. The prompt to the model is strict: shorten it, copy numbers, quotes and negations exactly, don't answer the prompt, output only the rewrite. Small prompts go in whole. Big prompts never go whole (see below).
3. The safety gate checks the answer. It must be a real string, it must be shorter, every protected piece must be there exactly as written (capital letters included), and it must not have dropped instruction words like only, must, detail, polite, step, under. The gate has no idea which model produced the text, and it doesn't need to.
4. Anything doubtful falls back. If the model is slow, rate limited, or fails the gate, the user keeps the rule-only result. The original prompt is the very last resort.

I also keep a list of every word removed and show it in the UI, so nobody has to take my word for it.

Things that went wrong (and what I changed)

• A 21,000-token prompt took 226 seconds, and saved 4 tokens. I was sending the entire thing to the model in one call, and most of it was code, which I protect on purpose. The model returned something the gate rejected, so I waited almost four minutes for nothing. Now code and <data> blocks never leave my server, only the largest prose paragraphs go out, and the whole AI step has a hard 8-second limit. The same prompt now answers in about 9 seconds.
• Models lowercased NOT, dropped digits, or answered the prompt instead of shortening it. That is exactly why the gate exists.
• Free API tiers rate-limit. So the instant rule result always shows first, and the UI never freezes on a blank screen.

How I trust it: 3,677 offline tests. They run the pipeline against fake "models" that misbehave on purpose (lowercasing everything, stripping negations, returning empty text, adding chatty preambles, crashing). The pipeline has to keep every protected token in all of those cases.

Stack: FastAPI, SQLite, plain JavaScript, Sentry, deployed on Vercel.

Honest limits: raw code that isn't in a fenced block is passed through unchanged, so wrap code in fences. Tight system prompts shrink only a little. The AI step runs on free tiers and short serverless time limits, so it can be skipped. Handoff history needs a real database to persist on Vercel.

What I'd build next: A and B together

The two designs don't have to compete. The next version would run B first, on a small GPU box with vLLM: one fast pass to cut the most predictable words, with protected tokens locked. Then A would clean up the result, and the same safety gate would check it. B gives speed and aggressive cutting, A gives readable sentences, and the gate keeps both honest.