TL;DR: Tested whether representing Claude's responses as bullets instead of prose loses any information. Ran it across 14 technical topics — 24-78% token savings depending on compression level, fidelity held up on 12 of 14 runs, and 2 runs (same topic, three attempts) had the local expansion model fabricate a detail. Built a Chrome extension that expands bullets back into prose locally via Ollama, so the frontier model only pays for reasoning.
GitHub: https://github.com/shouvik12/sir-shortoken
Background
Sir Shortoken is a system prompt that constrains Claude/GPT/Gemini to answer within information budgets — Quick, Balanced, Deep, Unlimited — instead of defaulting to maximum elaboration on every question.
The natural follow-up question once that was working: prose isn't the only way to represent a constrained answer. What if the response was bullets instead — stripped of connective language, just the structural content? Would that hold up, or would compressing the format also compress out information that mattered?
The hypothesis
Bullets and prose aren't necessarily lossy relative to each other — they're two different representations of the same underlying content. Prose adds transitions, connective phrasing, and narrative flow. Bullets drop that scaffolding and keep the structure.
If that's true, there's an interesting architecture hiding in it: the expensive model (Claude) only has to pay tokens for the reasoning — getting the facts and structure right. The rendering — turning that structure into readable prose — doesn't need frontier-level intelligence. A much smaller local model should be able to do that part for close to free.
The extension
Built a small Chrome extension that sits on Claude.ai. When Sir Shortoken responds in bullets, an "Expand to Prose" button appears under that message.
Clicking it sends the bullet content to a local model — I used qwen2.5:7b through Ollama, running entirely on-machine, no API calls — which rewrites the bullets into normal prose in roughly 10-15 seconds.
The button is attached to that specific message in the DOM, not to the current view. Scroll away, come back to an earlier message in a long conversation, the button is still there and still expands that same response. You're not committed to reading an entire conversation in bullets just because one message used that format.
Two implementation details worth calling out for anyone trying to reproduce this:
CORS: Ollama needs to be started with OLLAMA_ORIGINS="chrome-extension://*" ollama serve or the extension gets a 403 on every request.
Extraction: The content script has to be careful about what text it sends to the local model. Early versions grabbed the entire message DOM node's innerText, which included Claude's reasoning-summary line and the Sir Shortoken Ledger block — both ended up getting narrated into the expanded prose ("Claude responded with 'OAuth 2.'...") instead of just expanding the actual bullet content. Fixed by stripping everything from Sir Shortoken's Ledger onward and everything before the first real bullet line.
The 14-run test
Ran the same pipeline — bullets out of Sir Shortoken, then expansion via the local model — across 14 different technical topics, at two compression levels (Standard and Aggressive). Measured token counts at each stage and read every expansion manually to check whether anything was dropped or invented.
Topic Compression Prose Bullets Savings Fidelity
StatefulSet Standard 363 277 24% Clean
Git merge/rebase Standard 715 215 70% Clean
Rate limiting Standard 363 120 67% Clean
CSS specificity Standard 246 148 40% Clean
JWT auth Standard 349 202 42% Clean
ACID transactions Standard 519 251 52% Clean
Load balancing Standard 602 374 38% Minor drop
DNS resolution Aggressive 463 104 78% Clean
Raft leader election Aggressive 502 122 76% Clean
TCP handshake Standard 233 108 54% Minor drop
HTTP/2 mux (1st) Aggressive 323 74 77% Fabrication
Virtual memory/paging Aggressive 594 179 70% Clean
HTTP/2 mux (2nd) Aggressive 324 91 72% Clean
HTTP/2 mux (3rd) Aggressive 324 91 72% Fabrication
Savings: ranged 24-78% depending on compression mode. Standard compression averaged roughly 48%; Aggressive averaged closer to 75%.
Fidelity: 10 of 14 runs expanded cleanly, with no facts dropped and nothing invented. 2 runs (load balancing, TCP handshake) had minor drops — a term omitted here or there, nothing structurally wrong. 2 runs fabricated a detail that wasn't present in the source bullets.
The fabrication finding
Both fabrication cases were the same topic — HTTP/2 multiplexing — run as three separate attempts. It failed on the 1st and 3rd attempts, and expanded clean on the 2nd.
That's not evenly distributed across the dataset. Every other topic tested, including some fairly dense ones (Raft leader election, virtual memory paging), expanded cleanly every time it was run. HTTP/2 multiplexing failed 2 out of 3 tries on the same content.
That pattern — concentrated on one topic rather than spread randomly across the dataset — suggests this isn't just noise you average away. It's more consistent with the local model having weaker or more ambiguous training signal on that specific topic, and filling gaps with something plausible-sounding rather than sticking strictly to what was in the bullets.
I don't have a confirmed root cause yet. It could be topic-specific (something about how HTTP/2 internals are represented in the smaller model's training data), or it could be an artifact of how compressed the Aggressive-mode bullets were for that particular topic. Worth noting all four fidelity issues (2 minor, 2 fabrication) occurred in runs using either Standard compression on a dense topic or Aggressive compression generally — none occurred on a "medium density, standard compression" combination, though the sample size is too small to treat that as more than an observation.
What this means practically
The architecture works: token savings are real, and most content expands faithfully. But "trust the local expansion blindly" isn't a claim the data supports yet. For anything where getting a fact wrong actually matters, treat the expanded prose as a first draft on unfamiliar topics rather than a guaranteed-faithful rendering — especially if the source bullets were generated at Aggressive compression.
The 60-78% number is the one that's tempting to lead with, but the more useful finding is that fabrication risk appears to be topic-correlated rather than randomly distributed. That's the thing worth verifying if anyone reproduces this — does it hold up the same way with a different local model, or is qwen2.5:7b specifically weaker on networking-protocol internals?
Try it
/balanced /bullets Explain OAuth 2.0
Repo, skill.md, and the extension are all up: https://github.com/shouvik12/sir-shortoken
Genuinely curious whether anyone else sees the same topic-specific fabrication pattern, or if it's particular to this setup.
Top comments (0)