Our daily scout surfaces AI-tooling repos every morning. Most of them you should not adopt — and the engineering is in deciding which parts, if any, are actually load-bearing for your problem. Here's one worked example.
A context-compression project — Headroom (headroomlabs-ai/headroom, Apache-2.0) — made the rounds with a great pitch: cut 60–95% of your LLM context tokens with "zero degradation." For an agent that streams huge tool logs into a model, that's the dream. The reflex it triggers in most teams: vendor the whole thing.
We did the slower thing first. We opened the source.
The README names and the real names disagree
Headroom's README sells three components — CodeCompressor, SmartCrusher, CacheAligner. Open src/ and the actual classes are CodeStructureHandler, JSONStructureHandler, and AnthropicCacheOptimizer + PrefixCacheTracker. Not a scandal — just a reminder that the README is marketing copy and the source is the spec. The real design is a clean three-stage pipeline: detect the content type (Magika), run a type-specific handler to extract a structural skeleton, and only then lossy-compress whatever's left.
The parts worth understanding:
-
Code (
handlers/code_handler.py): tree-sitter parses the file (regex fallback) and keeps imports, signatures, and class declarations while dropping function bodies, comments, and whitespace. "Keep the API surface, drop the implementation." Works for Python/JS/TS/Go/Rust/Java/Perl. -
JSON (
handlers/json_handler.py): a per-token heuristic — keep keys, syntax, booleans, short values, and high-entropy strings (entropy ≥ 0.85); compress long strings; keep the first N array items in full. No schema, no dedup. -
Cache alignment (
prefix_tracker.py): one genuinely sharp invariant — freeze the already-cached prefix, only compress new content, so your compression never rewrites the prefix and silently invalidates the provider's KV cache.
The benchmark, read adversarially
This is where reading beats retweeting. The headline is "60–95% savings, zero degradation," measured with the standard lm-eval harness on a real model. Credit where due: the repo even ships its own worst-case and adversarial benchmarks and lists its failure modes — an honest signal most projects skip.
But line up the claim with what was actually measured:
- The zero-degradation numbers come from GSM8K / TruthfulQA / SQuAD — short-answer tasks with almost no context to compress. "Zero degradation" there is trivially true; it can't support the 60–95% claim.
- The runs that do compress heavily — agent traces — are n=5, scored by loose keyword matching.
- It's all self-reported, with no third-party reproduction. One external review of prompt-heavy chat measured 20–40% savings, not 60–95%.
None of that makes Headroom bad. It makes the headline number a claim, not a result — and the difference matters when you're deciding what to build on.
What we borrowed — and what we left on the shelf
We took three ideas, each re-implemented to fit our own stack, not the package:
- tree-sitter "keep signatures, drop bodies" for code blobs — a real semantic upgrade over blunt truncation.
- The JSON keep-keys / high-entropy heuristic — cheap, deterministic, no model required.
- Cache-aware compression — the "will this rewrite a cached prefix?" check, which is the highest-ROI idea in the whole repo and the one nobody talks about.
We explicitly declined:
- The reversible-retrieve loop (CCR) — it carries an extra store plus a retrieve round-trip to recover what compression dropped, costs hundreds of MB of RAM, and is already known to break under streaming (issue #1450). If your compression is allowed to be lossy, you don't need it.
- The ML text model (Kompress) — small, English-biased, and when it's unavailable the whole text path silently falls back to first-2/3 + last-1/3 truncation. A dependency that decays into the thing it replaced.
- The proxy / MCP server / cross-agent "learn loop" — orthogonal feature creep, not a compression upgrade.
The actual lesson
A viral repo is an input to a decision, not the decision. Read the source to find the two or three load-bearing ideas, check the benchmark against what it actually measured, and write down — on the record — what you didn't take and why. That discipline (read the code, not the README; borrow only what nets out positive) is exactly what we encode as a standard.
The standards we use to run these evaluations are MIT / CC-BY and open.
→ github.com/AsiaOstrich/universal-dev-standards
Top comments (0)