billion-context Hands-On: Compressing Coding-Agent History to Fit a Small Context Window

Long agent sessions keep failing the same way for me. The window fills up with test logs and file dumps, the harness runs its own auto-compaction, and the agent forgets the constraint I gave it an hour earlier. So when billion-context reached 5,427 GitHub stars about three months after the repo was created, I wanted to see what it actually does to a real session, measured in tokens and in what the agent still remembers afterwards.

This post covers what I installed, the one bug I ran into, the numbers from a real tool-calling session, and where I think the project's claims hold up and where they don't.

What billion-context is

billion-context (github.com/ranxianglei/billion-context, MIT plus an attribution clause) is a local proxy, bili, that sits between a coding agent and its model API. It works with Anthropic, OpenAI and Google-native requests, and it adds four tools to every request:

• compress: fold a range of messages into a summary
• decompress: restore a folded range
• search_context: keyword search over folded summaries
• acp_status: report how full the context is

The main difference from Claude Code-style auto-compaction is that the model decides what to fold, nudged by the proxy as the window fills. The proxy runs the fold on its side, keeps the originals in a local SQLite store, and from then on sends the folded view upstream. Summaries can be folded again (tier 2, tier 3), and the README calls this "an LSM tree over summaries."

The README makes three headline claims: "100K context is enough", "5× fewer tokens", and "month-long single sessions." The author's preprint in the repo reports 174,327 model calls over 4.5 months. A benchmark folder labeled "independent third-party" reports 79.2% vs 83.2% for full context on LongMemEval-S at 71% lower cost. Both live inside the project's own repo, so I count them as claims, not evidence.

The repo is also changing very fast: 565 commits between 2 Nov and today, and npm latest is 0.1.213. I tested that version.

Setup

Installing took about five seconds:

npm install -g billion-context --prefix=~/.local
bili --version   # 0.1.213 (133f3525c)

I had a Free Tier Gemini key and no other credentials, so I skipped the bili claude / bili codex launchers. Instead I used the universal path: start the proxy and prefix the upstream URL with /bili/. To get folds within an afternoon's budget, I pinned the window to 40K tokens:

{
  "port": 8787,
  "compress": {
    "modelContextLimit": 40000,
    "maxContextLimit": "60%",
    "nudgeGrowthTokens": 8000
  }
}

ACP_AUTO_UPDATE=0 bili start
curl -s http://127.0.0.1:8787/__bili/health

I set ACP_AUTO_UPDATE=0 on purpose. By default bili checks npm every three minutes and upgrades itself after a staggered delay. That's convenient for a fast-moving project, but I don't want it in a benchmark, and I'd think about it before putting this on a shared CI box.

The first bug: Gemini's OpenAI-compatible endpoint

My first harness used Gemini's OpenAI-compatible endpoint. The first request went through fine. Every request after a tool call failed with:

HTTP 400: Function call is missing a thought_signature in functionCall parts.

Gemini 3.x returns an opaque thought_signature with each tool call, and it has to come back in the next request. The same two-request script returned 200 when sent directly and 400 through the proxy, so the proxy is dropping that field on the chat-completions path. Switching to Gemini's native generateContent wire fixed it:

BASE = "http://127.0.0.1:8787/bili/https://generativelanguage.googleapis.com/v1beta"
r = requests.post(f"{BASE}/models/gemini-3.5-flash-lite:generateContent",
                  json=body,
                  headers={"x-goog-api-key": KEY, "x-acp-session": "s2"})

If you run Gemini 3 behind an OpenAI-style client, test multi-turn tool calls before anything else.

The test session

I wrote a small agent loop with a single bash tool and pointed it at a fresh clone of python-slugify. I scripted 11 user turns, the kind I'd actually type:

• ticket SLUG-482 for a client, Python 3.8 compatibility, no new dependencies
• run the tests, read slugify.py, explain max_length vs word_boundary
• a standup decision: keep the - separator and add max_words: int = 0
• a CLI flag, a CHANGELOG entry, a README update
• mid-session: staging port 7431, reviewer Priya Raman, Unicode refactor pushed to Q1 because of a Django adapter risk

The model was gemini-3.5-flash-lite. For the baseline, I ran Gemini's countTokens on every uncompressed request prefix, using the same system prompt and tool schema without the proxy. That gives an exact "what would I have sent" number for each of the 41 requests.

One thing about the agent itself: Flash-Lite never edited a file. It read code, ran tests, and then claimed in its final answer that it had implemented the feature. The git tree was clean. That's a model problem, not a compression problem. The fold summary was actually more accurate than the model's answer, because it listed the feature under "Open objectives."

The numbers

The session took 41 client requests and 184 seconds of wall-clock time. That includes three back-offs from Free Tier 429s (250K input tokens per minute); summed model latency was 45.9 seconds. The proxy added a median of 10 ms of local processing per request (max 26 ms over 50 requests).

• Folds: 2 events, 7 summary blocks of 111–284 characters each
• Fold 1 (turn 6): next prompt went from 27,853 to 9,396 tokens; ~14,095 tokens reclaimed
• Fold 2 (turn 10): 26,778 to 12,046 tokens; ~9,515 reclaimed
• Final request: 13,431 tokens via bili vs 38,816 uncompressed (2.9× smaller)
• Peak prompt: 27,853 vs 38,816
• Total input over the session: 663,321 vs 758,108 tokens (12.5% less)
• Fixed overhead: the first request was 4,243 tokens upstream against a 180-token uncompressed prompt, so the injected tools and instructions cost about 4,060 tokens on every request

This is the first block the model wrote:

[Compressed conversation section] — Repo Exploration
- User request (m00001): ticket SLUG-482 for client Northwind.
  Constraints: Python 3.8 compatibility, no new dependencies.
- Explored repo layout and located main `slugify()` in
  `slugify/slugify.py:126` taking 16 parameters.
- Ran pytest via `python -m pytest`: 133 passed, 0 skipped.

That's 284 characters standing in for a few thousand tokens of find output, a full pytest log, and an inspect.signature dump. The user messages themselves stayed verbatim, because the proxy protects them.

Did the agent still remember?

I asked two quizzes at the end of the session, once through the proxy (folded view) and once directly with the full history.

Easy set (10 questions about things I told it): ticket ID, client, Python version, separator decision, new kwarg, test file, first test result, port, reviewer, postponed refactor. Through bili: 10/10 on 13,821 prompt tokens. Full history: 10/10 on 39,114. Since user turns aren't folded, this is the case the design is built for, and it held up.

Hard set (8 questions about tool output): the line number of slugify(), the module marked "DO NOT MODIFY", the CLI --algorithm choices, the last commit hash, the top CHANGELOG version, the pytest/Python versions, the parameter count, and the CLI test files.

• Through bili with tools disabled: 4/8. It answered "unknown" for the details that existed only in folded tool output.
• Through bili with search_context/decompress allowed (streaming): 6/8. The model ran five search_context queries ("slugify.py", "CLI", "algorithm", "CHANGELOG", "git log") and then answered. The "algorithm" search matched zero summaries but five folded originals, so the search covers raw history too, not only the summaries.
• Full history, direct: 6/8, with different misses: it got the line number wrong (195 instead of 126) and named the wrong CLI test file.

So the folded agent matched the full-history agent, but only by paying for retrieval: six upstream rounds and 89,958 input tokens (72,497 of them cached), against one 39K call. Three practical notes. First, on the non-streaming lane the search_context output came back to my client as the response text, with no answer after it, so plan on streaming. Second, the search receipts ("Found 2 block(s) for ...") show up in the client-visible text stream, as do fold receipts. Third, in an earlier attempt the model decompressed a block to /tmp/acp-decompress-b6-*.txt and then tried to grep it with its own bash tool, so decompress-to-file assumes the agent can read local files.

What it cost

At Gemini 3.5 Flash-Lite list prices ($0.30/M input, $0.03/M cached, $2.50/M output):

• bili: 77% of input tokens were cache hits, about $0.067 for the session
• Uncompressed, assuming perfect prefix caching: about $0.039 (estimate)
• With no caching on either side: bili $0.205 vs $0.234

On a 39K-token session, compression lost on cost once caching is counted. Every fold rewrites the prompt prefix, and the proxy's own log said so: "the whole prefix rewrites and the prefix cache restarts from scratch on the next request." On top of that, the ~4K-token tool overhead is charged on every call. The savings come from staying under a window you'd otherwise overflow, and from long sessions where uncompressed history grows without limit. My session was neither, so treat 12.5% as a lower bound for a short session, not as evidence for the "5×" claim.

What I'd tell my team

Where it's useful:

• Long-running agent sessions on small-window or cheap models, where hitting the window is the real failure
• Teams that want folds they can audit and undo; bili export <session> dumps the folded view and every block summary
• Setups that already run a local proxy; median overhead was 10 ms

Where I'd hold off:

• Short tasks under ~50K tokens. With default settings (50K growth gate, nudge at 75% of the window) my session would never have folded, and the tool overhead would only have added cost.
• Gemini behind an OpenAI-compatible client until the thought_signature drop is fixed
• Anything user-facing without reading the license. It's MIT plus a required attribution line on your home page, docs, or About page.
• Pinned production environments. Hundreds of commits a week plus self-update on by default means you need ACP_AUTO_UPDATE=0 and a pinned version.

The design idea is good: let the model decide what's stale, keep the originals, and make folds reversible. In my run that held up on the questions that matter most, the constraints and decisions I gave it, at a third of the final context size. The "billions of tokens" claim is a separate test that needs a session thousands of calls long. I'll run that one with a stronger model before I put it in front of real agent traffic.

Originally published on Medium: https://medium.com/@ishank.iandroid/billion-context-hands-on-compressing-coding-agent-history-to-fit-a-small-context-window-7cdbac4bf6b8