DEV Community: Shouvik Palit

Stop explaining yourself to Claude

Shouvik Palit — Fri, 12 Jun 2026 04:26:55 +0000

You're wasting tokens. Not a little -a lot.

Here's a prompt I see constantly:

"I have a React app and I'm using the useState hook. My component re-renders every time the parent renders even though the props haven't changed. Why is this happening?"

Claude doesn't need any of that setup. It already knows React. It already knows what useState is. The only thing it needed was:

"Component re-renders on parent render. Props unchanged. Why."

Same answer. 64% fewer tokens.

The delta principle

Most prompts are written for humans. We explain context, name the framework, describe how things work before asking the question. That's how we communicate with each other.

But Claude already knows the context. The only thing it needs is the delta — the new information, the specific problem, the unknown.

Everything else is noise.

What you can safely strip

Claude already knows these — stop re-explaining them:

Framework names used as context ("I have a React app", "I'm using Python")
Concept explanations ("hooks are a React feature that...", "a closure is when...")
Stack introductions ("my app uses Node, Express, and MongoDB")

Social noise that adds zero signal:

Pleasantries: "hey", "hope you can help", "thanks in advance"
Permission requests: "could you please", "I was wondering if"
Hedging: "I think", "I'm not sure but", "maybe", "possibly"
Filler: "basically", "essentially", "just", "simply", "actually"

What you should never strip:

The actual error, bug, or problem
Numbers, thresholds, measurements
Variable names, function names, file names
Code blocks and URLs
Anything Claude could NOT know without being told

Before and after

Debugging:

Before (41 tokens):
"I'm working on a Node.js Express API and I'm getting a 401 unauthorized 
error when I try to call the endpoint. I'm passing the JWT token in the 
Authorization header."

After (12 tokens):
"401 on endpoint. JWT in Authorization header."

Code review:

Before (29 tokens):
"Could you please review this Python function and tell me if there are 
any issues or improvements I could make?"

After (6 tokens):
"Review. Issues + improvements."

Explanation:

Before (19 tokens):
"I was wondering if you could explain how database connection pooling 
works in simple terms?"

After (5 tokens):
"Explain connection pooling. Simple."

The compound effect

Single prompt savings look small. But across a real session, it compounds.

Here's a simulated 20-turn dev session — the kind where you're debugging something across multiple back-and-forths:

Turn	Verbose (tokens)	Delta (tokens)	Saved
1	48	18	30
2	35	12	23
3	52	14	38
4	29	8	21
5	41	11	30
6	33	9	24
7	44	15	29
8	38	10	28
9	31	8	23
10	45	13	32
11	27	7	20
12	39	11	28
13	50	16	34
14	36	10	26
15	42	12	30
16	28	8	20
17	46	14	32
18	33	9	24
19	40	11	29
20	37	10	27
Total	757	226	531

531 tokens saved in a single session. 70% reduction.

On Claude's API at Sonnet pricing, that's a small number in dollars. But if you're building on top of the API and running hundreds of sessions a day, it adds up fast. And even on claude.ai, fewer input tokens means less context noise — Claude processes cleaner signal and responds more precisely.

Three intensity levels

Not every prompt needs ultra-compression. I use three modes depending on the situation:

lite — strip pleasantries only, keep context (~20% reduction)

Use when: onboarding a new topic, first message in a session

full — strip everything Claude knows, keep only the delta (~60% reduction)

Use when: mid-session debugging, iterating on code, quick questions

ultra — compress to bare minimum signal (~70%+ reduction)

Use when: you know exactly what you want and don't care about polish

The skill file

I turned this into a Claude skill — a markdown file that instructs Claude to apply delta compression automatically, with activation/deactivation commands and intensity switching.

→ github.com/shouvik12/delta

The README has the full rule set, intensity examples, and instructions for adding it to your Claude setup.

One thing worth thinking about

This is a small optimization. But the principle behind it is bigger:

We've been writing prompts for humans. We explain, we hedge, we contextualize — because that's how we earn understanding from other people. With LLMs, that overhead is waste. The model doesn't need to be convinced you know what you're talking about. It doesn't need the social scaffolding.

Just send the delta.

Escalate the Model, Not the Conversation

Shouvik Palit — Wed, 10 Jun 2026 19:03:23 +0000

Trooper started as a fallback proxy for agents. Claude hits a quota, falls back to Ollama, session continues. No crashes, no lost context.

The interesting problem that came up wasn't model routing. It was context preservation.

When you're debugging something hard, you build up context over many turns. The problem statement, what you've tried, what failed. When you switch from a local model to Claude, all of that context has to go with it. And when you come back to local, the local model needs to know what Claude said.

That's what 4.0 solves.

How it works

A local model handles requests by default. Fast, free, private.

When it gets stuck, one click escalates to Claude — the full conversation history is injected automatically. Claude answers. Then control returns to the local model, which continues the conversation knowing exactly what Claude said.

No copy-pasting. No restarting the conversation. No lost context.

The escalation moment

You're debugging a slow Postgres query. Llama gives you a decent answer — check your EXPLAIN output, look for function calls on indexed columns. Good start.

Not enough. You hit Escalate.

Claude receives the full session. It knows you're debugging a slow query. It knows what Llama already told you. It picks up exactly where the conversation left off.

You click Back to local.

Now ask Llama to summarize what Claude said.

It does. Correctly. Because the session store was updated with Claude's response. Llama reads the full history including what Claude said and continues from there.

What's under the hood

Trooper is a Go proxy that sits between your client and any LLM provider. The chat UI is a static HTML file served by the same process.

When you escalate:

The UI fetches the full session history from /session/:id
Sends it to Claude via /v1/messages with X-Force-Cloud: true
Claude's response gets written back to the session store via /session/:id/append
Next local turn, Llama reads the full history including Claude's response

The SITREP panel on the right extracts intent, confidence, entities and open loops from the conversation using a rule-based classifier — no LLM call needed.

The proxy still works

The proxy layer is unchanged. Agents, SDK clients, curl — everything routes through /v1/messages the same way it always did.

# Agent flow — unchanged
export ANTHROPIC_BASE_URL=http://localhost:3000
export OPENAI_BASE_URL=http://localhost:3000

# Chat UI — new in 4.0
open http://localhost:3000/chat

Try it

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...  # optional
export OLLAMA_MODEL=llama3.1:8b
go run .
open http://localhost:3000/chat

Works without a Claude key-escalation falls back to Ollama. Add the key when you want real cloud escalation.

https://github.com/shouvik12/trooper

How I Cut Agent Token Usage by 89% Without Touching the Agent

Shouvik Palit — Fri, 05 Jun 2026 13:10:22 +0000

Every time your agent calls an LLM, it sends the full conversation history.

Turn 20 includes turns 1–19. Turn 50 includes turns 1–49. Nobody notices because it's happening inside the agent, silently, on every single request.

I noticed it while building Trooper - a Go proxy that sits between agents and LLMs. I was watching token counts climb across a long debugging session and realised the agent was replaying the same context over and over. Most of it was noise.

The model didn't need a transcript. It needed state.

What state actually means

After a few turns, most of what matters in a session falls into four categories:

Decisions made — what was chosen and why
Constraints locked — what cannot change
Open loops — what still needs to be resolved
Ruled out — what was tried and rejected

That's it. Everything else — the back and forth, the verbose LLM responses explaining things, the repeated context — is replay. The model doesn't need it again.

The SITREP

I added structured session memory to Trooper. After enough turns, Trooper's local Llama model generates a SITREP — a situation report — from the user messages in the session.

It looks like this:

INTENT: Build a RAG pipeline with ChromaDB and nomic-embed-text

DECISIONS: Use cosine similarity over MMR — focused queries not broad;
           Chunk size 256, overlap 30 — locked;
           Pure vector search — ChromaDB no hybrid support;
           Top k set to 5

CONSTRAINTS: Node 18 locked — platform team constraint, no exceptions;
             Re-ranking ruled out — latency jumped 200ms to 800ms

OPEN: Poor recall on technical queries — nomic-embed-text struggles with domain jargon;
      Evaluating bge-small as alternative

From that point forward, every request to the LLM sends:

Anchor (first 2 turns verbatim)
+ SITREP (structured state)
+ Tail (last N turns verbatim)

Instead of the full history.

The numbers

From a real 15-turn session:

Full history:    10,820 tokens per request
With Trooper:     1,157 tokens per request
Reduction:             89%

Visible live on the dashboard.

Does the LLM still answer correctly?

This was the question that mattered. Token savings are worthless if the model loses coherence.

To test it: I took the auto-generated SITREP, opened a completely fresh chat with no history, and asked questions about decisions made in the original session.

Questions:

What is the chunk size?
Why did we rule out hybrid search?
What retrieval method did we choose and why?
What is still open?

Result: All four answered correctly. The model worked entirely from the SITREP. No history. No context bleed.

That's the claim: structured state is sufficient for the model to continue reasoning correctly — and it costs 89% less to send.

How it works

Trooper is a Go proxy — one binary, no SDK, no instrumentation. You point your existing agent at it by changing one URL.

# Before
export ANTHROPIC_BASE_URL=https://api.anthropic.com

# After
export ANTHROPIC_BASE_URL=http://localhost:3000

Nothing else changes. Trooper intercepts every request, maintains session state, and when the SITREP is ready, rewrites the messages array before forwarding to the LLM.

The SITREP is built by a local Llama 3.1 8b model running via Ollama — fast, private, no cloud cost. The extraction happens asynchronously in the background. The main request path is not blocked.

// GetTripleAnchor assembles what gets sent to the LLM
func (s *SessionStore) GetTripleAnchor(sessionID string) []map[string]string {
    payload := append([]map[string]string{}, state.Anchor...)
    if state.SITREP != "" {
        payload = append(payload, map[string]string{
            "role":    "system",
            "content": fmt.Sprintf("[STATE_SITREP: %s]", state.SITREP),
        })
    }
    return append(payload, state.Tail...)
}

The dashboard shows the compression ratio live:

HISTORY COMPRESSED    89%
TOKENS SAVED          459
CONFIDENCE            100%

Why this is different from conversation summarisation

Most summarisation tools compress what was said. The SITREP extracts what matters for the next action.

Copilot's context compaction summarises the full conversation — useful for humans in long chats. The SITREP is structured specifically for agents: decisions, constraints, open loops, ruled-out paths. Not a narrative summary. A state snapshot.

The result is that subsequent turns stay coherent on intent without replaying noise. More relevant for agents running repeated structured workflows than for general chat.

The limitation

The SITREP works best for structured agentic workflows — debugging sessions, research pipelines, multi-step build tasks. For open-ended creative work where tangential context might become important later, you'd want a larger tail window or higher fidelity compression.

The tail window is configurable. You can keep more raw context for less structured sessions.

What else Trooper does

The compression is the latest addition. Trooper also:

Falls back to local Ollama when cloud quota hits — context preserved across the switch
Routes simple turns to Ollama automatically — cloud never contacted
Privacy routing — sensitive requests stay local via x_force_local
Live dashboard — intent, open loops, completed steps, transcript
Subagent recovery — /recovery/{session_id} tells you exactly where to resume

All from one URL change.

The bigger question

We tend to treat conversation history as memory. But a transcript is a log. Memory is state.

Humans don't replay every prior conversation before making a decision. They carry forward conclusions, constraints, unresolved questions, and relevant context — a structured snapshot, not a full transcript.

Long-running agents may need to do the same. Not because of token costs — though that helps — but because state is a better abstraction for agent memory than history.

The SITREP is an experiment in that direction.

github.com/shouvik12/trooper — Go, MIT, zero dependencies beyond Ollama.

I Added a Live Dashboard to My LLM Proxy. Zero Instrumentation. Just a URL Change.

Shouvik Palit — Thu, 28 May 2026 15:24:53 +0000

I built Trooper as a fallback proxy. Claude hits quota → falls back to Ollama. Useful but passive. It sat in the background, invisible, doing its job silently.

Today it became something different.

The Original Problem

When you're building with LLMs, quota hits are inevitable. Claude's free tier is generous until it isn't. A mid-session 429 kills your context, your workflow, your train of thought.

Trooper solved that. Point your app at http://localhost:3000 instead of the Claude API. When Claude fails, Trooper catches it, preserves the full session context via a 3-layer compaction system (Anchor + SITREP + Tail), and continues on local Ollama. Your app never knows anything happened.

That's still there. But it was passive. Useful when things broke. Invisible when they didn't.

The Problem With Passive

Passive infrastructure has an adoption problem. Developers install it, forget about it, and only notice it when something breaks. That's not a product — that's a safety net.

The question I kept asking: what does Trooper do that has daily value, not just failure value?

The answer was sitting in the code the whole time.

What Was Already There

Trooper captures every message in every session. It runs a classifier on each one — extracting intent, entities, open loops, completed steps, recent actions. All rule-based, zero LLM calls, zero latency.

This is what powers the fallback context preservation. When Claude fails and Ollama picks up, Ollama doesn't start blind — it receives a SITREP (Situation Report) that tells it what the session was about, what was completed, what's still pending.

That data exists for every session. It was just never visible to the developer.

The Dashboard

I added a live dashboard at localhost:3000/dashboard.

Point any agent at Trooper — just change your base URL:

export ANTHROPIC_BASE_URL=http://localhost:3000
# or
export OPENAI_BASE_URL=http://localhost:3000

Open the dashboard. Keep it on a second monitor while your agent runs.

From a single message, it already shows:

Intent — what your agent is trying to do, extracted automatically
Open Loops — what it's stuck on, highlighted in red
Completed Steps — what it finished, tracked as it happens
Entities — the key things being referenced
Session Transcript — every message, colour coded by role

Auto-refreshes every 5 seconds. No page reload needed.

What It Looks Like In Practice

I ran a 3-turn agent session simulating a database debugging workflow:

Turn 1: "I am building a Go API server. The database connection is failing with connection refused errors on port 5432."

Turn 2: "Checked the config. Postgres is running on port 5433 not 5432. Fixing the connection string now."

Turn 3: "Fixed the port. Database connection is working. API server is running successfully."

After Turn 1, the dashboard immediately showed:

Intent: "building a go api server. the database connection is failing with connection refused errors on port" (100% confidence)
Entities: Postgres, network
Open Loops: "fail with connection refused error on port"

After Turn 3:

Completed Steps: "successfully fixed the port"
Open loops cleared

Zero instrumentation. No SDK. No code changes to the agent. Just a URL change.

Why This Matters

Every observability tool requires you to instrument your code:

LangSmith — wrap your agent in LangChain
Langfuse — add their SDK
AgentOps — add @observe decorators

Trooper requires nothing. Your agent already communicates over HTTP to an LLM. Trooper sits in that path and observes everything automatically.

Helicone was the closest — proxy-based, zero instrumentation. But it went into maintenance mode in March 2026 and was cloud-only. Your data went to their servers.

Trooper is open source, local-first, and free forever. Your data never leaves your machine.

The Sessions Endpoint

Not sure which session to look at? Hit /sessions:

curl http://localhost:3000/sessions
# {"sessions":["agent-debug-123","agent-debug-456"],"count":2}

Click any session in the dashboard home page at localhost:3000/dashboard to open it.

The Recovery Endpoint

Still there. When an agent fails mid-task:

curl http://localhost:3000/recovery/{session_id}

Returns exactly what completed and where to resume. The dashboard makes this visual — completed steps are tracked in real time.

The Pivot

Trooper started as: "Claude failed. Trooper caught it."

Trooper is now: "Your agent communicates over HTTP to an LLM. Trooper can observe it."

The fallback is a feature. Observability is the product.

Your agent was always talking. Now you can hear it.

Get Started

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...
go run .

Open http://localhost:3000/dashboard in your browser.

Point your agent at Trooper:

export ANTHROPIC_BASE_URL=http://localhost:3000

That's it. Zero dependencies. Pure Go. Runs in under 60 seconds.

GitHub: github.com/shouvik12/trooper

Tags: llm, agents, observability, ollama, go, opensource

I Added a /recovery Endpoint to My LLM Proxy So Agents Never Lose Progress Mid-Task

Shouvik Palit — Sun, 24 May 2026 17:28:55 +0000

Most LLM proxies handle failures the same way -retry the request, fall back to another provider, or crash. None of them ask the more important question: what did the agent already complete before it failed?

That's the gap I built Trooper to fill.

The Problem

If you're running multi-agent workflows, you've probably hit this scenario:

A subagent starts a long task-reviewing PRs, processing documents, running analysis
It completes steps 1, 2, and 3
On step 4 it hits a quota error, rate limit, or provider failure
Your orchestration layer has no idea what completed
You restart from scratch and repeat all the work

Most proxies handle the failure. Nobody handles the recovery.

What Trooper Does Differently

Trooper is a Go-based LLM proxy that sits between your agents and your LLM providers. It already handled fallback routing -if Claude hits quota, it falls back to local Ollama automatically.

But the new /recovery/{session_id} endpoint goes further. It tracks every step your agent completes in real time and tells your orchestration layer exactly where to resume.

The Recovery Endpoint

When your agent sends requests through Trooper, it captures every assistant response and extracts completed steps as they happen. When something fails, you call:

GET http://localhost:3000/recovery/{session_id}

And Trooper returns:

{
  "session_id": "subagent-demo-1779630533",
  "completed_steps": [
    "completed pr #1",
    "completed pr #2",
    "completed pr #3"
  ],
  "resume_from": 4,
  "recovery_hint": "Resume from step 4"
}

Your parent agent now knows exactly what the subagent finished and where to restart it. No repeated work. No lost progress.

Demo

An agent reviewing 8 pull requests hits quota on PR #4. Trooper intercepts, returns the recovery payload, and the agent resumes from PR #4 using local Ollama.

[https://www.youtube.com/watch?v=NN2uwQZDCck]

How It Works

Trooper uses a two-tier memory system:

Anchor — the first two turns of a session, always preserved verbatim.

Tail — the most recent turns, stored in a rolling window.

When you call /recovery, Trooper scans all stored assistant messages for completion signals — words like "completed", "finished", "done", "merged", "deployed". It extracts one completed step per message, deduplicates by task identifier, and returns the ordered list.

The resume_from field is simply len(completed_steps) + 1 — telling your orchestration layer which step to restart the subagent on.

How to Use It

1. Start Trooper

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=your-key
go run .

2. Point your agent at Trooper instead of Claude

# Instead of https://api.anthropic.com/v1/messages
POST http://localhost:3000/v1/chat/completions

3. Pass a session ID with each request

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "X-Session-ID: my-agent-session-1" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":100,"messages":[...]}'

4. Call recovery when something fails

curl http://localhost:3000/recovery/my-agent-session-1

What's Next

The recovery endpoint is the foundation for proper subagentic orchestration. Upcoming work:

Parent agent integration — the recovery payload feeds directly back into the orchestration layer to automatically restart subagents from the right step
Structured step tracking — support for agents that emit structured JSON progress instead of natural language
Session replay — rewind any session to any point and branch from there

The Bigger Picture

As agent workflows get longer and more complex, failure recovery becomes a first-class concern. Trooper's approach — track everything as it happens, make recovery queryable — is a different philosophy from retry-and-hope.

Local-first by default. Cloud when you choose. And now recoverable when things go wrong.

GitHub: https://github.com/shouvik12/trooper

How are you handling agent failures in your workflows today? Drop a comment genuinely curious what patterns people are using.

I Tested Privacy-Aware Routing with 4 AI Agents: What Actually Stayed Local

Shouvik Palit — Wed, 13 May 2026 02:40:52 +0000

Following up on my earlier Trooper experiments, I wanted to see if per-request privacy routing actually works in practice.

The test: 4 agents running simultaneously. Some handling public knowledge (OAuth security, Redis vs Memcached). Others handling sensitive data (API keys, customer PII).

The rule: Credentials and PII stay on my machine. Everything else can use Claude.

The Setup

Each agent gets a x_force_local flag:

Agent 1 - security-analyst (☁️ Claude)

Task: "What are the top 3 OAuth2 vulnerabilities?"  
Routing: Public knowledge, let Claude handle it

Agent 2 - credential-formatter (🔒 Qwen local)

Task: "Format as JSON: api_key=sk-prod-x7f9k2m, vault_url=https://vault.acme.io:8200"  
Routing: Contains credentials — must stay on machine

Agent 3 - architecture-advisor (☁️ Claude)

Task: "Redis or Memcached for session storage?"  
Routing: General best practices, use cloud

Agent 4 - compliance-reporter (🔒 Qwen local)

`Task: "Summarize: 47 tickets today. 3 had PII (Alice Johnson, Bob Chen, Maria Garcia)"  
Routing: Contains customer names — privacy violation if sent to cloud`

The Result

Every agent completed successfully:

Cloud agents: 3.8s and 2.4s (Claude handled complex reasoning)
Local agents: 2.4s and 1.2s (Qwen formatted data locally)

The critical part: API keys, vault URLs, and customer names never left my machine. Zero network calls to Anthropic for those two agents.

What Happened Under the Hood

When Agent 2 (credential-formatter) ran with x_force_local: true:

Request intercepted by Trooper proxy
Privacy flag detected
Routed to local Ollama instead of Claude API
Session context maintained via 3-layer system (Anchor/SITREP/Tail)
JSON response returned — credentials never hit the network

The vault URL and API key stayed on my hardware.

The Code

Using the OpenAI SDK (works with any OpenAI-compatible client):

from openai import OpenAI

client = OpenAI(
    api_key="your-anthropic-key",
    base_url="http://localhost:3000/v1",  # Trooper proxy
)

# Regular request → Claude
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "OAuth2 vulnerabilities?"}],
    extra_headers={"X-Session-ID": "security-analyst"}
)

# Privacy request → Qwen local
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Format: api_key=sk-prod..."}],
    extra_headers={"X-Session-ID": "credential-formatter"},
    extra_body={"x_force_local": True}  # This keeps it local
)

That's the entire API. One boolean flag controls routing.

Why This Matters

Most LLM proxies route between cloud providers. LiteLLM falls back from Claude to OpenAI. That's useful for uptime, but both destinations are someone else's servers.

Trooper's x_force_local routes to your machine. Different failure mode, different privacy guarantee.

When you need it:

Code refactoring with internal URLs
Proprietary algorithms (not secret, just yours)
Customer data that shouldn't leave your network
Cost control (force expensive operations local)
Offline work (flights, train rides, API outages)

When you don't:

Public API questions
General best practices
Complex reasoning that needs Claude's horsepower

The point isn't "local always" or "cloud always." It's per-request control based on what you're asking.

How Context Preservation Works

The hardest part of routing isn't switching models — it's maintaining conversation state.

Trooper uses a 3-layer compaction system:

**Anchor (~10%):** First 2 turns verbatim, never dropped  
**SITREP (~20%):** Rule-based summary of middle turns  
**Tail (~70%):** Last N turns verbatim

Total budget: 6144 tokens (configurable)

When Agent 4 (compliance-reporter) ran locally, Qwen received the anchor, a compressed SITREP of what Claude said earlier, and the immediate context.

What Doesn't Work Great

Local models aren't Claude. Qwen 2.5 is fast and solid for structured tasks (JSON formatting, parsing, summarization). But if you need deep reasoning, route to Claude.

Context compression is lossy. Trooper compresses middle turns into summaries. For precision-critical workflows, keep sessions short or increase the context window.

You need Ollama running. This isn't plug-and-play:

ollama pull qwen2.5:3b
ollama serve

I use qwen2.5:3b (2GB, fast) for most tasks. Switch to 7b (5GB) when I need better output quality.

Compared to My Previous Post

Last time I showed what happens when Claude quota runs out: Trooper automatically falls back to Ollama with context preserved. That's reactive — something breaks, the system recovers.

This is proactive: you tell it "keep this request local" before sending. Different problem, same underlying context system.

Try It Yourself

# 1. Pull local model
ollama pull qwen2.5:3b

# 2. Clone and run Trooper
git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...
go run main.go providers.go classifier.go

Trooper starts on localhost:3000.

Point any OpenAI-compatible client at it and add x_force_local: true when you want privacy routing.

Repo: https://github.com/shouvik12/trooper

Feedback welcome — especially on edge cases or use cases I haven't considered.

This is v3.1. The x_force_local feature shipped last week. Still iterating on auto-routing classification.

How I built a Go proxy that keeps your LLM conversation alive when cloud quota runs out

Shouvik Palit — Sun, 03 May 2026 01:23:28 +0000

Introduction
If you've ever been mid-conversation with Claude or GPT, hit a quota limit, and switched to a local Ollama model,you know the pain. The local model has zero context. It's like walking into a meeting 45 minutes late and nobody catches you up.
I got frustrated enough to build something about it. That something is Trooper.

What is Trooper
Trooper is a lightweight Go proxy (~850 lines, two files) that sits between your application and your LLM providers. When a cloud provider returns a quota error (429, 402, 529), Trooper automatically falls back to a local Ollama instance without dropping the conversation context.
Single binary. Zero dependencies. Easy to audit since it sits in front of your API keys.

The real problem: context loss on fallback
Most fallback proxies solve the routing problem but ignore the context problem. They either pass the raw message history as-is (which blows up the local model's context window) or they truncate the oldest turns (which kills continuity).
Neither works well in practice.

The solution: three-layer context compaction
Trooper uses a structured compaction strategy before handing off to Ollama:
Anchor : The first two turns of the conversation are always preserved. These establish the original intent and set the tone.
SITREP : The middle turns get compressed into a structured summary called a SITREP. It extracts intent, entities, open loops, recent actions, and resolved items. The local model gets situational awareness, not raw history.
Tail : The most recent turns are preserved within a configurable token budget.

A real SITREP looks like this in the logs:

📦  Context compaction triggered — 538 tokens exceeds 500 budget
📦  Context compaction complete
    Total turns    : 7
    Anchor turns   : 2 (~43 tokens)
    Middle turns   : 2 → SITREP (~71 tokens)
    Recent turns   : 3 (~323 tokens)
    Tokens used    : 437 / 500
    SITREP         : intent="trooper" stage=unclear confidence=0.60 open=1 actions=0 resolved=0

The local model knows what you were working on, what's broken, what's been resolved, and what the last few exchanges were. That's enough to keep the conversation coherent.

Why Go
Single binary distribution was the main reason. No runtime, no dependencies, drop it anywhere and it runs. The codebase being ~850 lines also means anyone can read the whole thing in an afternoon — important for something that proxies API keys.

Provider support
Trooper currently supports Claude, Gemini, and OpenAI as cloud providers with automatic fallback to Ollama. The provider chain is configurable via environment variables.

What's next
V3.0 is focused on foundation hardening — concurrency fixes and improved error handling. V3.1 will improve the SITREP extraction quality on longer conversations, which is where intent detection starts to degrade today.

Try it
github.com/shouvik12/trooper
Would love feedback on the context compaction approach — especially from anyone running larger local models. What's your cold-start latency on fallback?