DEV Community: Shouvik Palit

What happens when you give an AI a budget

Shouvik Palit — Sun, 28 Jun 2026 18:22:22 +0000

I gave Claude a budget. It changed what it built.

Over the past few weeks I've been exploring whether LLMs can work with execution budgets and whether that constraint changes how they behave.

The short answer: it does.

The Problem

Researchers studying frontier models found they are consistently over-optimistic about budget. Instead of stopping and alerting the user early, they keep spending tokens on work that's unlikely to succeed. Even after fine-tuning specifically targeting budget awareness, calibration caps at 47%.

You cannot train your way out of this. External enforcement is required.

What I Tried

I started giving Claude implementation tasks with a fixed token budget.

The behavior changed.

Instead of building everything it could think of, the model focused on completing the requested specification before asking for more budget.

Unconstrained Claude on a REST API task added:

JWT authentication (not requested)
Health check endpoint (not requested)
Full test suite (not requested)
nodemon dev dependency (not requested)
URL validation middleware (not requested)

Token Sensei at budget 400 delivered:

The API
Nothing else

The Numbers

Three tasks. Two conditions each. Same model (claude-sonnet-4-5).

Task	Unconstrained	Token Sensei	Savings
Notes REST API	3,584 tokens	2,578 tokens	28%
Bookmark REST API	4,000+ (truncated)	1,604 tokens	60%+
Python CLI	2,466 tokens	993 tokens	60%

The Bookmark Manager task is the most striking. The unconstrained model was still generating when it hit the response ceiling. Token Sensei completed the same task in 1,604 tokens and stopped.

How Token Sensei Works

Set a task and budget
The runtime streams output and tracks exact token usage
When the budget is exhausted, execution pauses
A checkpoint shows what was completed and what remains
You decide — approve a loan to continue, or ship what exists

The key insight: the budget does not just control how much the model produces. It changes what the model optimizes for. Under constraint, it prioritizes completing the requested work. Reduced token usage is an outcome of that shift, not the objective.

The Protocol

Token Sensei is both a runtime and a protocol. The LOAN_REQUEST format tells the model how to surface checkpoints:

LOAN_REQUEST
Completed: ✓ [working deliverables produced]
Remaining: • [specific remaining items]
Requested Budget: [minimum needed]
This budget will complete: ✓ [specific deliverables]

The runtime enforces budget externally using exact provider-reported token counts — not estimates.

Try It

git clone https://github.com/shouvik12/token-sensei
cd token-sensei
npm install
node server.js

Open localhost:3000. Set a task, set a budget, run.

MIT licensed. Would love feedback from the dev.to community — especially whether the checkpoint UX feels right and whether the protocol is something you'd use outside the runtime.

Every AI task should have a budget.

Ship Happens: How I Made a 3B Local Model Trustworthy for Kubernetes (By Never Trusting It)

Shouvik Palit — Sun, 21 Jun 2026 13:36:46 +0000

Small local models are surprisingly capable. They're also wrong in
small, easy-to-miss ways: a deprecated API version here, a
placeholder value that looks real until you actually deploy it
there. The usual fix is better prompting. I tried something
different: stop trying to make the model right, and just make sure
nothing wrong ever ships.

The idea

Ship Happens lets you describe a Kubernetes deployment in plain
English. A local model (qwen2.5:3b, via Ollama) drafts the
manifest. Before anything counts as "done," that manifest gets
tested against a real local cluster (kind) using a server-side
dry-run:

kubectl apply --dry-run=server -f manifest.yaml

If the cluster rejects it, the actual error message, not a
vague "try again", gets fed straight back into the model:

func RegenerateFromError(instruction, imageName, clusterError string) (string, error) {
    correction := fmt.Sprintf(
        "A previous attempt was tested against a real Kubernetes "+
            "cluster and REJECTED with this exact error:\n\n%s\n\n"+
            "Fix the YAML to resolve this specific error.",
        clusterError)
    // ...
}

Then it gets checked again. The model never gets to just "try
again" blindly. Every retry is grounded in what the cluster
actually said was wrong.

What this caught, that I didn't expect

1. Missing `---` silently merges resources

Ask for a Deployment + Service and the model sometimes forgets the
YAML document separator between them. The result isn't a parse
error. It's worse. Without ---, you get one YAML document with
duplicate keys, and most parsers (including the one kubectl uses)
just keep the last value for each duplicate key. The Deployment's
spec silently gets overwritten by the Service's spec. Half your
config vanishes with zero warning.

// Models often show a wrong example before the right one, or just
// forget the separator. Either way, this normalizes it before
// validation ever runs:
func fixMissingDocumentSeparators(yamlContent string) string {
    // detects a second "apiVersion:" line without a preceding ---
    // and inserts one automatically
}

2. Placeholder images that pass every check

The model would write image: your-app-image:v1, syntactically
perfect YAML. Dry-run validates schema, not whether the string
happens to be a real, pullable image. This passes validation
cleanly and only fails 20 minutes later, at actual pod scheduling,
as ImagePullBackOff.

The fix wasn't a smarter prompt (tried that, model just produced a
different placeholder). It was deterministic substitution:

var knownImages = map[string]string{
    "nginx": "nginx:latest", "redis": "redis:7", "postgres": "postgres:16",
    // ...
}

func fixKnownPlaceholderImage(yamlContent string) (string, bool) {
    // finds a known technology keyword inside the placeholder text
    // itself ("your-nginx-image" contains "nginx") and substitutes
    // the real image, deterministically. No second model call,
    // no risk of generating a different placeholder
}

3. The repair that "fixed" the version by deleting it

This one was my favorite. A manifest used the deprecated
policy/v1beta1 for a PodDisruptionBudget. Dry-run rejected it
correctly. The repair loop kicked in, read the error, and
"corrected" it to... v1. Not policy/v1. Just v1, dropping the
API group entirely.

Technically a different string. Still completely wrong, since
PodDisruptionBudget doesn't exist under the bare v1 group at
all. The model recognized something needed to change, defaulted to
the most common generic apiVersion it knew, and got it wrong in a
new way.

The fix was explicit, not clever:

prompt := `IMPORTANT: If the error is about a deprecated or invalid
apiVersion, the API GROUP prefix (the part before the slash)
usually stays the same. Only the version number changes. For
example, "policy/v1beta1" should become "policy/v1", NOT just "v1".
Dropping the group prefix entirely produces a different, equally
invalid value.`

Re-validated after that, and it passed correctly.

"But couldn't jsonschema do this?"

Came up when I posted this, and it's a fair question. Mostly: no,
not for the bugs above.

A static schema is pinned to whatever Kubernetes version it was
bundled for. The policy/v1beta1 bug is a real, structurally valid
shape. It's just been removed from newer clusters. A schema
package might still call that "valid" depending on how current it
is. Dry-run against the real cluster has no such lag: it's checking
against exactly what this cluster, right now, actually accepts.

It also covers CRDs for free. If your cluster has cert-manager,
Prometheus operators, anything custom: there's no generic schema
for that, but dry-run validates against whatever's actually
installed with zero extra configuration. And it goes through the
real admission chain (RBAC, any webhooks), which schema validation
never sees, since it's only checking shape in isolation.

jsonschema is faster and works offline, so it's a reasonable
first-pass filter. It's just not sufficient on its own for what I
was trying to catch.

What it doesn't catch

Worth being honest about the ceiling here. Dry-run proves syntax,
not architecture. It will not catch:

A ReadWriteOnce PersistentVolumeClaim shared across 3 replicas (perfectly valid YAML, breaks at pod scheduling)
A model picking a real-but-wrong image for an ambiguous request (asked for "prometheus," once got handed Alertmanager's image, also real, also wrong)
Container-internal startup requirements, like a Bitnami image refusing to start without ALLOW_EMPTY_PASSWORD=yes set. That's application logic baked into the image, completely invisible to the Kubernetes API

That last one is now caught, just not by dry-run. It's caught by a
Dashboard that polls live pod health, and on a crash, pulls the
actual logs and asks the local model to summarize the root cause
in plain English instead of a raw log dump:

🔴 redis-high-availability
0/3 replicas ready
Missing REDIS_PASSWORD leads to startup failure.

That line is genuinely the model's own summary of a much longer,
uglier log output. Different use of the same local model, not
generating infrastructure, just explaining a failure after the
fact.

What's still unverified

I'd rather say this directly than let someone find out the hard
way: there's a third mode, "My Code," that's supposed to build your
own source into an image and deploy it directly into the cluster.
It's implemented. I have not run it end-to-end yet. The README says
exactly that, in those words, instead of pretending otherwise.

"Public Image" mode (describe it, AI drafts it) and "My YAML" mode
(paste your own, skip generation entirely) have both been tested
repeatedly and are confirmed working. The build-from-source path
hasn't earned that yet.

The actual point

This isn't really about Kubernetes, and it isn't really about this
one model. The pattern is just: don't try to make an unreliable
generator more reliable through better instructions alone. Give it
a real, external thing to be checked against, feed the real
failure back in when it's wrong, and let that loop run until
something true comes out the other side.

A 3B model doesn't need to be smart for this to work. It just needs
something honest to check it against.

Code: github.com/shouvik12/ship-happens

I tracked every GitHub traffic spike for my open source LLM proxy for 7 weeks. Then I did the exact same thing again, and it worked again.

Shouvik Palit — Mon, 15 Jun 2026 03:42:08 +0000

When I shipped Trooper, a privacy-aware LLM proxy written in Go, I didn't have a marketing plan. I had GitHub traffic analytics and a habit of checking them obsessively.

Seven weeks later, I have something more useful than a viral moment: a ranked table of every traffic spike, what caused each one, and proof that the exact same playbook that worked at launch still works when you have something new to say.

What is Trooper?

Trooper sits between your app and your LLM provider. When your cloud quota runs out, it automatically falls back to a local Ollama instance with zero code changes on your end. It also tracks session context, so your agents don't go blind between calls.

It's not a chatbot wrapper. It's plumbing. Which makes the distribution story more interesting, because plumbing doesn't go viral the way demos do.

The Data

GitHub gives you 14-day rolling windows for clones and views. I screenshotted them obsessively and tracked every spike. Here's the full ranked table:

Rank	Date	Clones	Unique Cloners	Views	Unique Visitors	Driver
🥇 1	May 13	375	173	1,113	~140	Reddit wave peak
🥈 2	May 10-12	312	137	974	133	Reddit launch spike
🥉 3	Jun 10	289	124	749	101	"Escalate the model" r/ollama post
4	Jun 11	268	112	840	95	Decaying from Jun 10 spike
5	Jun 12	240	99	739	74	Decaying from Jun 10 spike
6	Jun 9	175	102	802	100	Organic
7	Apr 25	174	71	664	113	Early Reddit posts
8	Jun 7	171	110	876	110	Organic recovery
9	Jun 6	163	104	755	102	Organic recovery
10	May 29-30	122	73	610	83	LinkedIn post
11	May 25	76	48	495	53	Claude Code integration chat

What I learned

1. Reddit is the only thing that moved the needle, and community fit matters more than size

The #1 and #2 peaks were both Reddit-driven. On May 10-11, I posted across r/ollama, r/LocalLLM, r/ClaudeCode, and r/Gemini simultaneously. Total views across those posts: ~7,000.

r/ollama alone drove nearly 4,000 of those views. Not r/LocalLLM. Not r/ClaudeCode. r/ollama, the smallest of the four communities.

The reason: Trooper solves an Ollama-specific problem. Quota exhaustion hitting your local Ollama fallback is something that community lives with daily. Posting to a larger but less relevant community got less traction, even with identical content.

Lesson: Precision beats reach. Find the subreddit where your exact problem is a lived experience, not just a relatable concept.

2. The post that doesn't feel like a post performs best

The r/ollama post that drove the May peak wasn't "I built a thing, please star it." It was structured around the problem first:

"I kept hitting Claude quota limits mid-session and losing context. So I built a proxy that falls back to Ollama automatically."

Nobody wants to read a launch announcement. Everyone wants to read about a problem they recognize.

Lesson: Lead with the pain, not the product.

3. Organic discovery is real, and it compounds quietly

Ranks #3 and #4 (the highest views since the May peak) happened with zero posts in the preceding two weeks. Pure organic discovery.

The referring traffic breakdown tells the story: github.com is the top referrer, meaning developers are finding Trooper by browsing related repos. Google is sending traffic too. Someone is searching for "LLM proxy ollama fallback" and landing on Trooper.

This doesn't show up immediately. It built slowly over six weeks. But it's now driving more daily traffic than the LinkedIn post did.

Lesson: SEO and GitHub organic discovery are slow, but they compound. Write a good README. Use keywords people actually search for.

3.5. The playbook worked a second time, on purpose

Three weeks after writing the first draft of this article, rank #3 happened: 289 clones and 124 unique cloners in roughly 24 hours. Second only to the original launch peak, and bigger than the entire Reddit launch week for views-per-day.

This time it wasn't a mystery. I had shipped a new feature: smart escalation, where Trooper bumps a request up to a bigger model mid-conversation when the local model can't handle it, instead of dropping context and starting over. I wrote it up using the exact same framing from lesson #2, problem first, no "I built a feature" language, and posted it to r/ollama again.

The title was "Escalate the model, not the conversation". Same subreddit as the original launch. Same problem-first structure. The referring sites confirmed it: reddit.com and com.reddit.frontpage combined for over 60 views and 16 unique visitors arriving via Reddit's front page.

Lesson: the launch playbook isn't a one-time unlock. It's a template. Same community, same problem-first framing, new thing to say, comparable result. The hard part was never "how do I get r/ollama's attention once", it's having something genuinely new and useful worth bringing back to them. If you ship something that actually matters to the community you launched in, the same channel that worked once will work again.

4. LinkedIn drove less than you'd expect

The May 29-30 LinkedIn post landed at rank #5. It did move the needle, 34 more clones than the pre-post baseline, but the effect faded within 48 hours and the unique cloner ratio was lower than Reddit.

My read: LinkedIn audiences share content but don't clone repos at the same rate as Reddit or HN. They're evaluating Trooper as a concept, not as a tool they're about to run.

Lesson: LinkedIn is good for reach and credibility. It is not a clone driver.

5. The view-to-clone ratio tells you who's actually landing

During the Reddit peak (May 13): ~140 unique visitors, 173 unique cloners. Ratio above 1.0, meaning people were cloning on multiple machines, or recommending it to colleagues.

During the organic recovery (Jun 7): 111 unique visitors, 106 unique cloners. Near 1.0, meaning developers landing on the page are converting almost immediately.

During the LinkedIn period: ratio dropped. More browsers, fewer cloners.

High view-to-clone ratio means right audience. Low ratio means wrong audience or unclear value prop.

What's next

The biggest shift in how I think about distribution: it's not "do one big launch and hope it compounds." It's "every time you ship something that solves a real problem for the community you launched in, go back and tell them, the same way you did the first time."

The escalation feature post worked because it followed the same rules as the original: problem first, no launch language, posted to the subreddit where the problem is lived daily. Two data points isn't a lot, but it's enough to convince me this is a repeatable loop, not a one-time stroke of luck.

I'm also shipping BRIEFING, a feature that lets agents carry context forward across sessions by reading a structured log on startup. Zero instrumentation, the agent doesn't know it's happening. Same plan as before: ship it, frame the problem it solves, post it to r/ollama.

If you're building open source infrastructure tools and wondering where to start with distribution: r/ollama if you're in the Ollama ecosystem, r/LocalLLaMA if you're broader. Write about the problem before you write about the solution, every time you have something new, not just at launch.

Trooper is open source and MIT licensed: github.com/shouvik12/trooper

Happy to answer questions about the proxy architecture, the session handling, or anything else in the comments.

Your Session History Is Bleeding Tokens Every Time You Paste

Shouvik Palit — Sat, 13 Jun 2026 18:15:03 +0000

Everyone optimizes what they type into Claude.
Nobody optimizes what they paste.

But developers paste constantly. GitHub READMEs. Research papers. API docs. Jira tickets. Confluence pages. Slack threads.

Every time you copy a webpage, your clipboard picks up everything. The navigation. The footer. The boilerplate. The cookie banners. The share buttons rendered as plain text.

You wanted the content. Claude got the garbage too. And you paid for every token of it.

The numbers

Real token counts from live Claude Code sessions:

Content	Before	After	Saved
GitHub README (trooper)	4,600	1,200	74%
Research paper (arXiv)	4,800	1,900	60%
GitHub README (caveman)	1,800	600	67%
API documentation	105	35	67%
tokslayer's own README	800	170	79%
Average			67%

Where the savings actually land

First turn input tokens are paid in full. The saving is on two things:

Output tokens. Claude responds to the compressed version, so answers are shorter and more focused.

Session history. The compressed version stays in context, not the bloated original. Every subsequent turn in that session carries 3,400 fewer tokens of history. Long sessions with multiple pastes, this compounds hard.

Note: A subagent-based approach was considered but doesn't fully solve this .It can't shrink the user's original pasted message in that turn. A write-path
proxy is the correct fix, rewriting the outgoing request on every turn
including history resends. That's the roadmap direction.

What's actually happening when you paste

When you copy a GitHub page you get the content plus:

"Skip to content" navigation
Repo tabs (Code, Issues, Pull requests, Actions...)
Breadcrumbs and branch selectors
Footer links
Share buttons rendered as text
License metadata

None of that is the README. All of it hits Claude's context window. All of it costs tokens.

The fix

A Claude Code skill that sits between your clipboard and Claude. Detects pasted content. Strips the noise. Sends only the signal.

No proxy. No server. No MCP. No configuration. One file. Drop it in. Restart Claude Code. Done.

Receipt on every paste:

ORIGINAL:  "Skip to content shouvik12 trooper Repository..."  (~4,600 tokens)
OPTIMIZED: "Trooper: LLM proxy. Local-first. Ollama default..."  (~1,200 tokens)
SAVED:     ~3,400 tokens (74%)

What gets stripped: nav chrome, footers, filler phrases, redundant sentences, marketing boilerplate.

What stays: headings, code blocks, API signatures, URLs, numbers, technical terms, proper nouns.

Install

curl -fsSL https://raw.githubusercontent.com/shouvik12/tokslayer/main/install.sh | bash

Restart Claude Code. Works on every paste automatically from that point on.

The meta test

Ran tokslayer's own README through itself and typed "summarize":

ORIGINAL:  "tokslayer, Slays tokens before they reach Claude..."  (~800 tokens)
OPTIMIZED: "Tokslayer: Claude Code skill. Compresses pasted content..."  (~170 tokens)
SAVED:     ~630 tokens (79%)

A tool that eats its own cooking.

Where it fits

This covers the input side. Pair it with caveman for output compression
(65% reduction on Claude responses).

Together: lean input, lean output

https://github.com/shouvik12/tokslayer

Stop explaining yourself to Claude

Shouvik Palit — Fri, 12 Jun 2026 04:26:55 +0000

You're wasting tokens. Not a little -a lot.

Here's a prompt I see constantly:

"I have a React app and I'm using the useState hook. My component re-renders every time the parent renders even though the props haven't changed. Why is this happening?"

Claude doesn't need any of that setup. It already knows React. It already knows what useState is. The only thing it needed was:

"Component re-renders on parent render. Props unchanged. Why."

Same answer. 64% fewer tokens.

The delta principle

Most prompts are written for humans. We explain context, name the framework, describe how things work before asking the question. That's how we communicate with each other.

But Claude already knows the context. The only thing it needs is the delta — the new information, the specific problem, the unknown.

Everything else is noise.

What you can safely strip

Claude already knows these — stop re-explaining them:

Framework names used as context ("I have a React app", "I'm using Python")
Concept explanations ("hooks are a React feature that...", "a closure is when...")
Stack introductions ("my app uses Node, Express, and MongoDB")

Social noise that adds zero signal:

Pleasantries: "hey", "hope you can help", "thanks in advance"
Permission requests: "could you please", "I was wondering if"
Hedging: "I think", "I'm not sure but", "maybe", "possibly"
Filler: "basically", "essentially", "just", "simply", "actually"

What you should never strip:

The actual error, bug, or problem
Numbers, thresholds, measurements
Variable names, function names, file names
Code blocks and URLs
Anything Claude could NOT know without being told

Before and after

Debugging:

Before (41 tokens):
"I'm working on a Node.js Express API and I'm getting a 401 unauthorized 
error when I try to call the endpoint. I'm passing the JWT token in the 
Authorization header."

After (12 tokens):
"401 on endpoint. JWT in Authorization header."

Code review:

Before (29 tokens):
"Could you please review this Python function and tell me if there are 
any issues or improvements I could make?"

After (6 tokens):
"Review. Issues + improvements."

Explanation:

Before (19 tokens):
"I was wondering if you could explain how database connection pooling 
works in simple terms?"

After (5 tokens):
"Explain connection pooling. Simple."

The compound effect

Single prompt savings look small. But across a real session, it compounds.

Here's a simulated 20-turn dev session — the kind where you're debugging something across multiple back-and-forths:

Turn	Verbose (tokens)	Delta (tokens)	Saved
1	48	18	30
2	35	12	23
3	52	14	38
4	29	8	21
5	41	11	30
6	33	9	24
7	44	15	29
8	38	10	28
9	31	8	23
10	45	13	32
11	27	7	20
12	39	11	28
13	50	16	34
14	36	10	26
15	42	12	30
16	28	8	20
17	46	14	32
18	33	9	24
19	40	11	29
20	37	10	27
Total	757	226	531

531 tokens saved in a single session. 70% reduction.

On Claude's API at Sonnet pricing, that's a small number in dollars. But if you're building on top of the API and running hundreds of sessions a day, it adds up fast. And even on claude.ai, fewer input tokens means less context noise — Claude processes cleaner signal and responds more precisely.

Three intensity levels

Not every prompt needs ultra-compression. I use three modes depending on the situation:

lite — strip pleasantries only, keep context (~20% reduction)

Use when: onboarding a new topic, first message in a session

full — strip everything Claude knows, keep only the delta (~60% reduction)

Use when: mid-session debugging, iterating on code, quick questions

ultra — compress to bare minimum signal (~70%+ reduction)

Use when: you know exactly what you want and don't care about polish

The skill file

I turned this into a Claude skill — a markdown file that instructs Claude to apply delta compression automatically, with activation/deactivation commands and intensity switching.

→ github.com/shouvik12/delta

The README has the full rule set, intensity examples, and instructions for adding it to your Claude setup.

One thing worth thinking about

This is a small optimization. But the principle behind it is bigger:

We've been writing prompts for humans. We explain, we hedge, we contextualize — because that's how we earn understanding from other people. With LLMs, that overhead is waste. The model doesn't need to be convinced you know what you're talking about. It doesn't need the social scaffolding.

Just send the delta.

Escalate the Model, Not the Conversation

Shouvik Palit — Wed, 10 Jun 2026 19:03:23 +0000

Trooper started as a fallback proxy for agents. Claude hits a quota, falls back to Ollama, session continues. No crashes, no lost context.

The interesting problem that came up wasn't model routing. It was context preservation.

When you're debugging something hard, you build up context over many turns. The problem statement, what you've tried, what failed. When you switch from a local model to Claude, all of that context has to go with it. And when you come back to local, the local model needs to know what Claude said.

That's what 4.0 solves.

How it works

A local model handles requests by default. Fast, free, private.

When it gets stuck, one click escalates to Claude — the full conversation history is injected automatically. Claude answers. Then control returns to the local model, which continues the conversation knowing exactly what Claude said.

No copy-pasting. No restarting the conversation. No lost context.

The escalation moment

You're debugging a slow Postgres query. Llama gives you a decent answer — check your EXPLAIN output, look for function calls on indexed columns. Good start.

Not enough. You hit Escalate.

Claude receives the full session. It knows you're debugging a slow query. It knows what Llama already told you. It picks up exactly where the conversation left off.

You click Back to local.

Now ask Llama to summarize what Claude said.

It does. Correctly. Because the session store was updated with Claude's response. Llama reads the full history including what Claude said and continues from there.

What's under the hood

Trooper is a Go proxy that sits between your client and any LLM provider. The chat UI is a static HTML file served by the same process.

When you escalate:

The UI fetches the full session history from /session/:id
Sends it to Claude via /v1/messages with X-Force-Cloud: true
Claude's response gets written back to the session store via /session/:id/append
Next local turn, Llama reads the full history including Claude's response

The SITREP panel on the right extracts intent, confidence, entities and open loops from the conversation using a rule-based classifier — no LLM call needed.

The proxy still works

The proxy layer is unchanged. Agents, SDK clients, curl — everything routes through /v1/messages the same way it always did.

# Agent flow — unchanged
export ANTHROPIC_BASE_URL=http://localhost:3000
export OPENAI_BASE_URL=http://localhost:3000

# Chat UI — new in 4.0
open http://localhost:3000/chat

Try it

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...  # optional
export OLLAMA_MODEL=llama3.1:8b
go run .
open http://localhost:3000/chat

Works without a Claude key-escalation falls back to Ollama. Add the key when you want real cloud escalation.

https://github.com/shouvik12/trooper

How I Cut Agent Token Usage by 89% Without Touching the Agent

Shouvik Palit — Fri, 05 Jun 2026 13:10:22 +0000

Every time your agent calls an LLM, it sends the full conversation history.

Turn 20 includes turns 1–19. Turn 50 includes turns 1–49. Nobody notices because it's happening inside the agent, silently, on every single request.

I noticed it while building Trooper - a Go proxy that sits between agents and LLMs. I was watching token counts climb across a long debugging session and realised the agent was replaying the same context over and over. Most of it was noise.

The model didn't need a transcript. It needed state.

What state actually means

After a few turns, most of what matters in a session falls into four categories:

Decisions made — what was chosen and why
Constraints locked — what cannot change
Open loops — what still needs to be resolved
Ruled out — what was tried and rejected

That's it. Everything else — the back and forth, the verbose LLM responses explaining things, the repeated context — is replay. The model doesn't need it again.

The SITREP

I added structured session memory to Trooper. After enough turns, Trooper's local Llama model generates a SITREP — a situation report — from the user messages in the session.

It looks like this:

INTENT: Build a RAG pipeline with ChromaDB and nomic-embed-text

DECISIONS: Use cosine similarity over MMR — focused queries not broad;
           Chunk size 256, overlap 30 — locked;
           Pure vector search — ChromaDB no hybrid support;
           Top k set to 5

CONSTRAINTS: Node 18 locked — platform team constraint, no exceptions;
             Re-ranking ruled out — latency jumped 200ms to 800ms

OPEN: Poor recall on technical queries — nomic-embed-text struggles with domain jargon;
      Evaluating bge-small as alternative

From that point forward, every request to the LLM sends:

Anchor (first 2 turns verbatim)
+ SITREP (structured state)
+ Tail (last N turns verbatim)

Instead of the full history.

The numbers

From a real 15-turn session:

Full history:    10,820 tokens per request
With Trooper:     1,157 tokens per request
Reduction:             89%

Visible live on the dashboard.

Does the LLM still answer correctly?

This was the question that mattered. Token savings are worthless if the model loses coherence.

To test it: I took the auto-generated SITREP, opened a completely fresh chat with no history, and asked questions about decisions made in the original session.

Questions:

What is the chunk size?
Why did we rule out hybrid search?
What retrieval method did we choose and why?
What is still open?

Result: All four answered correctly. The model worked entirely from the SITREP. No history. No context bleed.

That's the claim: structured state is sufficient for the model to continue reasoning correctly — and it costs 89% less to send.

How it works

Trooper is a Go proxy — one binary, no SDK, no instrumentation. You point your existing agent at it by changing one URL.

# Before
export ANTHROPIC_BASE_URL=https://api.anthropic.com

# After
export ANTHROPIC_BASE_URL=http://localhost:3000

Nothing else changes. Trooper intercepts every request, maintains session state, and when the SITREP is ready, rewrites the messages array before forwarding to the LLM.

The SITREP is built by a local Llama 3.1 8b model running via Ollama — fast, private, no cloud cost. The extraction happens asynchronously in the background. The main request path is not blocked.

// GetTripleAnchor assembles what gets sent to the LLM
func (s *SessionStore) GetTripleAnchor(sessionID string) []map[string]string {
    payload := append([]map[string]string{}, state.Anchor...)
    if state.SITREP != "" {
        payload = append(payload, map[string]string{
            "role":    "system",
            "content": fmt.Sprintf("[STATE_SITREP: %s]", state.SITREP),
        })
    }
    return append(payload, state.Tail...)
}

The dashboard shows the compression ratio live:

HISTORY COMPRESSED    89%
TOKENS SAVED          459
CONFIDENCE            100%

Why this is different from conversation summarisation

Most summarisation tools compress what was said. The SITREP extracts what matters for the next action.

Copilot's context compaction summarises the full conversation — useful for humans in long chats. The SITREP is structured specifically for agents: decisions, constraints, open loops, ruled-out paths. Not a narrative summary. A state snapshot.

The result is that subsequent turns stay coherent on intent without replaying noise. More relevant for agents running repeated structured workflows than for general chat.

The limitation

The SITREP works best for structured agentic workflows — debugging sessions, research pipelines, multi-step build tasks. For open-ended creative work where tangential context might become important later, you'd want a larger tail window or higher fidelity compression.

The tail window is configurable. You can keep more raw context for less structured sessions.

What else Trooper does

The compression is the latest addition. Trooper also:

Falls back to local Ollama when cloud quota hits — context preserved across the switch
Routes simple turns to Ollama automatically — cloud never contacted
Privacy routing — sensitive requests stay local via x_force_local
Live dashboard — intent, open loops, completed steps, transcript
Subagent recovery — /recovery/{session_id} tells you exactly where to resume

All from one URL change.

The bigger question

We tend to treat conversation history as memory. But a transcript is a log. Memory is state.

Humans don't replay every prior conversation before making a decision. They carry forward conclusions, constraints, unresolved questions, and relevant context — a structured snapshot, not a full transcript.

Long-running agents may need to do the same. Not because of token costs — though that helps — but because state is a better abstraction for agent memory than history.

The SITREP is an experiment in that direction.

github.com/shouvik12/trooper — Go, MIT, zero dependencies beyond Ollama.

I Added a Live Dashboard to My LLM Proxy. Zero Instrumentation. Just a URL Change.

Shouvik Palit — Thu, 28 May 2026 15:24:53 +0000

I built Trooper as a fallback proxy. Claude hits quota → falls back to Ollama. Useful but passive. It sat in the background, invisible, doing its job silently.

Today it became something different.

The Original Problem

When you're building with LLMs, quota hits are inevitable. Claude's free tier is generous until it isn't. A mid-session 429 kills your context, your workflow, your train of thought.

Trooper solved that. Point your app at http://localhost:3000 instead of the Claude API. When Claude fails, Trooper catches it, preserves the full session context via a 3-layer compaction system (Anchor + SITREP + Tail), and continues on local Ollama. Your app never knows anything happened.

That's still there. But it was passive. Useful when things broke. Invisible when they didn't.

The Problem With Passive

Passive infrastructure has an adoption problem. Developers install it, forget about it, and only notice it when something breaks. That's not a product — that's a safety net.

The question I kept asking: what does Trooper do that has daily value, not just failure value?

The answer was sitting in the code the whole time.

What Was Already There

Trooper captures every message in every session. It runs a classifier on each one — extracting intent, entities, open loops, completed steps, recent actions. All rule-based, zero LLM calls, zero latency.

This is what powers the fallback context preservation. When Claude fails and Ollama picks up, Ollama doesn't start blind — it receives a SITREP (Situation Report) that tells it what the session was about, what was completed, what's still pending.

That data exists for every session. It was just never visible to the developer.

The Dashboard

I added a live dashboard at localhost:3000/dashboard.

Point any agent at Trooper — just change your base URL:

export ANTHROPIC_BASE_URL=http://localhost:3000
# or
export OPENAI_BASE_URL=http://localhost:3000

Open the dashboard. Keep it on a second monitor while your agent runs.

From a single message, it already shows:

Intent — what your agent is trying to do, extracted automatically
Open Loops — what it's stuck on, highlighted in red
Completed Steps — what it finished, tracked as it happens
Entities — the key things being referenced
Session Transcript — every message, colour coded by role

Auto-refreshes every 5 seconds. No page reload needed.

What It Looks Like In Practice

I ran a 3-turn agent session simulating a database debugging workflow:

Turn 1: "I am building a Go API server. The database connection is failing with connection refused errors on port 5432."

Turn 2: "Checked the config. Postgres is running on port 5433 not 5432. Fixing the connection string now."

Turn 3: "Fixed the port. Database connection is working. API server is running successfully."

After Turn 1, the dashboard immediately showed:

Intent: "building a go api server. the database connection is failing with connection refused errors on port" (100% confidence)
Entities: Postgres, network
Open Loops: "fail with connection refused error on port"

After Turn 3:

Completed Steps: "successfully fixed the port"
Open loops cleared

Zero instrumentation. No SDK. No code changes to the agent. Just a URL change.

Why This Matters

Every observability tool requires you to instrument your code:

LangSmith — wrap your agent in LangChain
Langfuse — add their SDK
AgentOps — add @observe decorators

Trooper requires nothing. Your agent already communicates over HTTP to an LLM. Trooper sits in that path and observes everything automatically.

Helicone was the closest — proxy-based, zero instrumentation. But it went into maintenance mode in March 2026 and was cloud-only. Your data went to their servers.

Trooper is open source, local-first, and free forever. Your data never leaves your machine.

The Sessions Endpoint

Not sure which session to look at? Hit /sessions:

curl http://localhost:3000/sessions
# {"sessions":["agent-debug-123","agent-debug-456"],"count":2}

Click any session in the dashboard home page at localhost:3000/dashboard to open it.

The Recovery Endpoint

Still there. When an agent fails mid-task:

curl http://localhost:3000/recovery/{session_id}

Returns exactly what completed and where to resume. The dashboard makes this visual — completed steps are tracked in real time.

The Pivot

Trooper started as: "Claude failed. Trooper caught it."

Trooper is now: "Your agent communicates over HTTP to an LLM. Trooper can observe it."

The fallback is a feature. Observability is the product.

Your agent was always talking. Now you can hear it.

Get Started

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...
go run .

Open http://localhost:3000/dashboard in your browser.

Point your agent at Trooper:

export ANTHROPIC_BASE_URL=http://localhost:3000

That's it. Zero dependencies. Pure Go. Runs in under 60 seconds.

GitHub: github.com/shouvik12/trooper

Tags: llm, agents, observability, ollama, go, opensource

I Added a /recovery Endpoint to My LLM Proxy So Agents Never Lose Progress Mid-Task

Shouvik Palit — Sun, 24 May 2026 17:28:55 +0000

Most LLM proxies handle failures the same way -retry the request, fall back to another provider, or crash. None of them ask the more important question: what did the agent already complete before it failed?

That's the gap I built Trooper to fill.

The Problem

If you're running multi-agent workflows, you've probably hit this scenario:

A subagent starts a long task-reviewing PRs, processing documents, running analysis
It completes steps 1, 2, and 3
On step 4 it hits a quota error, rate limit, or provider failure
Your orchestration layer has no idea what completed
You restart from scratch and repeat all the work

Most proxies handle the failure. Nobody handles the recovery.

What Trooper Does Differently

Trooper is a Go-based LLM proxy that sits between your agents and your LLM providers. It already handled fallback routing -if Claude hits quota, it falls back to local Ollama automatically.

But the new /recovery/{session_id} endpoint goes further. It tracks every step your agent completes in real time and tells your orchestration layer exactly where to resume.

The Recovery Endpoint

When your agent sends requests through Trooper, it captures every assistant response and extracts completed steps as they happen. When something fails, you call:

GET http://localhost:3000/recovery/{session_id}

And Trooper returns:

{
  "session_id": "subagent-demo-1779630533",
  "completed_steps": [
    "completed pr #1",
    "completed pr #2",
    "completed pr #3"
  ],
  "resume_from": 4,
  "recovery_hint": "Resume from step 4"
}

Your parent agent now knows exactly what the subagent finished and where to restart it. No repeated work. No lost progress.

Demo

An agent reviewing 8 pull requests hits quota on PR #4. Trooper intercepts, returns the recovery payload, and the agent resumes from PR #4 using local Ollama.

[https://www.youtube.com/watch?v=NN2uwQZDCck]

How It Works

Trooper uses a two-tier memory system:

Anchor — the first two turns of a session, always preserved verbatim.

Tail — the most recent turns, stored in a rolling window.

When you call /recovery, Trooper scans all stored assistant messages for completion signals — words like "completed", "finished", "done", "merged", "deployed". It extracts one completed step per message, deduplicates by task identifier, and returns the ordered list.

The resume_from field is simply len(completed_steps) + 1 — telling your orchestration layer which step to restart the subagent on.

How to Use It

1. Start Trooper

git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=your-key
go run .

2. Point your agent at Trooper instead of Claude

# Instead of https://api.anthropic.com/v1/messages
POST http://localhost:3000/v1/chat/completions

3. Pass a session ID with each request

curl -X POST http://localhost:3000/v1/chat/completions \
  -H "X-Session-ID: my-agent-session-1" \
  -H "Content-Type: application/json" \
  -d '{"model":"claude-haiku-4-5","max_tokens":100,"messages":[...]}'

4. Call recovery when something fails

curl http://localhost:3000/recovery/my-agent-session-1

What's Next

The recovery endpoint is the foundation for proper subagentic orchestration. Upcoming work:

Parent agent integration — the recovery payload feeds directly back into the orchestration layer to automatically restart subagents from the right step
Structured step tracking — support for agents that emit structured JSON progress instead of natural language
Session replay — rewind any session to any point and branch from there

The Bigger Picture

As agent workflows get longer and more complex, failure recovery becomes a first-class concern. Trooper's approach — track everything as it happens, make recovery queryable — is a different philosophy from retry-and-hope.

Local-first by default. Cloud when you choose. And now recoverable when things go wrong.

GitHub: https://github.com/shouvik12/trooper

How are you handling agent failures in your workflows today? Drop a comment genuinely curious what patterns people are using.

I Tested Privacy-Aware Routing with 4 AI Agents: What Actually Stayed Local

Shouvik Palit — Wed, 13 May 2026 02:40:52 +0000

Following up on my earlier Trooper experiments, I wanted to see if per-request privacy routing actually works in practice.

The test: 4 agents running simultaneously. Some handling public knowledge (OAuth security, Redis vs Memcached). Others handling sensitive data (API keys, customer PII).

The rule: Credentials and PII stay on my machine. Everything else can use Claude.

The Setup

Each agent gets a x_force_local flag:

Agent 1 - security-analyst (☁️ Claude)

Task: "What are the top 3 OAuth2 vulnerabilities?"  
Routing: Public knowledge, let Claude handle it

Agent 2 - credential-formatter (🔒 Qwen local)

Task: "Format as JSON: api_key=sk-prod-x7f9k2m, vault_url=https://vault.acme.io:8200"  
Routing: Contains credentials — must stay on machine

Agent 3 - architecture-advisor (☁️ Claude)

Task: "Redis or Memcached for session storage?"  
Routing: General best practices, use cloud

Agent 4 - compliance-reporter (🔒 Qwen local)

`Task: "Summarize: 47 tickets today. 3 had PII (Alice Johnson, Bob Chen, Maria Garcia)"  
Routing: Contains customer names — privacy violation if sent to cloud`

The Result

Every agent completed successfully:

Cloud agents: 3.8s and 2.4s (Claude handled complex reasoning)
Local agents: 2.4s and 1.2s (Qwen formatted data locally)

The critical part: API keys, vault URLs, and customer names never left my machine. Zero network calls to Anthropic for those two agents.

What Happened Under the Hood

When Agent 2 (credential-formatter) ran with x_force_local: true:

Request intercepted by Trooper proxy
Privacy flag detected
Routed to local Ollama instead of Claude API
Session context maintained via 3-layer system (Anchor/SITREP/Tail)
JSON response returned — credentials never hit the network

The vault URL and API key stayed on my hardware.

The Code

Using the OpenAI SDK (works with any OpenAI-compatible client):

from openai import OpenAI

client = OpenAI(
    api_key="your-anthropic-key",
    base_url="http://localhost:3000/v1",  # Trooper proxy
)

# Regular request → Claude
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "OAuth2 vulnerabilities?"}],
    extra_headers={"X-Session-ID": "security-analyst"}
)

# Privacy request → Qwen local
response = client.chat.completions.create(
    model="claude-sonnet-4-6",
    messages=[{"role": "user", "content": "Format: api_key=sk-prod..."}],
    extra_headers={"X-Session-ID": "credential-formatter"},
    extra_body={"x_force_local": True}  # This keeps it local
)

That's the entire API. One boolean flag controls routing.

Why This Matters

Most LLM proxies route between cloud providers. LiteLLM falls back from Claude to OpenAI. That's useful for uptime, but both destinations are someone else's servers.

Trooper's x_force_local routes to your machine. Different failure mode, different privacy guarantee.

When you need it:

Code refactoring with internal URLs
Proprietary algorithms (not secret, just yours)
Customer data that shouldn't leave your network
Cost control (force expensive operations local)
Offline work (flights, train rides, API outages)

When you don't:

Public API questions
General best practices
Complex reasoning that needs Claude's horsepower

The point isn't "local always" or "cloud always." It's per-request control based on what you're asking.

How Context Preservation Works

The hardest part of routing isn't switching models — it's maintaining conversation state.

Trooper uses a 3-layer compaction system:

**Anchor (~10%):** First 2 turns verbatim, never dropped  
**SITREP (~20%):** Rule-based summary of middle turns  
**Tail (~70%):** Last N turns verbatim

Total budget: 6144 tokens (configurable)

When Agent 4 (compliance-reporter) ran locally, Qwen received the anchor, a compressed SITREP of what Claude said earlier, and the immediate context.

What Doesn't Work Great

Local models aren't Claude. Qwen 2.5 is fast and solid for structured tasks (JSON formatting, parsing, summarization). But if you need deep reasoning, route to Claude.

Context compression is lossy. Trooper compresses middle turns into summaries. For precision-critical workflows, keep sessions short or increase the context window.

You need Ollama running. This isn't plug-and-play:

ollama pull qwen2.5:3b
ollama serve

I use qwen2.5:3b (2GB, fast) for most tasks. Switch to 7b (5GB) when I need better output quality.

Compared to My Previous Post

Last time I showed what happens when Claude quota runs out: Trooper automatically falls back to Ollama with context preserved. That's reactive — something breaks, the system recovers.

This is proactive: you tell it "keep this request local" before sending. Different problem, same underlying context system.

Try It Yourself

# 1. Pull local model
ollama pull qwen2.5:3b

# 2. Clone and run Trooper
git clone https://github.com/shouvik12/trooper
cd trooper
export CLAUDE_API_KEY=sk-ant-...
go run main.go providers.go classifier.go

Trooper starts on localhost:3000.

Point any OpenAI-compatible client at it and add x_force_local: true when you want privacy routing.

Repo: https://github.com/shouvik12/trooper

Feedback welcome — especially on edge cases or use cases I haven't considered.

This is v3.1. The x_force_local feature shipped last week. Still iterating on auto-routing classification.

How I built a Go proxy that keeps your LLM conversation alive when cloud quota runs out

Shouvik Palit — Sun, 03 May 2026 01:23:28 +0000

Introduction
If you've ever been mid-conversation with Claude or GPT, hit a quota limit, and switched to a local Ollama model,you know the pain. The local model has zero context. It's like walking into a meeting 45 minutes late and nobody catches you up.
I got frustrated enough to build something about it. That something is Trooper.

What is Trooper
Trooper is a lightweight Go proxy (~850 lines, two files) that sits between your application and your LLM providers. When a cloud provider returns a quota error (429, 402, 529), Trooper automatically falls back to a local Ollama instance without dropping the conversation context.
Single binary. Zero dependencies. Easy to audit since it sits in front of your API keys.

The real problem: context loss on fallback
Most fallback proxies solve the routing problem but ignore the context problem. They either pass the raw message history as-is (which blows up the local model's context window) or they truncate the oldest turns (which kills continuity).
Neither works well in practice.

The solution: three-layer context compaction
Trooper uses a structured compaction strategy before handing off to Ollama:
Anchor : The first two turns of the conversation are always preserved. These establish the original intent and set the tone.
SITREP : The middle turns get compressed into a structured summary called a SITREP. It extracts intent, entities, open loops, recent actions, and resolved items. The local model gets situational awareness, not raw history.
Tail : The most recent turns are preserved within a configurable token budget.

A real SITREP looks like this in the logs:

📦  Context compaction triggered — 538 tokens exceeds 500 budget
📦  Context compaction complete
    Total turns    : 7
    Anchor turns   : 2 (~43 tokens)
    Middle turns   : 2 → SITREP (~71 tokens)
    Recent turns   : 3 (~323 tokens)
    Tokens used    : 437 / 500
    SITREP         : intent="trooper" stage=unclear confidence=0.60 open=1 actions=0 resolved=0

The local model knows what you were working on, what's broken, what's been resolved, and what the last few exchanges were. That's enough to keep the conversation coherent.

Why Go
Single binary distribution was the main reason. No runtime, no dependencies, drop it anywhere and it runs. The codebase being ~850 lines also means anyone can read the whole thing in an afternoon — important for something that proxies API keys.

Provider support
Trooper currently supports Claude, Gemini, and OpenAI as cloud providers with automatic fallback to Ollama. The provider chain is configurable via environment variables.

What's next
V3.0 is focused on foundation hardening — concurrency fixes and improved error handling. V3.1 will improve the SITREP extraction quality on longer conversations, which is where intent detection starts to degrade today.

Try it
github.com/shouvik12/trooper
Would love feedback on the context compaction approach — especially from anyone running larger local models. What's your cold-start latency on fallback?

DEV Community: Shouvik Palit

What happens when you give an AI a budget

Ship Happens: How I Made a 3B Local Model Trustworthy for Kubernetes (By Never Trusting It)

The idea

What this caught, that I didn't expect

1. Missing --- silently merges resources

2. Placeholder images that pass every check

3. The repair that "fixed" the version by deleting it

"But couldn't jsonschema do this?"

What it doesn't catch

What's still unverified

The actual point

I tracked every GitHub traffic spike for my open source LLM proxy for 7 weeks. Then I did the exact same thing again, and it worked again.

What is Trooper?

The Data

What I learned

1. Reddit is the only thing that moved the needle, and community fit matters more than size

2. The post that doesn't feel like a post performs best

3. Organic discovery is real, and it compounds quietly

3.5. The playbook worked a second time, on purpose

4. LinkedIn drove less than you'd expect

5. The view-to-clone ratio tells you who's actually landing

What's next

Your Session History Is Bleeding Tokens Every Time You Paste

The numbers

Where the savings actually land

What's actually happening when you paste

The fix

Install

The meta test

Where it fits

Stop explaining yourself to Claude

The delta principle

What you can safely strip

Before and after

The compound effect

Three intensity levels

The skill file

One thing worth thinking about

Escalate the Model, Not the Conversation

How it works

The escalation moment

What's under the hood

The proxy still works

Try it

How I Cut Agent Token Usage by 89% Without Touching the Agent

What state actually means

The SITREP

The numbers

Does the LLM still answer correctly?

How it works

Why this is different from conversation summarisation

The limitation

What else Trooper does

The bigger question

I Added a Live Dashboard to My LLM Proxy. Zero Instrumentation. Just a URL Change.

The Original Problem

The Problem With Passive

What Was Already There

The Dashboard

What It Looks Like In Practice

Why This Matters

The Sessions Endpoint

The Recovery Endpoint

The Pivot

Get Started

I Added a /recovery Endpoint to My LLM Proxy So Agents Never Lose Progress Mid-Task

The Problem

What Trooper Does Differently

The Recovery Endpoint

Demo

How It Works

How to Use It

What's Next

The Bigger Picture

I Tested Privacy-Aware Routing with 4 AI Agents: What Actually Stayed Local

The Setup

The Result

What Happened Under the Hood

The Code

1. Missing `---` silently merges resources