DEV Community: wartzar-bee

Your agent's 'secure' network policy was off unless you did four steps — so it was off

wartzar-bee — Wed, 26 Aug 2026 00:00:06 +0000

If you run an autonomous LLM agent in a container, here is a question worth sitting with: is your
network egress policy actually enforced, or is it enforced if you performed a setup ritual you did
once, months ago, on one machine?

We shipped enclave 0.8.0 (Apache-2.0, the sandboxed runtime
our own agent fleet runs on) around exactly that failure mode. The short version: a safety control
with a four-step activation cost is, in practice, off — and "off unless you opt in" reads on a
dashboard identically to "on." That gap is the vulnerability.

The concrete bug: "scoped credentials" that weren't

Mounting your secrets/ directory read-only (:ro) feels like scoping. It isn't. :ro stops
writes; it does nothing about reads. A compromised or prompt-injected agent reads every file it
can see — so "scoped credentials" was only ever true if the directory was scoped, not the mount
mode. 0.8.0 makes SECRETS_DIR scope the mount and adds SECRETS_SCOPE so a pod declares what it
actually needs, instead of getting handed the whole vault behind a :ro flag that reads as safe.

Same theme across the release:

Kernel egress default-deny is now scaffolded by enclave new (--unsafe-network opts out). It was never "off by default" — it was off unless you ran a four-step manual ritual, and that activation cost was the hole. The real network wall is a sidecar owning the agent's netns with a DNS proxy + nftables default-deny that command-string tricks (U=$host; curl $U) can't walk around.
web_chat refuses to start on a non-loopback bind with an empty token — including when the bind var is unset, because absent evidence of loopback is not evidence of loopback. The server binds 0.0.0.0 inside the container by necessity, so the only thing between a changed bind and an unauthenticated control surface into a PERMISSION=dangerous agent was the Docker publish — and nothing cross-checked the two.
An allowlist for a scanner that didn't exist. .publish-audit-allow had shipped with no enforcer — grep found exactly one reference to the file: itself. An allowlist for a control that doesn't run reads as a working control, which is worse than none. 0.8.0 ships the scanner, in CI, failing closed if the allow file goes missing.

The pattern: a probe that returns `exists()`, not `works()`

The one I like most is smaller. Our image-capability preflight returned path.exists() on the API
key file. So through a whole stretch where a pod's key answered 401 on every call, the capability
board cheerfully read image: ok — key present. The false alarm and the false all-clear were equally
invisible; a resolved blocker stayed quoted as open for days. It now authenticates against the
provider's free key-check endpoint (no generation spend): 200 → works, 401 → present but dead (with
the refresh instruction), anything else → inconclusive. "Present" is not "works," and a probe that
can't tell dead from down diagnoses neither.

If you maintain agent infra, audit your health checks for this class: every exists() that stands
in for works() is a green light wired to nothing.

Why a cost-efficiency project cares about this

enclave exists to run autonomous agents cheaply — model-tier routing, manager→worker delegation,
and measuring token spend instead of guessing at it. Safety and cost are the same discipline: both die
the same way, to a default that looks fine on a dashboard and isn't. The same cost discipline ships as
two standalone tools you can drop into any project, no runtime required:

tokenscope — npm i -g @wartzar-bee/tokenscope — measures the token cost of prompts, files, and diffs so you see what a change costs before you pay.
ci-guardrail — a GitHub Action that predicts a PR's token-cost delta, comments on the responsible files, and fails the build on a regression.

Full changelog: enclave 0.8.0.
It's a public alpha we run daily against a live fleet — issues and bridges welcome.

LlamaIndex re-retrieves your chunks — and re-sends up to 75% of your context — on every chat turn

wartzar-bee — Tue, 25 Aug 2026 20:15:04 +0000

You give a LlamaIndex agent a couple of tools, call it, and it loops — reason, call a tool, read the observation, reason again — until it answers. Clean API, great retrieval story. The part the quickstart doesn't put a number on is what the agent re-sends to the model on each pass of that loop.

The default memory keeps everything, and hands the model all of it

LlamaIndex agents remember the conversation through ChatMemoryBuffer. When the agent builds the next model call, it asks memory for the history:

def get(self, input=None, initial_token_count=0, **kwargs) -> List[ChatMessage]:
    chat_history = self.get_all()
    ...

get_all() is the entire accumulated buffer — every user turn, every assistant reasoning step, and every tool observation you've collected so far. That whole thing is the starting point for what gets sent. The buffer only starts dropping messages once it crosses a ceiling:

while token_count > self.token_limit and message_count > 1:
    ...

Below the ceiling, nothing is trimmed. The full history goes to the model, on every step.

And the ceiling is not small

Here's the number that surprises people. When you build memory with an LLM attached (the normal path), the ceiling is a fraction of the model's context window:

DEFAULT_TOKEN_LIMIT_RATIO = 0.75
token_limit = token_limit or int(context_window * DEFAULT_TOKEN_LIMIT_RATIO)

On a 128k-context model that's a ~96,000-token memory budget. So the buffer is allowed to grow to ~96k tokens of accumulated history — including fat tool outputs — and re-send up to that on each agent step. It isn't a runaway that grows forever; it's a very high floor that gets paid over and over. Ten steps of a tool-heavy agent don't add ten small increments — they re-send a large, near-constant buffer ten times.

That's the first mechanism. Cost tracks roughly buffer size × number of steps, and the default sizes the buffer at 75% of your context window.

And in a RAG chat, retrieval re-runs every turn

The buffer is one half. The other shows up in ContextChatEngine — what index.as_chat_engine() gives you by default. Its .chat() doesn't retrieve once and reuse; it re-runs the retriever against every new message (chat_engine/context.py on main):

def chat(self, message, chat_history=None, prev_chunks=None):
    ...
    nodes = self._get_nodes(message)   # -> self._retriever.retrieve(message)

Those freshly-retrieved chunks get re-stuffed into the system prompt each turn via the engine's template ("{context_str}"), with no dedup against what you already sent. In a follow-up-heavy chat ("and what about X?", "summarize that"), the retriever frequently pulls the same top-k chunks again — and you pay to upload them again, on top of the growing history buffer above. Two independent re-sends, stacking, every turn.

The knobs are real — set them on purpose

This isn't a bug and it isn't a strawman: LlamaIndex gives you the lever right there. Cap the buffer explicitly instead of inheriting the 0.75-of-context default:

from llama_index.core.memory import ChatMemoryBuffer

memory = ChatMemoryBuffer.from_defaults(token_limit=1500)

or move to a summarizing/vector memory so old turns and stale tool observations stop riding along. And for the re-retrieval half, CondenseQuestionChatEngine condenses history into one standalone query so you retrieve on the condensed intent instead of re-stuffing chunks each turn, or lower similarity_top_k. The point isn't "LlamaIndex is expensive" — it's that both the default re-send size and the per-turn re-retrieval are decisions the framework makes for you, and they compound.

Measure it before you argue about it

Before you tune anything, put a dollar figure on one real run — priced, not guessed. That's what @wartzar-bee/tokenscope does (npm i @wartzar-bee/tokenscope): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so "the deep agent costs N× the shallow one" stops being a hunch.

If it runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute max-usd ceiling — so a memory-config change doesn't ship as a silent 4× before anyone notices.

- uses: wartzar-bee/ci-guardrail@v1
  with:
    max-usd: "0.50"

If you run LlamaIndex chat or agents: what's your token_limit, how big are the tool observations in that buffer, and how many of the same chunks get re-retrieved across a real session? Worth pricing one real conversation before the next invoice does it for you.

Pydantic AI keeps one growing message list per run — and re-sends the whole thing every step

wartzar-bee — Sat, 22 Aug 2026 12:37:48 +0000

Pydantic AI gives you a clean, typed agent: define an Agent, hand it tools, call agent.run(...), and it loops — model call, tool call, model call — until it produces a validated result. The typed ergonomics are great. What the quickstart doesn't spell out is what the model receives on each pass of that loop.

I read the run graph (pydantic_ai_slim/pydantic_ai/_agent_graph.py on main) to find out. The mechanism is structural, and it's the same shape I found in the OpenAI Agents SDK and smolagents.

One list, appended twice per turn

Each run holds a single mutable conversation list on its state:

message_history: list[_messages.ModelMessage] = dataclasses.field(default_factory=list[_messages.ModelMessage])

On every model step the graph appends to it — first the outgoing request, then the model's response:

ctx.state.message_history.append(self.request)
...
ctx.state.message_history.append(response)

Nothing is removed. The list only grows: request, response, request, response — with tool calls and, crucially, tool outputs riding inside those messages.

The full list is re-sent every step

When the graph builds the input for the next model call, it takes the entire accumulated history — a full copy:

messages = ctx.state.message_history[:]
...
messages[:] = _clean_message_history(ctx.state.message_history)

That [:] is the whole conversation to date. So on step 1 the model sees your prompt; on step 2 it sees your prompt + step 1's request + step 1's response (including the tool output); on step 5 it sees all of that plus steps 2–4. The payload you pay for grows every single step, and the heaviest passengers are usually the tool outputs — the search results, file contents, and API responses you least want re-uploaded five times.

Why it's quadratic, and why nothing warns you

A run of n steps sends roughly 1 + 2 + 3 + … + n copies of history — O(n²) cumulative tokens in the step count. A 3-step agent is fine. A 12-step agent that reads a couple of files is not: each file's contents rides along on every later step. The run still succeeds, your tests still pass — the only artifact is a bigger number on the usage line, and you don't see it until the invoice.

The knob you actually have

Pydantic AI hands you the full transcript back (result.all_messages()) and every run method takes a message_history parameter:

message_history: Sequence[_messages.ModelMessage] | None = None

— "History of the conversation so far." That's the lever: across a multi-turn conversation you decide what prior history to replay, so you can pass a trimmed or summarized history into the next run instead of the raw accumulation. Inside a single deep tool-loop the re-send is inherent to how tool-calling works (it's true of every framework) — which is exactly why the move is to measure it, not assume it's free.

Measure it before you argue about it

Before refactoring anything, put a number on it — the token cost of your run, priced in dollars, not tokens. That's what @wartzar-bee/tokenscope does (npm i @wartzar-bee/tokenscope): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run dollar figure, so "the 12-step version costs 4× the 4-step version" stops being a hunch.

And if this runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute max-usd ceiling — so the quadratic step doesn't reach production as a silent 4× before anyone notices.

- uses: wartzar-bee/ci-guardrail@v1
  with:
    max-usd: "0.50"

If you run Pydantic AI: how many steps does your deepest agent take, and how big are the tool outputs it re-sends on every one? Worth pricing one real run before the next invoice does it for you.

Your compiled DSPy program re-sends up to 20 few-shot demos on every single call

wartzar-bee — Fri, 21 Aug 2026 07:17:22 +0000

DSPy's pitch is that you stop hand-writing prompts and let an optimizer compile them for you. You write a program out of modules, hand it a metric and a trainset, run a teleprompter, and it finds good few-shot examples for each step. It genuinely works. The part the tutorials don't put a number on is what "found good few-shot examples" costs you — not once, but on every call your compiled program makes in production.

Compiling attaches demos. Every call re-sends them.

When you compile with the default optimizer, DSPy bootstraps few-shot demonstrations and pins them onto each predictor:

# dspy/teleprompt/bootstrap.py
def __init__(self, ..., max_bootstrapped_demos=4, max_labeled_demos=16, ...):

That's up to 20 demos per predictor by default — 4 bootstrapped (full input→output traces, including the chain-of-thought rationale) plus up to 16 labeled examples. They live on the compiled program, not in any prompt string you wrote.

Then, on every inference, the module hands all of them to the adapter:

# dspy/predict/predict.py
demos = kwargs.pop("demos", self.demos)

self.demos is the full set the optimizer attached. There's no "use them for the first call only" — each forward() defaults to sending the whole list.

What "sending the demos" actually means in tokens

The adapter turns every demo into a pair of chat messages — a user turn and an assistant turn — and appends all of them ahead of your real input:

# dspy/adapters/base.py — format()
messages.append({"role": "system", "content": system_message})
messages.extend(self.format_demos(signature, demos))
...
# format_demos(), per complete demo:
messages.append({"role": "user", "content": self.format_user_message_content(signature, demo)})
messages.append({"role": "assistant", "content": ...})

So a predictor compiled with 12 demos prepends 24 messages to every call. Because bootstrapped demos carry the full reasoning trace, those messages aren't small. This is fixed overhead you pay on request #1 and request #1,000,000 alike — and it's invisible in your code, because you never wrote those messages. The optimizer did.

And a real program has more than one predictor

DSPy's whole point is composition: a pipeline is several modules — a couple of ChainOfThought steps, a retriever-reader, a router. Each is a predictor, each gets its own demo set, each re-sends it on every call. The per-call prompt overhead is roughly:

demos_per_predictor  ×  predictors  ×  (rationale is long, so each demo is not cheap)

Compile a 4-module pipeline with the defaults and you can be prepending 60–80 demo messages across the pipeline for a single end-user request — none of which appear anywhere in your source.

The knob is real — set it on purpose

This isn't a bug and it isn't a strawman: DSPy hands you the lever at compile time. Choose the demo budget instead of inheriting 4 + 16:

from dspy.teleprompt import BootstrapFewShot

optimizer = BootstrapFewShot(metric=my_metric,
                             max_bootstrapped_demos=2,
                             max_labeled_demos=2)
compiled = optimizer.compile(program, trainset=trainset)

And check what actually got attached before you ship it:

for p in compiled.predictors():
    print(len(p.demos))   # how many few-shot pairs ride along on every call

The point isn't "DSPy is expensive" — it's that the number of demos re-sent per call is a decision the optimizer makes for you, and the default is generous.

Measure it before you argue about it

Before you tune anything, put a dollar figure on one real run of the compiled program — priced, not guessed. That's what @wartzar-bee/tokenscope does (npm i @wartzar-bee/tokenscope): it takes real usage and prices each bucket — input, output, cache-write (~1.25×), cache-read (~0.1×) — into an actual per-run cost, so "the compiled pipeline costs N× the zero-shot one" stops being a hunch.

If it runs in CI, gate it: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on tokenscope) that fails the check when a run crosses an absolute max-usd ceiling — so a recompile that bumps the demo count doesn't ship as a silent 3× before anyone notices.

- uses: wartzar-bee/ci-guardrail@v1
  with:
    max-usd: "0.50"

If you run compiled DSPy programs: how many demos are on each predictor, and how long is each one? Worth pricing one real run before the next invoice does it for you.

Enclave vs. the agent frameworks we audited — it's a layer map, not a fight

wartzar-bee — Thu, 20 Aug 2026 06:10:08 +0000

Enclave vs. the agent frameworks we audited — it's a layer map, not a fight

I spent two weeks auditing five agent frameworks for token cost — AutoGen, CrewAI, LangChain, LangGraph and smolagents. Every write-up ended with the same question in the comments: "so which one should I use — and where does enclave fit?"

The honest answer is that it's the wrong question, because enclave and those frameworks aren't the same kind of thing. They're different layers of the stack. You don't pick enclave instead of CrewAI. You can run a CrewAI crew inside an enclave pod. This post is the map.

The two layers

The frameworks are orchestration DSLs. They're how you wire agent logic — who talks to whom, what tools an agent can call, how a multi-step task is decomposed. AutoGen gives you multi-agent conversations; CrewAI gives you role-based crews; LangGraph gives you a state graph; smolagents gives you a tight code-writing ReAct loop. That's real, valuable work, and enclave does none of it. Enclave has no crew abstraction, no conversation patterns, no graph builder. If you want agents that talk to each other, you reach for a framework.

Enclave is the runtime substrate. It's where an agent process runs and what it's allowed to touch — the container boundary, the network policy, the credential handling, the memory that survives a restart. It's brain-agnostic (README.md:192 — BRAIN=claude | api | local | optimize, one env var), so the model underneath is yours to pick. It doesn't care whether the logic inside is a CrewAI crew, a LangGraph, or a plain script.

Composable, not competing:

┌─────────────────────────────────────────────┐
│  YOUR AGENT LOGIC                            │
│  (AutoGen / CrewAI / LangGraph / smolagents  │   ← the framework: orchestration DSL
│   / a plain loop — your choice)              │
├─────────────────────────────────────────────┤
│  ENCLAVE RUNTIME                             │
│  container isolation · egress policy ·       │   ← the substrate: where it runs,
│  credential vault · memory vault · delegation│      what it can touch
├─────────────────────────────────────────────┤
│  Docker + your host                          │
└─────────────────────────────────────────────┘

The map, side by side

Layer concern	The five frameworks	Enclave
What it is	An orchestration DSL — wires agent logic (conversations, crews, graphs, ReAct loops)	A runtime substrate — a hardened container an agent process runs inside (`README.md:3`)
Multi-agent patterns	Yes — this is their core (AutoGen conversations, CrewAI crews, LangGraph state)	No. Enclave has no crew/graph/conversation abstraction. Run a framework inside it for that.
Default cost failure	All five re-send accumulated context uncapped by default (see each audit)	N/A at this layer — enclave runs whatever logic you give it; the framework's default is still the framework's to fix
Isolation boundary	Not their job — runs in your process, sees your whole env	Kernel-enforced: `--cap-drop=ALL --security-opt=no-new-privileges`, no inbound ports, reads only the mounts you gave it + a read-only `secrets/` (`README.md:13-14`). A prompt injection does not change that.
Network / egress	Not their job	An egress allowlist that logs disallowed hosts, and blocks them under `GUARD_EGRESS_ENFORCE=1` (`README.md:16-20`)
Credential handling	You pass keys in yourself	Read-only mounted `secrets/`; an AES-256 `vault-encrypt` archive with the key kept out of git (`README.md:267`)
Durable memory	Framework memory is in-process (and is the cost problem — see audits)	A git-tracked linked markdown vault, auto-snapshotted each tick (`README.md:247-267`)
Cost discipline built in	Opt-in, one-liner you have to know exists	Manager→worker delegation forces bulk code-writing to a cheap/local worker (`README.md:93`); context epochs bound a session's token spend (`platform/agentd/agentloop.py:136,144`)
Model choice	Framework-dependent	Brain-agnostic BYOM — same container, one env var (`README.md:192`)

How to read this (the honest caveat)

This is not a benchmark leaderboard, and I won't pretend it is. The frameworks and enclave live at different layers, so most cells above aren't a "who wins" — they're "whose job is this." The "default cost failure" numbers from the audit series (AutoGen 5.5×→15.5×, CrewAI 6×, LangChain O(N²), LangGraph 7×, smolagents up to 9.5×) each come from that framework's own worked scenario — different N, different turn counts — so they are not comparable across rows. You cannot read this table as "enclave is cheaper than CrewAI." Enclave isn't in that race; it's the floor the race runs on.

What the map does tell you: the layer where you write agent logic and the layer where an agent runs and is contained are different concerns, and a lot of production pain comes from bolting containment onto a framework after the fact. Pick a framework for the logic. Pick a substrate for the isolation, the egress policy, and the credential boundary.

What enclave deliberately does NOT do

Being category-honest cuts both ways:

No orchestration DSL. No Crew, no StateGraph, no RoundRobinGroupChat. If you want structured multi-agent conversation, use AutoGen (or wire it yourself and run it in a pod).
Egress is report-only by default. The container boundary is always on, but the network allowlist only logs until you set GUARD_EGRESS_ENFORCE=1 (README.md:16-20). Ship it on for anything real.
Public alpha. Apache-2.0, used daily by its authors to run a live fleet, but the API and layout still move (README.md, "Status").

So: which do I use?

Both, usually. The question isn't "enclave or a framework" — it's:

What logic am I building? → a framework (or plain code). Whichever fits your agent's shape.
Where does it run, and what can it reach? → enclave, if you want kernel-enforced isolation, an egress policy, and a credential + memory boundary you didn't have to build.

The frameworks' shared default cost bug (re-sending accumulated context — the whole reason this series exists) is still yours to fix at the framework layer; enclave doesn't magic it away. But when you've fixed it and you want to actually run the thing with credentials and network access it shouldn't fully have, that's the substrate problem, and it's the one enclave is for.

Enclave is open-source (Apache-2.0): github.com/wartzar-bee/enclave. The five-part cost-audit series that grounds the framework column is on dev.to/wartzarbee. Want the numbers for your own project? tokenscope reads your Claude Code logs and shows exactly where the tokens went (npx @wartzar-bee/tokenscope — read-only, nothing leaves your machine), and ci-guardrail fails a PR when a token-cost regression sneaks in. wartzar-bee builds tools for operating cost-efficient autonomous agents.

A cost gate that fails your build should tell you where, not just that

wartzar-bee — Wed, 19 Aug 2026 01:54:06 +0000

Most teams that gate agent cost in CI eventually hit the same wall. You wire up a check that fails the PR when the token/compute cost of an agent step regresses past a budget — good, the surprise bill is now a surprise review comment instead. But the check tells you that cost went up 34%. It doesn't tell you where. So the author reopens the diff, eyeballs 600 changed lines, and guesses which hunk did it.

That guessing step is where the gate quietly loses its teeth. A regression you can't locate in ten seconds is a regression people learn to bump the threshold past.

Three things have made cost gates actually stick for me — none of them tool-specific:

1. Attribute the delta to a diff, then to a hunk. "This PR adds 40k tokens/run" is a start. "This PR adds 40k tokens/run, and 31k of it is the fixture you pasted into prompts/system.md lines 88–140" is a fix. Cost that points at a file is useful; cost that points at the lines is actionable before the reviewer even asks.

2. Put the finding where the reviewer already is. A cost report in a separate comment, or worse in the Actions log, is a second place someone has to remember to look. The reviewer is looking at the Files changed tab. If the cost warning renders inline on the responsible lines — same surface as a failing lint or a type error — it gets read, because it's in the path of the review that's already happening.

3. Rank, don't dump — and don't oversell the number. You don't need an exact per-line tokenizer count to point someone at the right block. A cheap ranking (which added hunks carry the most characters) is enough to say "start here." Use one number as the summary (the whole-change total) and the ranking only to aim inside it. And be honest about what the number is: most of these estimates, including the whole-change total, are a chars/token proxy for relative comparison, not a billing figure. A gate that oversells its precision gets distrusted the first time it's slightly off — and a distrusted gate gets disabled.

The gate itself is the part teams still rebuild by hand every time. If you want it off the shelf: npx @wartzar-bee/tokenscope measures the token/$ cost of a run over just the changed files, and there's an Apache-2.0 GitHub Action (wartzar-bee/ci-guardrail) that fails a PR when an agent step's cost regresses past your budget — and, as of this week, annotates the responsible lines inline on the PR diff, so a blocked build points at the hunk to trim instead of just flashing a percentage.

For those of you gating agent cost in CI already: when a cost check fails, how does the author find which change did it — do you attribute it to the diff, or is it still "the number went up, go hunt"?

Gate your CI on a dollar ceiling, not a percentage — the number your finance team actually asks for

wartzar-bee — Sun, 16 Aug 2026 00:00:12 +0000

Gate your CI on a dollar ceiling, not a percentage — the number your finance team actually asks for

Most cost gates for agent/LLM workflows check a delta: did this PR make the run more expensive than the last one, by more than X%? That's a good regression alarm. But it answers a developer's question ("did I make it worse?"), not a budget owner's question ("are we going to blow the monthly number?").

Those are genuinely different gates, and a team that only has the percentage one keeps getting surprised. A workflow can pass every percentage check — each PR adds a harmless-looking 3% — and still cross the line where the absolute monthly spend stops being okay. Percentages compound quietly; dollars are what shows up on the invoice.

So the second gate I want on any agent workflow is an absolute ceiling: "a single run of this job must not cost more than $N," full stop, regardless of whether it went up or down since yesterday.

Three things make that gate actually usable rather than theater:

1. The ceiling is priced, not token-counted. "Under 2M tokens" is meaningless to the person who signs off on spend, because a token of Opus output and a token of cached Haiku input differ by ~100× in price. The gate has to multiply each token bucket (input, output, cache-write at ~1.25×, cache-read at ~0.1×) by that model's real per-token price and sum to an actual dollar figure. If your gate reports tokens and makes a human convert, nobody converts, and the ceiling drifts.

2. The ceiling is per-run and per-workflow, not global. A nightly full-repo audit and a per-PR lint agent have wildly different legitimate costs; one global number is either too loose for the small job or too tight for the big one. You want to set max-usd on the specific workflow, so each job carries the ceiling that matches what it's for.

3. It shows the headroom, not just pass/fail. "$0.43 of a $0.50 ceiling — 86%" on every run is the line that lets you move the limit before it starts failing builds, instead of discovering the wall by hitting it. A gate that only says "blocked" the first time you cross is a gate people rip out.

The percentage gate and the dollar gate aren't competitors — I run both. The percentage one catches the sudden regression (someone pasted a 50k-token fixture into the prompt); the dollar one catches the slow one (the workflow that was always a bit expensive and finally crossed what the budget can absorb). Different failure modes, and the second is the one your finance owner will actually ask you about.

If you want it off the shelf: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (built on @wartzar-bee/tokenscope) that prices token usage into real dollars per run and can gate on an absolute max-usd ceiling as well as a delta — so you can wire the "don't exceed $N per run" check straight into the workflow that needs it. But the mental model is the point whatever tool you reach for: a percentage answers the developer's question, a dollar ceiling answers the budget owner's.

For those of you already gating agent cost: do you gate on the relative delta, an absolute dollar ceiling, or both — and if you had to pick one to start with, which caught more real problems for you?

I scanned the code of 25 MCP servers — what they can do to your machine before you sandbox them

wartzar-bee — Sat, 15 Aug 2026 11:13:54 +0000

An MCP server isn't a remote API you call over HTTPS. For the stdio servers that make up most of the popular ones, npx some-mcp-server downloads code and runs it as a local process — with your shell's environment, your filesystem, and your network. Your agent then hands that process instructions.

That's a normal, useful design. It's also a capability surface nobody itemizes for you. So I measured it.

Method (static source scan — honest about its limits)

I pulled 25 popular Model Context Protocol server packages and statically scanned the published package code (not runtime behavior) for four things:

net-call sites — does the code make outbound network calls?
secret-env reads — does it read credentials from process.env (e.g. process.env.API_KEY)?
subprocess spawns — does it shell out / spawn child processes (exec_shapes)?
bundled binaries — does the package ship a prebuilt binary?

Static scanning tells you what a package can reach for, not what it does on a given run. A process.env.API_KEY read is almost always the server loading its own credential — completely legitimate. The point isn't "these are malicious." The point is the aggregate surface you're granting, unseen, every time you add a server.

The capability surface, across 25 servers

Capability (code contains…)	Servers	of 25
A network-call site	17	68%
Reads a secret from `process.env`	16	64%
A hardcoded outbound host reference	12	48%
Spawns a subprocess / shells out	8	32%
Ships a bundled binary	4	16%

So a typical multi-server agent setup is running a dozen-plus local processes, most of which read your environment and make network calls, several of which can spawn subprocesses — all with the same privileges as the shell you launched them from. No install step surfaces this. No directory lists it.

Why this matters more for agents than for humans

When you run a CLI tool, you roughly know what it does. When your agent connects a fleet of MCP servers and then acts in a loop, the blast radius is: every server's capability × the agent's autonomy × however many turns the loop runs. One over-scoped server, one compromised dependency in one of those packages, and the credential in process.env is one network call away from leaving your machine — and until recently a wildcard like *.amazonaws.com on an allowlist would have waved it straight through to an attacker-nameable S3 bucket.

You don't fix this by auditing 25 packages by hand every time you add one. You fix it by running the servers (and the agent) in a sandbox that constrains filesystem, egress, and secrets by default — so a server's capability stops being an authority.

That's exactly what enclave is: an Apache-2.0 sandboxed runtime for agents, with a default-deny egress allowlist, secret redaction, and a guard that blocks the sandboxed process from rewriting its own guardrails. Point your MCP-using agent at it and the capability surface above becomes something you grant explicitly instead of something you inherit silently.

And if you want the other half of the bill — the token context-tax those same servers add to every request (I measured a 544× spread, from 35 to 19,054 tokens/request) — that's tokenscope: npx @wartzar-bee/tokenscope.

Numbers are from a static source scan of 25 published MCP server packages; every count is reproducible from the package code. It's a lower bound on capability, not an accusation about any specific server's behavior.

You can see your cloud bill. Can you see what your AI agent's context costs?

wartzar-bee — Tue, 11 Aug 2026 08:15:53 +0000

You can see your cloud bill. You can see your CI minutes tick down. But the fastest-growing line item in an AI-agent app is the one number you can't see: how many tokens your agent's context is worth on every single model call.

It's invisible because it never shows up as one big charge. It's a few thousand tokens, resent hundreds of times a day, quietly compounding. By the time it's a real number on the invoice, it's baked into every request you make.

Here's how to read it in ten seconds — no account, no logs, no config:

npx @wartzar-bee/tokenscope scan .

Point it at the directory that holds your agent's prompts, tools, and configs. It prints the token footprint and the files responsible:

tokenscope scan — src
Estimated token footprint: 8,454 tokens across 6 files
(estimate ≈ 4 chars/token — a tokenizer-free proxy for relative comparison, not a billing figure)

Top files by estimated tokens:
      2991  share.mjs
      2134  scan.mjs
      1092  core.mjs
       835  report.mjs
       718  pricing.mjs
       684  benchmark.mjs

That's the whole point: the top file is usually a system prompt, a tool schema, or a wall of few-shot examples that someone added "just to be safe." Now you can see which one, and how much it weighs, before it's part of every call.

Why this is the number that bites

Most agent frameworks resend the accumulated context on each step — memory, history, tool definitions, the lot. So a prompt that's 2,000 tokens heavier isn't a one-time cost; it's 2,000 tokens × every call × every user. The per-call log looks fine. The monthly bill does not. (I watched an agent on a timer burn 136M tokens overnight doing almost nothing — same root cause.)

tokenscope gives you a single, reproducible footprint number so a "let's just add this to the prompt" PR stops being invisible.

Three ways to use it

1. Ad-hoc, right now — measure any repo before you ship a prompt change:

npx @wartzar-bee/tokenscope scan ./agent

2. As a local cost gate — fail your own commit if the footprint blows a budget, so a runaway prompt never leaves your machine:

# .git/hooks/pre-push  (chmod +x)
npx @wartzar-bee/tokenscope scan . --max-total 50000 || {
  echo "Context footprint over budget — trim it before pushing."; exit 1;
}

Using the pre-commit framework? It's a four-line entry — no hook scripting.

3. On every PR, in CI — the same check as a GitHub Action that comments the token-cost delta on the responsible files and (optionally) blocks the merge:

- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    mode: warn   # report-only until you trust it; switch to block later

That's ci-guardrail — tokenscope wired into your pipeline.

Get it

Run it: npx @wartzar-bee/tokenscope scan .
npm: @wartzar-bee/tokenscope
Source (Apache-2.0): github.com/wartzar-bee/tokenscope

It's free, open-source, and tokenizer-free — an estimate for relative comparison, not a billing oracle, so you can run it on any codebase without wiring up a provider SDK. If it saves you one "why is the bill up 40%?" afternoon, it did its job.

I scanned 25 popular MCP servers — the hidden context-tax spans 544

wartzar-bee — Mon, 10 Aug 2026 17:28:51 +0000

Every MCP server you connect ships a set of tool-definition schemas. Those schemas get injected into every single model request for the whole session — before you ask the agent anything. That's a fixed tax on your context window and your bill, and no MCP directory lists it.

So I measured it. I pulled 25 popular Model Context Protocol servers, spawned each one locally, ran the real tools/list handshake, and counted the tokens its tool schemas add per request (chars/4, the same heuristic tokenscope uses — labelled an estimate). 16 of the 25 register their tools without live credentials, so those are measured; the rest need OAuth/remote transport or real keys and are marked not measured, never estimated.

The context-tax, measured

Server	Tools	Tokens / request
notion-mcp	24	19,054
kubernetes	23	5,964
github	26	4,242
memory	9	2,875
everything	13	2,031
tavily	5	1,924
sequential-thinking	1	1,176
browsermcp	12	1,052
slack	8	822
google-maps	7	704
puppeteer	7	648
exa	2	535
brave-search	2	375
gitlab	9	371
perplexity-ask	1	149
postgres	1	35

That's a 544× spread — from 35 tokens to just over 19,000, paid on every request for the life of the session. Connect notion-mcp and github together and you're near ~23k tokens of pure schema overhead before a single user turn. On a long agent run that's not a rounding error; it's most of your context budget and a real line on the bill.

The lesson isn't "don't use big servers." It's that the cost is invisible at connect time and none of the directories that rank these servers by stars/installs will tell you. You find out from the invoice.

While I had the packages open, I also scanned what they do

Same 25 servers, static source scan (nothing executed): network egress, secret-env reads, exec/shell surface. A few things worth knowing:

16 of 25 read a secret env var; 8 have an exec/shell surface. upstash-context7-mcp reads OPENAI_APPS_CHALLENGE_TOKEN + CLIENT_IP_ENCRYPTION_KEY and egresses login.microsoftonline.com.
4 of 25 ship a bundled/minified dist (exa has 293k-char lines; notion 582k). You can't statically separate a minified blob into call-site hosts vs embedded strings, so I label those bundled — egress not statically resolvable rather than render a misleading "clean." exa's bundle references an undisclosed api.agnost.ai alongside api.exa.ai.

Getting the egress scan honest at 25 servers was most of the work — naive scans count comment/jsdoc URLs as egress (one server "phoned home" to wikipedia.org and rfc-editor.org... from doc-strings). A host only counts as egress if its URL literal sits at an actual network call site or a declared API base.

Measure your own

The context-tax above is server-side and fixed. Your own agent's per-session cost — which server's schemas are eating your window, what a run actually cost — you can measure directly:

npx @wartzar-bee/tokenscope --demo   # zero-setup: a sample cost report, no session needed
npx @wartzar-bee/tokenscope          # or point it at your most recent Claude Code session

And if you want to stop cost regressions before they ship, wartzar-bee/ci-guardrail gates token cost in CI the same way you'd gate a failing test.

Everything here is reproducible from published npm packages — npm pack the servers, run the scanner, spawn each for the live tools/list. Found a cell that's wrong? The method is open; correct it.

Part of wartzar-bee — tools for building and operating cost-efficient autonomous agents. Apache-2.0.

The cheapest way to try a CI cost gate: run it in warn mode for a week

wartzar-bee — Mon, 03 Aug 2026 09:00:47 +0000

The cheapest way to try a CI cost gate: run it in warn mode for a week

The objection I hear most about gating agent cost in CI isn't "I don't need it." It's "I'm not wiring up a new check that can fail my builds on day one." Fair. A gate that red-X's a PR the first afternoon it's installed — over a threshold nobody has calibrated yet — gets ripped back out by Friday.

So the way I actually roll one out now is: install it in report-only mode first, and let it be wrong for a week before it's allowed to block anything.

Concretely, that's three properties I'd look for in any cost gate, not just the one I work on:

1. A report-only mode that never fails the build. You want the cost delta to show up on every PR — the number, the top files, the trend — while the check itself stays green no matter what. That's the whole evaluation period: you're watching whether the gate's numbers track reality on your repo before you give it teeth. If a tool only ships in block-or-nothing form, you're calibrating in production against angry teammates.

2. A noise floor, so tiny diffs don't cry wolf. The commonest false positive for a percentage gate is a small base: a helper that goes from 10 tokens to 30 is +200%, and a naive gate fails the build over a fraction of a cent. You want an absolute floor — "only flag when the real increase is at least N tokens" — so the gate speaks up about the fixture someone pasted in, not about a two-line refactor.

3. A report that shows up even when there's no PR. Pushes to main, scheduled runs, manual dispatches — those have no PR to comment on, but they're exactly where a slow cost creep hides. A gate that also writes its table to the run summary (GitHub's $GITHUB_STEP_SUMMARY) keeps a cost record on every event, not just on pull requests.

The rollout that sticks: add it with enforcement off, watch the numbers for a sprint, set the threshold where it would have caught the two regressions you actually remember, then flip it to blocking. By then nobody argues with it, because they've watched it be right.

If you want that off the shelf: wartzar-bee/ci-guardrail is an Apache-2.0 GitHub Action (powered by @wartzar-bee/tokenscope) with a mode: warn that does exactly the report-only-never-fail thing, and it writes the same cost table to the Actions run summary on any event (push, schedule, manual) — not just on PRs — so the first PR you point it at can't break, and you decide when it earns the right to block.

The whole rollout is one step — report-only from the first PR:

# .github/workflows/agent-cost.yml
- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    mode: warn            # report the cost delta on every PR, never fail the build
    threshold-pct: 20      # the line it *would* flag — tune it, then flip mode off to enforce
    min-delta-tokens: 500  # noise floor: ignore tiny diffs so small bases don't cry wolf

For those of you who gate cost (or coverage, or bundle size) in CI: how long do you run a new gate in warn/report-only before you let it block — a week, a sprint, longer? Or do you just turn it to blocking on day one and eat the noise?

AutoGen's hidden token tax: why a 3-agent chat costs 15 what you expect

wartzar-bee — Thu, 30 Jul 2026 15:37:46 +0000

AutoGen's hidden token tax: why a 3-agent chat costs 15× what you expect

Cost-audit series, episode 2. This series began with an AI agent that burned 136M tokens overnight →.

AutoGen is Microsoft's multi-agent framework. It's genuinely good at orchestrating agents that hand off work to each other. But its default memory model has a cost shape that surprises almost every team that hits it in production.

This audit shows you exactly where the tokens go, with line numbers.

The setup: a 3-agent RoundRobin chat

The canonical AutoGen pattern is a RoundRobinGroupChat with N agents taking turns on a task. Here's the minimal version from the docs:

from autogen_agentchat.agents import AssistantAgent
from autogen_agentchat.teams import RoundRobinGroupChat
from autogen_agentchat.conditions import MaxMessageTermination

planner  = AssistantAgent("planner",  model_client=client, system_message="You plan.")
coder    = AssistantAgent("coder",    model_client=client, system_message="You code.")
reviewer = AssistantAgent("reviewer", model_client=client, system_message="You review.")

team = RoundRobinGroupChat(
    [planner, coder, reviewer],
    termination_condition=MaxMessageTermination(max_messages=10),
)
await team.run(task="Build a web scraper for Hacker News.")

Three agents, 10 turns total (~3–4 turns each). Seems cheap. It isn't.

The default context: unbounded, per-agent

Every AssistantAgent gets its own UnboundedChatCompletionContext by default:

# autogen-agentchat/src/autogen_agentchat/agents/_assistant_agent.py, __init__
if model_context is not None:
    self._model_context = model_context
else:
    self._model_context = UnboundedChatCompletionContext()

source

UnboundedChatCompletionContext.get_messages() returns self._messages — the full list, no cap, no truncation:

# autogen-core/.../model_context/_unbounded_chat_completion_context.py (a ~20-line file)
async def get_messages(self) -> List[LLMMessage]:
    """Get at most `buffer_size` recent messages."""
    return self._messages

source

(The docstring says "at most buffer_size" — that's a copy-paste artifact from BufferedChatCompletionContext. There is no buffer. It returns everything.)

The handoff tax: every agent sees every message

When an agent's turn arrives, on_messages_stream adds all incoming messages to its own context before calling the LLM:

# _assistant_agent.py, in on_messages_stream (STEP 1: "Add new user/handoff messages
# to the model context")
await self._add_messages_to_context(
    model_context=model_context,
    messages=messages,          # ← the full message_thread from the group manager
    ...
)

And _add_messages_to_context appends each one:

# _assistant_agent.py, static method _add_messages_to_context
await model_context.add_message(llm_msg)
...
await model_context.add_message(msg.to_model_message())

source

The group manager (BaseGroupChatManager) maintains a single _message_thread and appends every response to it:

# _base_group_chat_manager.py
self._message_thread: List[BaseAgentEvent | BaseChatMessage] = []
...
await self.update_message_thread(delta)   # called after every agent response

source

So at turn T, the agent receiving the baton gets T-1 messages added to its already-growing context. Its context now contains everything it has ever seen.

The math: O(N × T²) total tokens

Let's be precise. Define:

T = total turns in the conversation
N = number of agents
m = average tokens per message (system prompt + response, ~300 tokens is realistic for a coding task)

Each agent speaks every N turns. When agent i speaks on turn t, its context contains all t-1 prior messages (because it has been accumulating them since turn 1).

Tokens consumed by agent i on turn t:

context_tokens(t) = (t - 1) × m

Total tokens for agent i across all its turns (it speaks at turns N, 2N, 3N, … up to T):

Σ (kN - 1) × m  for k = 1 to T/N
≈ m × N × (T/N)² / 2
= m × T² / (2N)

Total tokens across all N agents:

N × m × T² / (2N) = m × T² / 2

The N cancels. Total cost scales as T² regardless of how many agents you add.

Worked example: 10 turns, 3 agents, 300 tokens/message

Turn	Agent	Context size (messages)	Tokens in this call
1	planner	0 prior + system	~300
2	coder	1 prior + system	~600
3	reviewer	2 prior + system	~900
4	planner	3 prior + system	~1,200
5	coder	4 prior + system	~1,500
6	reviewer	5 prior + system	~1,800
7	planner	6 prior + system	~2,100
8	coder	7 prior + system	~2,400
9	reviewer	8 prior + system	~2,700
10	planner	9 prior + system	~3,000
Total			~16,500 tokens

Naïve expectation (10 calls × 300 tokens each): 3,000 tokens

Actual: ~16,500 tokens — 5.5× more.

At 20 turns it's ~63,000 tokens vs 6,000 expected — 10.5× more.

At 30 turns: ~139,500 tokens vs 9,000 — 15.5× more.

The multiplier grows linearly with T. This is the same O(T²) shape as ConversationBufferMemory in LangChain — but AutoGen's version is per-agent, so it's easy to miss in per-call logs.

Why per-call logs hide this

If you're watching your LLM provider's per-call token counts, you see something like:

call 1:  300 tokens  ✓ cheap
call 2:  600 tokens  ✓ fine
call 3:  900 tokens  ✓ ok
...
call 10: 3,000 tokens  ← this one looks expensive

Each call looks like a modest increase. The cumulative total — 16,500 — only shows up when you sum across the run. Most observability dashboards show per-call costs, not per-run totals. The runaway is invisible until the bill arrives.

The fix: cap the context

AutoGen ships two bounded alternatives, named in the AssistantAgent class docstring (around L178 of _assistant_agent.py): BufferedChatCompletionContext (limits message count) and TokenLimitedChatCompletionContext (limits tokens):

Option 1: `BufferedChatCompletionContext` (sliding window)

from autogen_core.model_context import BufferedChatCompletionContext

coder = AssistantAgent(
    "coder",
    model_client=client,
    model_context=BufferedChatCompletionContext(buffer_size=5),  # last 5 messages
)

Cost shape becomes O(T × buffer_size) — linear. For buffer_size=5 and 30 turns: ~42,000 tokens vs 139,500 unbounded. 3.3× cheaper.

Option 2: `TokenLimitedChatCompletionContext` (token budget)

from autogen_core.model_context import TokenLimitedChatCompletionContext

coder = AssistantAgent(
    "coder",
    model_client=client,
    model_context=TokenLimitedChatCompletionContext(token_limit=2000),
)

Caps the context at a fixed token budget. More predictable than a message count because message sizes vary.

Which to use?

Scenario	Recommendation
Short tasks (≤10 turns)	Default is fine; monitor cumulative cost
Long tasks (>10 turns)	`BufferedChatCompletionContext(buffer_size=8–12)`
Strict cost budget	`TokenLimitedChatCompletionContext(token_limit=N)`
Need full history	Default + add per-run cost alerting (see below)

Detecting this in CI before it hits production

The pattern is detectable statically: any file that instantiates AssistantAgent without a model_context= argument is using the unbounded default.

# Flag unbounded AssistantAgent instantiations
grep -rn "AssistantAgent(" src/ | grep -v "model_context="

For dynamic detection — measuring actual token growth across a run — this is exactly what tokenscope does: it instruments LLM calls, tracks per-run cumulative cost, and can block a CI build when a PR's token delta exceeds a threshold.

The wartzar-bee/ci-guardrail GitHub Action wraps tokenscope into a one-line workflow addition:

- uses: wartzar-bee/ci-guardrail@v1
  with:
    github-token: ${{ secrets.GITHUB_TOKEN }}
    threshold-pct: 20   # block if the PR's predicted token cost rises >20% vs base

Summary

	Naïve expectation	Actual (unbounded)	With BufferedContext(5)
10 turns, 3 agents	3,000 tokens	~16,500 tokens	~12,000 tokens
20 turns, 3 agents	6,000 tokens	~63,000 tokens	~27,000 tokens
30 turns, 3 agents	9,000 tokens	~139,500 tokens	~42,000 tokens

The default UnboundedChatCompletionContext is correct for short tasks and full-history use cases. It becomes a cost trap in long multi-agent conversations. The fix is one constructor argument — but you have to know to add it.

The broader pattern: every major agent framework defaults to unbounded context because it's the safest correctness choice. Cost is a second-class citizen in the default config. That's the gap this series documents.

Next in the series: CrewAI — the delegation overhead. How hierarchical agent trees multiply your token bill.

tokenscope on npm · wartzar-bee/ci-guardrail · @wartzarbee on dev.to

DEV Community: wartzar-bee

Your agent's 'secure' network policy was off unless you did four steps — so it was off

The concrete bug: "scoped credentials" that weren't

The pattern: a probe that returns exists(), not works()

Why a cost-efficiency project cares about this

LlamaIndex re-retrieves your chunks — and re-sends up to 75% of your context — on every chat turn

The default memory keeps everything, and hands the model all of it

And the ceiling is not small

And in a RAG chat, retrieval re-runs every turn

The knobs are real — set them on purpose

Measure it before you argue about it

Pydantic AI keeps one growing message list per run — and re-sends the whole thing every step

One list, appended twice per turn

The full list is re-sent every step

Why it's quadratic, and why nothing warns you

The knob you actually have

Measure it before you argue about it

Your compiled DSPy program re-sends up to 20 few-shot demos on every single call

Compiling attaches demos. Every call re-sends them.

What "sending the demos" actually means in tokens

And a real program has more than one predictor

The knob is real — set it on purpose

Measure it before you argue about it

Enclave vs. the agent frameworks we audited — it's a layer map, not a fight

Enclave vs. the agent frameworks we audited — it's a layer map, not a fight

The two layers

The map, side by side

How to read this (the honest caveat)

What enclave deliberately does NOT do

So: which do I use?

A cost gate that fails your build should tell you where, not just that

Gate your CI on a dollar ceiling, not a percentage — the number your finance team actually asks for

Gate your CI on a dollar ceiling, not a percentage — the number your finance team actually asks for

I scanned the code of 25 MCP servers — what they can do to your machine before you sandbox them

Method (static source scan — honest about its limits)

The capability surface, across 25 servers

Why this matters more for agents than for humans

You can see your cloud bill. Can you see what your AI agent's context costs?

Why this is the number that bites

Three ways to use it

Get it

I scanned 25 popular MCP servers — the hidden context-tax spans 544

The context-tax, measured

While I had the packages open, I also scanned what they do

Measure your own

The cheapest way to try a CI cost gate: run it in warn mode for a week

The cheapest way to try a CI cost gate: run it in warn mode for a week

AutoGen's hidden token tax: why a 3-agent chat costs 15 what you expect

AutoGen's hidden token tax: why a 3-agent chat costs 15× what you expect

The setup: a 3-agent RoundRobin chat

The default context: unbounded, per-agent

The handoff tax: every agent sees every message

The math: O(N × T²) total tokens

Worked example: 10 turns, 3 agents, 300 tokens/message

Why per-call logs hide this

The fix: cap the context

Option 1: BufferedChatCompletionContext (sliding window)

Option 2: TokenLimitedChatCompletionContext (token budget)

Which to use?

Detecting this in CI before it hits production

Summary

The pattern: a probe that returns `exists()`, not `works()`

Option 1: `BufferedChatCompletionContext` (sliding window)

Option 2: `TokenLimitedChatCompletionContext` (token budget)