DEV Community: Mandar S

Stopping a runaway AI agent before the provider bills you

Mandar S — Sun, 01 Nov 2026 13:40:31 +0000

Disclosure: I build Cognocient AI, an LLM proxy that does this. The post is mostly about the approaches, which apply whatever you use.

Most cost tools report spend after the provider has billed you. For agents that's too late, because a loop can spend hundreds of dollars in minutes. Checks that work before the call goes out:

Per-run budgets. Each call in a loop looks cheap; only the run total is the problem. Tag every call in a run with one run ID and enforce the budget across all of them.
Loop detection. Stop a session after N identical tool calls in a row, or a streak of errors. (In our product: 3 identical calls, Growth plan.)
Velocity limits in dollars, not just requests. Tokens per minute and dollars per minute catch a flood that request-count limits miss when calls get long.
Reserve before calling. Reserve the expected cost atomically before forwarding, then settle with the real cost afterwards. Otherwise 20 parallel calls all pass a budget check that only one of them should have passed.
Read/write-aware degrading. When the budget runs out, hard-stop calls whose tools look like writes, and let read-only calls continue on a cheaper model.
A kill switch. One button that blocks everything, for when you're not sure what's happening.

Trade-off: all of this means sitting in the request path. That's a code change (one base URL) and some latency. If you only need reports after the fact, a read-only billing integration is simpler.

from openai import OpenAI

client = OpenAI(base_url="https://api.cognocient.com/v1", api_key="<cognocient key>")

client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[...],
    extra_headers={
        "X-Cost-Feature": "support-agent",
        "X-Cost-Run-ID": run_id,      # one budget across every call in this run
    },
)

Your Anthropic bill doesn't fit FOCUS. Here's how to convert it (and the cached-token trap)

Mandar S — Sun, 01 Nov 2026 13:35:30 +0000

FOCUS is the FinOps Foundation's open billing format. AWS, Azure and Google Cloud can export it, so cloud costs from different providers land in the same columns. AI providers don't export it yet, and their CSVs don't map cleanly.

The cached-token trap

Anthropic's Console splits input tokens into four buckets, each at a different price:

Token type	Price relative to base input
Uncached input	1x
Cache read	0.1x
Cache write, 5 minutes	1.25x
Cache write, 1 hour	2x

OpenAI does it differently: cached tokens are counted inside prompt tokens. If you treat both exports the same way, you either count cached input twice (OpenAI) or lose the cache writes (Anthropic). Cache writes are billed, not just reported, which surprises people.

One row per meter

The fix is to give every token type its own row, its own SKU and its own unit price, like a cloud bill does for each meter. Here's one real day from my own October Anthropic export after conversion (a few of the 40 columns shown; prices are per million tokens):

ChargePeriodStart	SkuId	SkuMeter	ConsumedQuantity	ListUnitPrice	BilledCost
2026-10-28	anthropic/claude-sonnet-4-5/input	Input Tokens	52	3.00	0.000156
2026-10-28	anthropic/claude-sonnet-4-5/cache-read	Cache Read Input Tokens	5,775	0.30	0.0017325
2026-10-28	anthropic/claude-sonnet-4-5/cache-write-5m	Cache Write Input Tokens (5 min)	1,925	3.75	0.00721875
2026-10-28	anthropic/claude-sonnet-4-5/output	Output Tokens	16	15.00	0.00024
2026-10-28	anthropic/claude-haiku-4-5/cache-write-5m	Cache Write Input Tokens (5 min)	23,016	1.25	0.02877

Look at the Sonnet rows: 52 uncached input tokens, but 7,700 tokens went through the cache. A converter that only reads "input tokens" reports 52 and misses most of that day's input cost. The 1-hour cache write gets its own .../cache-write-1h row when it appears; this month's file didn't have any.

The total of the rows has to match the provider's own figure. If the computed price doesn't round to the cost in the export, the converter trusts the provider's number and records that it did so.

For the whole month: 41 rows, $1.06 in FOCUS BilledCost against $1.06 on Anthropic's own cost export.

Try it

For Anthropic, pass both the Cost and the Usage exports from the Console:

npx ai-bill-to-focus convert cost.csv usage.csv -o focus.csv
npx ai-bill-to-focus check focus.csv

Or in the browser, where the file never leaves your machine: https://www.cognocient.com/tools/focus-checker

What isn't tested yet

Anthropic is tested on real Console exports. OpenAI and Together AI are built from their published formats, and I haven't had a real file. If you have one, npx ai-bill-to-focus report <your files> prints a summary with no dollars or account names that you can paste into an issue.

Repo (MIT): https://github.com/mandarvshinde/ai-bill-to-focus

I build Cognocient AI, which tracks AI spend per feature, team and customer. This tool is standalone and needs no account.

I built an async wrapper for OpenAI/Anthropic SDKs because I didn't want a proxy in my request path

Mandar S — Sat, 08 Aug 2026 12:46:54 +0000

I kept running into the same tradeoff building cost tooling for teams shipping LLM features. Every attribution tool in this space works the same way: you point base_url at a proxy, and it sees every call before it happens. That's genuinely useful if you want to block or downgrade a call before it fires. It also means the proxy's uptime is now your uptime, and you've added a network hop to every single request.

I wanted the attribution without touching the request path at all. So I wrote a wrapper instead.

Cognocient wraps the OpenAI and Anthropic Python clients directly. You still call client.chat.completions.create() exactly the way you always did. The wrapper times the call, then fires a cost report on a background thread after your real response has already returned to your code.

The part I actually spent the most time on wasn't the happy path, it was making sure a dead reporting endpoint can never touch your application. If Cognocient's ingestion API is slow, down, or just does not exist, that failure has to stay invisible to whatever you're building. No exception bubbling up, no retry pile-up blocking your real call. There is a test, test_reporter_failure_isolation.py, that specifically points the reporter at an unreachable host and asserts the actual API call still returns clean. Writing that test is what convinced me the design was sound, not the other way around.

What you get: per-call cost, and tags for feature/team/user if you want chargeback-style reporting later.

What you do not get, on purpose: pre-call blocking. If a call is about to blow your budget, this wrapper finds out after it already happened, same as any billing dashboard does. If you need to stop a call before it fires, you want a proxy, and honestly LiteLLM or Portkey do that well. This is for people who've already decided visibility without a critical-path dependency is the right tradeoff for their setup.

Also worth knowing before you reach for it: streaming responses (stream=True) aren't reported yet. If most of your traffic streams, this won't give you complete numbers right now. Non-streaming is solid, I would call it production-safe there. Streaming is next.

from cognocient import CognocientOpenAI as OpenAI

client = OpenAI(api_key="sk-...", cognocient_key="sk-cog-...")
# everything else about the client works exactly like before

MIT licensed, signed provenance on the PyPI release, genuinely early (v0.1.x). If you're already routing through a gateway and it is working for you, this probably isn't for you. If you have been putting off cost visibility specifically because you didn't want another hop in the path, I would like to hear if this is useful or if I have missed something obvious.

GitHub: https://github.com/mandarvshinde/cognocient-python-wrapper
PyPI: pip install cognocient