Disclosure: I build Cognocient AI, an LLM proxy that does this. The post is mostly about the approaches, which apply whatever you use.
Most cost tools report spend after the provider has billed you. For agents that's too late, because a loop can spend hundreds of dollars in minutes. Checks that work before the call goes out:
- Per-run budgets. Each call in a loop looks cheap; only the run total is the problem. Tag every call in a run with one run ID and enforce the budget across all of them.
- Loop detection. Stop a session after N identical tool calls in a row, or a streak of errors. (In our product: 3 identical calls, Growth plan.)
- Velocity limits in dollars, not just requests. Tokens per minute and dollars per minute catch a flood that request-count limits miss when calls get long.
- Reserve before calling. Reserve the expected cost atomically before forwarding, then settle with the real cost afterwards. Otherwise 20 parallel calls all pass a budget check that only one of them should have passed.
- Read/write-aware degrading. When the budget runs out, hard-stop calls whose tools look like writes, and let read-only calls continue on a cheaper model.
- A kill switch. One button that blocks everything, for when you're not sure what's happening.
Trade-off: all of this means sitting in the request path. That's a code change (one base URL) and some latency. If you only need reports after the fact, a read-only billing integration is simpler.
from openai import OpenAI
client = OpenAI(base_url="https://api.cognocient.com/v1", api_key="<cognocient key>")
client.chat.completions.create(
model="gpt-4o-mini",
messages=[...],
extra_headers={
"X-Cost-Feature": "support-agent",
"X-Cost-Run-ID": run_id, # one budget across every call in this run
},
)
Top comments (1)
The 'reserve before calling' point is the one most cost tools miss. Parallel calls are the killer — twenty requests all pass a budget check that only one of them should have passed. Atomic reservation is the only way to prevent that race condition.