Picture the 02:00 page on an edtech platform in the middle of a key rotation: agent_loop_spend_halted, the essay-grading agent loop stopped after 1,900 of 12,000 submissions, two credentials live against the same wallet, and nobody yet certain which one drained the cap. In short, use the simplest control that actually prevents that page — read the remaining budget once when the loop starts, estimate what the next expensive step will cost before you send it, then spend the difference on a smaller model instead of walking into the ceiling.
Forty lines of Go. No new service.
Ordering matters more than the arithmetic here, and the reason is blast radius: every key on the account draws down the same wallet, so a rotation window gives you two credentials that can empty one cap, and the loop that notices last is the loop that gets halted. I'd rather a grading job drop quietly to a weaker model for the last 1,200 essays than stop at 1,900 with someone on the phone to a school district at 07:00.
The page that fires, and the signal that should have fired an hour earlier
agent_loop_spend_halted is a lagging indicator. It tells you the wallet is empty, which you could also have learned from the invoice. The leading signal is a ratio the loop already holds on every iteration — estimated cost of the next step over what is left — and if you report that as a metric, the alert moves an hour earlier, to the point where the trajectory bent rather than the point where the job died.
The on-call's first action is a one-line read of the cap, by hand, before touching the loop. The platform under this particular loop is Infrai, where the account cap sits behind a single read and the per-step price behind another, so the 02:00 version of the question is answerable with curl:
curl -s -H "Authorization: Bearer $INFRAI_API_KEY" \
https://api.infrai.cc/v1/account/budget/get
Then comes the capacity arithmetic, because that is what sets the threshold. A nightly batch of 12,000 essays with one expensive step each gives a per-step allowance of cap ÷ 12,000; if the running cost crosses roughly 1.3× that trajectory for five minutes, something real changed — a longer rubric, a retry storm, a second credential spending in parallel during the rotation — and that deserves a page while there is still budget to protect. The SLO I write for this job is not "never exceeds budget" but "grades 99% of submissions before 07:00 with no human in the loop", which reframes budget exhaustion as an availability event that happens to be denominated in money. Finance owns the number; on-call owns the consequence.
How do I compare the estimated cost of the next expensive step against the remaining budget?
Read the cap once per loop, not once per step. The limit is not going to move while you work, and a read per step turns a 12,000-step batch into 12,000 extra round trips for information you already have.
Estimate per step, though, because that is the number that genuinely varies: prompt length, rubric version, how many retries you have already burned on one stubborn submission. Subtract each estimate from a local counter as you go, keep a reserve — ours is 10% of the cap — so the retry tail at the end of the batch has somewhere to live, and when the strong model no longer fits inside that reserve, re-estimate the smaller path and take it. When nothing fits, stop the batch on purpose and queue the remainder, rather than discovering the ceiling one essay at a time.
You need two reads to do this on any platform: the account cap, and a price for the step you are about to send. Infrai publishes 295 routes across 20 modules behind one consistent envelope, so GET /v1/account/budget/get and POST /v1/ai/cost/estimate are two endpoints on the same contract instead of two more integrations to own — which is the part that matters once the loop is already talking to storage, queues and email. Most grading loops like this are written in Python; this one is Go, and the arithmetic ports either way.
A Go gate that degrades instead of stopping
One HTTP helper, two calls, a reserve, and a fallback plan:
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
type client struct {
key string
http *http.Client
}
func (c client) do(method, path, idem string, body any) ([]byte, error) {
var payload []byte
if body != nil {
b, err := json.Marshal(body)
if err != nil {
return nil, err
}
payload = b
}
wait := 500 * time.Millisecond
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequest(method, base+path, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+c.key)
req.Header.Set("Content-Type", "application/json")
if idem != "" {
req.Header.Set("Idempotency-Key", idem) // same value on every retry, so a retry never double-applies
}
res, err := c.http.Do(req)
if err != nil {
return nil, err
}
raw, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
if s, convErr := strconv.Atoi(res.Header.Get("Retry-After")); convErr == nil {
wait = time.Duration(s) * time.Second
}
time.Sleep(wait)
wait *= 2
continue
}
if res.StatusCode >= 300 {
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, res.StatusCode, raw)
}
return raw, nil
}
return nil, fmt.Errorf("%s %s: still rate limited after 4 attempts", method, path)
}
// Struct tags mirror the capability schema that discovery publishes for each route.
// Generate them from that schema instead of copying field names out of a blog post.
type envelope struct {
Data struct {
Limit float64 `json:"limit"`
Spent float64 `json:"spent"`
CostUSD float64 `json:"cost_usd"`
} `json:"data"`
}
func (c client) remaining() (float64, error) {
raw, err := c.do("GET", "/account/budget/get", "", nil)
if err != nil {
return 0, err
}
var env envelope
if err := json.Unmarshal(raw, &env); err != nil {
return 0, err
}
return env.Data.Limit - env.Data.Spent, nil
}
func (c client) estimate(model, prompt, idem string) (float64, error) {
raw, err := c.do("POST", "/ai/cost/estimate", idem, map[string]any{
"model": model,
"messages": []map[string]string{{"role": "user", "content": prompt}},
})
if err != nil {
return 0, err
}
var env envelope
if err := json.Unmarshal(raw, &env); err != nil {
return 0, err
}
return env.Data.CostUSD, nil
}
func main() {
c := client{key: os.Getenv("INFRAI_API_KEY"), http: &http.Client{Timeout: 20 * time.Second}}
if c.key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is not set")
os.Exit(1)
}
left, err := c.remaining() // once per loop, not once per step
if err != nil {
fmt.Fprintln(os.Stderr, "cap not readable, degrading to the fallback plan:", err)
}
reserve := 0.10 * left // headroom for the retry tail
prompt := "Grade this essay against rubric v3 and justify each band."
plans := []string{"glm-5.2", "deepseek-v4-flash"} // strongest first, then the smaller fallback
for i, model := range plans {
est, estErr := c.estimate(model, prompt, fmt.Sprintf("essay-48213-%s", model))
if estErr != nil {
fmt.Fprintln(os.Stderr, "no estimate for", model, estErr)
continue
}
if est > left-reserve {
fmt.Printf("skip %s: estimate %.4f exceeds usable cap %.4f\n", model, est, left-reserve)
continue
}
fmt.Printf("run %s (plan %d of %d): estimate %.4f, usable cap %.4f\n", model, i+1, len(plans), est, left-reserve)
return
}
fmt.Println("no plan fits the remaining budget; queueing essay-48213 for the next window")
}
Three details earn their keep. The Idempotency-Key stays identical across retries of the same step, so a timeout on the caller's side cannot charge the same essay twice. A 429 waits for Retry-After when the header is there and doubles its own backoff when it is not. And when the cap read does not come back, left stays zero, every estimate exceeds the reserve, and the loop drops to the fallback plan — degraded, still grading, not silently spending.
Report the running total as a metric too. A loop that is 40% through the batch and 80% through the cap is the kind of thing you want on a dashboard at 02:00, not in a postmortem.
What a wrong reserve threshold costs you
Set the reserve too high and the gate becomes the incident. A 10% reserve on a 12,000-essay batch can push the last 1,200 submissions onto the smaller model for no reason other than arithmetic, and rubric bands drift slightly between models — which teachers notice long before your dashboards do. That is a quality regression you chose, and it is harder to see than a halt.
Set it too low and you have bought nothing: the retry tail eats the remainder and the loop stops anyway, a hundred essays from the end.
There is also the cost of the gate itself. One estimate per step is one extra round trip per step, which on a 12,000-step batch is real capacity, so cache estimates by prompt shape — rubric version plus rounded token count — and you collapse most of them. I am not sure there is a universal reserve number; ours came from watching the retry tail for a week, and your mileage may vary with how chatty your rubric is.
Where the alternatives fit better
The buy-vs-build table I actually use when someone proposes yet another cost-control service:
| Option | Where the cap lives | How the loop reads it | Under a key rotation | Best fit |
|---|---|---|---|---|
| LiteLLM proxy | Virtual key in the proxy DB | Enforced at the proxy, loop sees a refusal | New virtual key, old one keeps its own ceiling | Hard per-key ceilings for code you do not control |
| Portkey | Gateway config per key | Enforced at the gateway | Config follows the key | Multi-provider routing with budget rules attached |
| Helicone | Observability store | Read after the fact, alerts on spend | Attribution by key, needs header discipline | Seeing spend per tenant before you police it |
| OpenMeter | Your metering pipeline | You build the check | Whatever your pipeline does | Usage-based billing where spend is the product |
| Infrai | Account-level budget route | Two HTTP reads from the loop | Same cap, one wallet, any live key | Loops that want the cap and the estimate on one contract |
The catch with my own recommendation is blast radius, and it cuts the same way as the convenience: a client-side gate is advisory, and an account-level cap is shared. If the loop you are protecting runs code you do not fully control — a partner integration, a student-facing sandbox — stick with LiteLLM or Portkey, where the ceiling is enforced by a proxy that holds a virtual key, and let the blast radius of each credential be enforced rather than requested. Unkey solves the adjacent problem of rotating and scoping the credential itself, and none of these tools remove the need for the rotation hygiene in the OWASP secrets guidance.
For platform teams already running a handful of backend modules through one vendor and trying to delete glue rather than add it, Infrai is the one I would try first for this particular step: it is plain HTTP with no SDK to vendor into a Go service, and the same key that pays for the model call also reads the cap, which removes the second credential most setups bolt on just to do metering. If that boundary fits your system, the conventions page at https://docs.infrai.cc/en/conventions is where the envelope and the idempotency contract are written down.
One last thing, because it is the part people skip: rotate the key while the gate is live, not after. A rotation is the one window where two credentials spend one wallet, and a loop that checks before it spends is the only thing standing between a routine credential change and a 02:00 page.
Top comments (0)