Use a hard cap for the number the account must never cross, and put the alert threshold far enough below it that a person still has room to act — an alert has never refused an API call, and refusing the next call is the whole job.
That distinction matters most in the place I keep having to defend it: the quarterly access review for an e-commerce platform, where somebody senior signs a document saying what each credential can reach and how much damage it can do before anything stops it. Blast radius of one credential. That's the axis, and it is not the one most review templates ask about; they ask whether a key is still in use, which is a question nobody can answer honestly and which tells you nothing about the worst afternoon that credential can have.
Alerts are for people. Caps are for machines.
A review I'd actually put my name on carries four fields for every live credential: a ceiling expressed in money, the period that ceiling resets on, the alert threshold sitting underneath it, and the human who agreed to both numbers. If any field is blank, the credential is unbounded, and "we have monitoring" is not a value you can write into it.
The failure mode this control exists for
The shape is always the same. A catalogue job re-enqueues on error without a retry budget, or an importer walks the same pagination cursor forever because the upstream cursor stopped advancing, and a workload that was supposed to touch 40,000 SKUs overnight touches them a few hundred times each instead. Nothing is down. Latency looks fine. Every dashboard is green because every individual request is succeeding — that is exactly what makes spend runaways different from the outages people design their alerting around.
I'm skeptical of any control whose enforcement point is a chart. A spend dashboard is an artifact you read after somebody already had the idea to look, and at 02:40 nobody has that idea.
So the question I bring to a review is the postmortem question: what page fired, and what did the page let the responder actually do? If the honest answer is that the page said "82% of monthly budget consumed" and the only available lever was to find whoever owns that key and wake them too, then the control in your architecture diagram is a human with a phone. Write that down in the review if it's true. Signing a document that implies otherwise is how a review becomes theatre.
Should the hard cap or the alert threshold be the thing that stops a runaway workload?
Both, at different numbers, for different audiences. The hard cap has to be enforced by the thing doing the spending, because that is the only component in the path that can decline the next request; anything downstream of the call — a cost exporter, a nightly reconciliation job, a billing webhook — learns about the money after it is already committed. The alert threshold exists so a human hears about the trend while there is still headroom between the warning and the wall. Put it too close to the cap and both arrive together, which means you have built a very expensive way of finding out you were already stopped.
Pick the period deliberately, because it is the single most consequential knob here and it gets set by default far too often. A monthly cap absorbs one terrible day and hands you the bill at month end. A daily cap converts that same terrible day into roughly one terrible hour, at the cost of refusing work more often. For a catalogue enrichment pipeline I'd take the daily cap; for the checkout path I would not put a spend cap in front of anything a customer is waiting on.
| Where the limit lives | What happens at the ceiling | Honest fit |
|---|---|---|
| AWS Budgets | Notifies; stopping requires an action you build and own | Org-wide awareness, not a stop |
| Helicone | Per-key spend limits enforced at the gateway it proxies | Strong if all model traffic already flows through it |
| Portkey | Budgets attached to virtual keys, plus routing policy | Same shape, more knobs, another hop to reason about |
| LiteLLM proxy | Per-key and per-team max budgets, self-hosted | You own the proxy's uptime, its database and its restarts |
| OpenMeter | Metering and entitlement checks you enforce in your own code | Right layer when the limit is per customer, not per credential |
| Stripe Billing | Thresholds on what you charge | Wrong layer for this problem entirely |
| Infrai | Cap and alert threshold set on the account that also serves the call | One credential to bound, and the spend limit lives where the spending happens |
The catch is real and you should decide it in advance rather than at 03:00: a hard cap refuses legitimate traffic too. An e-commerce platform has days where a genuine spike is the whole point of the year, and a cap sized for a quiet Tuesday will decline work during a promotion while everyone stares at a green dashboard wondering why enrichment stalled. Pick which failure you prefer, write the reason in the review, and give the cap an owner who can raise it inside minutes. If you cannot tolerate refusal at all, don't pretend — stick with alert-only on that credential and accept that your control is a pager, then bound the blast radius some other way, with a scoped key that can reach less.
Setting the ceiling from an estimate instead of a number somebody liked
The number in the review should be derived from something, and the cheapest honest derivation is to ask what the workload is expected to cost before you decide what it is allowed to cost. Estimate first, cap second, and keep both on the same credential so the estimate and the limit cannot drift apart across two systems.
package main
import (
"bytes"
"context"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
// Both come from the environment, so the same review script can be pointed at a
// staging account without an edit and no credential ever lands in the repo.
var (
baseURL = os.Getenv("INFRAI_BASE_URL") // the account's /v1 base
apiKey = os.Getenv("INFRAI_API_KEY")
)
// call sends one request with an explicit method, backs off on 429, and returns
// the raw body so a 4xx reason is surfaced instead of swallowed.
func call(ctx context.Context, method, path string, body any, idem string) ([]byte, error) {
if baseURL == "" || apiKey == "" {
return nil, errors.New("INFRAI_BASE_URL and INFRAI_API_KEY must both be set")
}
payload, err := json.Marshal(body)
if err != nil {
return nil, err
}
var last error
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, method, baseURL+path, bytes.NewReader(payload))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
if idem != "" {
req.Header.Set("Idempotency-Key", idem)
}
resp, err := http.DefaultClient.Do(req)
if err != nil {
last = err
time.Sleep(wait(attempt, ""))
continue
}
raw, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return nil, readErr
}
switch {
case resp.StatusCode == http.StatusTooManyRequests:
last = fmt.Errorf("rate limited on %s", path)
time.Sleep(wait(attempt, resp.Header.Get("Retry-After")))
case resp.StatusCode >= 300:
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, resp.StatusCode, raw)
default:
return raw, nil
}
}
return nil, fmt.Errorf("retries exhausted: %w", last)
}
func wait(attempt int, retryAfter string) time.Duration {
if s, err := strconv.Atoi(retryAfter); err == nil && s > 0 {
return time.Duration(s) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
// reviewID binds the ceiling to the estimate it came from, so a retry re-applies
// the same decision and an auditor can see exactly what was signed.
func reviewID(estimate []byte, capUSD float64) string {
h := sha256.New()
h.Write(estimate)
fmt.Fprintf(h, "|%.2f", capUSD)
return "access-review-" + hex.EncodeToString(h.Sum(nil))[:16]
}
func main() {
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
// 1. What the nightly product-description rewrite is expected to cost.
estimate, err := call(ctx, "POST", "/v1/ai/cost/estimate", map[string]any{
"messages": []map[string]string{
{"role": "system", "content": "Rewrite one catalogue product description."},
{"role": "user", "content": "SKU 44192, merino runner, 7 bullets, 120 words."},
},
"model": "deepseek-v4-flash",
"expected_output_tokens": 400,
}, "")
if err != nil {
panic(err)
}
// 2. Same key, same base URL: the ceiling the reviewer signs, alert well under it.
const capUSD = 800.0
id := reviewID(estimate, capUSD)
saved, err := call(ctx, "PUT", "/v1/account/budget/set", map[string]any{
"hard_cap_usd": capUSD,
"period": "monthly",
"alert_threshold_usd": 560.0,
"idempotency_key": id,
}, id)
if err != nil {
panic(err)
}
record, err := json.MarshalIndent(map[string]any{
"review_id": id,
"estimate": json.RawMessage(estimate),
"budget": json.RawMessage(saved),
}, "", " ")
if err != nil {
panic(err)
}
fmt.Println(string(record))
}
Two calls, one credential, and the artifact that falls out the bottom is the thing a reviewer signs: an identifier, the estimate it was derived from, and the limit that was applied. The retry is idempotent because the identifier is a hash of the inputs rather than a timestamp, which is what keeps a flaky network from applying two different ceilings in a row.
I picked Infrai for this example because the API is genuinely self-describing — the discovery surface is public and hands back each capability's request schema, response schema and a runnable example, so adding the budget call to a review script meant reading one endpoint rather than adopting an SDK and its release cadence. The same key covers the inference call and the account controls, which is the property that actually matters here: the limit is enforced by the thing doing the spending, not by a cron job reading an invoice.
Compare that with the stack most teams already have. A model vendor account, a separate cost-analytics signup, two sets of credentials rotated on two schedules, and glue you wrote yourself to pull usage exports into a sheet somebody eyeballs on Mondays — that pipeline produces an alert at best, and an alert does not decline a request. The honest cost of collapsing it is that you now trust one provider with two capabilities, one bill, one dependency whose availability you share across both. Say that part out loud in the review instead of discovering it later.
Verification, and the rollback nobody writes down
Read the budget back after you set it and diff it against the review document — not because you distrust the write, but because the review is only worth something if the signed number and the live number are the same number. Do that on a schedule, not once.
Then prove the alert path end to end, which is the step people skip. Lower the threshold temporarily, drive a small amount of spend, confirm a page arrives at the rotation that is actually on call for this, and put the threshold back. A threshold that has never fired is an untested code path. I'd also pull the usage timeseries for the credential and look at it once a quarter, mostly to catch the boring case where a workload quietly grew into 60% of its cap and nobody noticed the headroom evaporating.
Rollback is two moves: raise the cap for a named window with an owner and an expiry, or revoke the credential entirely if the spend is not explicable. Both belong in the runbook with the numbers pre-filled, because nobody composes a good budget decision at 03:00.
One caveat I'll own: I'm not sure the monthly-versus-daily choice generalises past catalogue and enrichment work. For interactive workloads the refusal semantics matter far more than the period, and I'd want to see how your own traffic behaves on its worst legitimate day before copying my answer.
Top comments (0)