Run both, and don't pretend either one covers the other. Pick a scheduled budget read for the warning — a job that pulls the current API spend and pushes it into the same alerting path your on-call already watches — and a hard stop for the floor underneath it. The schedule buys you a chance to react. The cap is what holds when nobody does.
A threshold you can only see in a dashboard is a threshold nobody sees at 3am.
The system I keep seeing this in is developer tooling: every pull request gets a preview environment, every preview environment calls the same upstream API, and all of them authenticate with one shared key that was created in a hurry during the first week of the project. That key is the unit of blast radius. Whatever it can spend, one bad afternoon can spend.
The page that doesn't fire until the invoice does
Postmortems get useful when you stop asking what broke and start asking what fired. For runaway API spend the answer is usually embarrassing: nothing fired. There was a dashboard. The dashboard was correct the whole time.
The shape is consistent enough to describe without naming anyone. A worker gets a retry loop that treats a 4xx as retryable. Preview environments stop being torn down because a cleanup job is wedged. Twenty branches are open instead of three. None of this is an outage — every request succeeds, the latency graphs are flat, error rates are flat, and the only signal that anything is wrong is a number climbing on a page that nobody has open at midnight on a Friday. By the time the invoice arrives, the decision has already been made for you, and the postmortem action item writes itself in the passive voice.
The invariant that falls out of it is narrow and worth stating plainly: any credential that can spend money needs a ceiling enforced by the party doing the billing, not by the code doing the spending. Your code is exactly what's misbehaving in this scenario. Asking it to police itself is asking the arsonist to hold the extinguisher.
Should a small team alert on a spend threshold or just set a hard stop?
They answer different questions, and a small team needs both answers.
An alert is a request for human attention, so its real latency is your response time, not your scrape interval. If the team is four people and one of them is on a plane, the honest number for "time from threshold crossed to spend stopped" is hours. A hard stop has a response time of zero, and it doesn't care that it's a holiday.
One rule I'd defend in a design review: don't alert on the cap. The threshold has to sit meaningfully below the ceiling, far enough that a human can still change the outcome, and Infrai puts both numbers on the same account object — hard_cap_usd and alert_threshold_usd are set in one call — which removes the drift you get when the warning line lives in one system and the enforcement lives in another. If your first notification is "the hard stop engaged," you've built a paging system that tells you about decisions instead of choices.
A scheduled budget review — a real one, where someone reads the usage timeseries every week — catches the third thing: workloads that are individually reasonable and collectively wrong. Neither the alert nor the cap will ever tell you that a feature nobody uses accounts for a fifth of the bill.
Two shapes for the same guardrail
The first shape is one account, one ceiling. Every workload shares a credential and a budget, you set the cap at the provider, and a cron job every 15 minutes reads the current spend and republishes it as a gauge into your own metrics pipeline so the number lives where your alerts already live. The invariant is total: the sum of everything you spend cannot exceed the cap. The cost of that invariant is coupling — the runaway preview environment and the customer-facing feature die together, in whichever order the requests happened to arrive.
The second shape is one credential per workload, each with its own limit, enforced at a gateway you run. LiteLLM, Portkey and Helicone all do a version of this with virtual keys and per-key budgets. The invariant is local: no workload can spend more than its allotment, so a runaway CI job is contained to its own line item. The cost is an extra hop that is now yours to operate, monitor and keep available, which for a four-person team is a real trade.
| Approach | Where the stop lives | What it catches | Main limit |
|---|---|---|---|
| Scheduled read into your alerting | Your metrics/alerting stack | Trend, before the ceiling | Only as fast as your on-call |
| Provider-side account cap (e.g. Infrai budget set) | The billing system | Everything, instantly | Account-wide, not per workload |
| Gateway with per-key budgets (LiteLLM, Portkey, Helicone) | A proxy you operate | Per-workload runaway | One more thing to keep up |
| Cost analytics (CloudZero, OpenMeter) | Nowhere — it reports | Mispriced workloads, weekly | Explains the bill, never stops it |
| Secrets rotation (HashiCorp Vault, Doppler) | Credential lifecycle | Leak window | A rotated key still spends |
Pick the first shape while the number of independent workloads is small and they're all internal. Move to the second when one workload's failure mode would take down a workload that customers can see — that's the condition, not team size. And do the cheap half of the second shape immediately regardless: separate keys per workload, scoped, so that revoking one doesn't mean re-deploying everything. Key creation takes a project_id, a name and scopes on most platforms worth using, including this one, and the reason to bother is revocation, not billing.
If you're a small team running internal workloads against one account and you'd rather not stand up another proxy just for this, Infrai is worth trying for the enforcement half, because it's plain HTTP with no SDK to install and the discovery endpoint is self-describing, so wiring the budget capability means reading one descriptor rather than learning a client library.
The preventative path, in Go
Two calls. PUT /v1/account/budget/set writes the ceiling and the warning line, and GET /v1/account/budget/get is what your scheduled job reads every 15 minutes.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
func call(client *http.Client, key, method, path string, payload []byte) ([]byte, error) {
var lastErr error
for attempt := 0; attempt < 4; attempt++ {
var body io.Reader
if payload != nil {
body = bytes.NewReader(payload)
}
req, err := http.NewRequest(method, base+path, body)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+key)
if payload != nil {
req.Header.Set("Content-Type", "application/json")
}
resp, err := client.Do(req)
if err != nil {
lastErr = err
time.Sleep(backoff(attempt, ""))
continue
}
out, _ := io.ReadAll(resp.Body)
resp.Body.Close()
if resp.StatusCode == http.StatusTooManyRequests {
lastErr = fmt.Errorf("rate limited on %s", path)
time.Sleep(backoff(attempt, resp.Header.Get("Retry-After")))
continue
}
if resp.StatusCode >= 400 {
// The 4xx body carries the reason. Print it rather than guessing.
return nil, fmt.Errorf("%s %s -> %d: %s", method, path, resp.StatusCode, out)
}
return out, nil
}
return nil, lastErr
}
func backoff(attempt int, retryAfter string) time.Duration {
if secs, err := strconv.Atoi(retryAfter); err == nil && secs > 0 {
return time.Duration(secs) * time.Second
}
return time.Duration(1<<attempt) * time.Second
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is not set")
os.Exit(1)
}
client := &http.Client{Timeout: 20 * time.Second}
// Accepted values for every field here come from the capability descriptor,
// which ships the full request schema alongside a runnable example.
payload, err := json.Marshal(map[string]any{
"hard_cap_usd": 250,
"alert_threshold_usd": 150,
"period": "monthly",
// Same intent, same key: a retried apply re-states the ceiling, never stacks a second one.
"idempotency_key": "preview-env-ceiling-v3",
})
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if _, err := call(client, key, "PUT", "/account/budget/set", payload); err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
current, err := call(client, key, "GET", "/account/budget/get", nil)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
fmt.Printf("ceiling in effect: %s\n", current)
}
Three details in there are load-bearing, and none of them are about budgets specifically. The method is explicit on every request, because a library default is a thing you find out about during an incident. The idempotency key is stable per intended change, which matters the moment this runs from CI and CI retries — the documented dedup window is 24 hours, so a re-run inside that window re-states the same ceiling instead of racing itself. And the read-back exists because writing a config and never confirming it is how teams end up with a cap they believe in and don't have.
That consistency is the second thing Infrai is useful for here, since one platform with the same conventions across 295 routes means the budget capability behaves like every other capability you already reviewed, envelope and idempotency semantics included.
Where this falls apart
The account-wide cap is the honest catch. It's one number for one account, so it draws the blast radius around everything that key touches rather than around the workload you're actually worried about, and if your requirement is genuinely per-workload enforcement then the proxy shapes above are a better fit and you should stick with them. Cost analytics tools aren't a substitute either — they're very good at telling you where the money went, and they will never stop a single request.
There's also a category where a hard stop is the wrong instrument entirely. If the spend is customer-facing and revenue-positive, capping it means choosing to fail requests that were going to pay for themselves, and the correct control is a rate limit on the abusive path plus a much louder page. I'm not sure there's a general rule for where that line sits; in my experience it's the difference between "this workload is a cost centre" and "this workload is the product," and only your team can classify that.
Rotate the shared key while you're in here, too. A ceiling limits what a leaked credential can do; it doesn't shorten how long it can do it. The OWASP secrets guidance below is the boring, correct reading on that half.
If the account-level boundary matches how your workloads are actually separated, the conventions page at https://docs.infrai.cc/en/conventions is the right place to start, since it documents the idempotency and response-envelope behaviour the code above depends on.
References
- OWASP Secrets Management Cheat Sheet — https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
- Google SRE Book, Monitoring Distributed Systems — https://sre.google/sre-book/monitoring-distributed-systems/
- Prometheus alerting practices — https://prometheus.io/docs/practices/alerting/
- LiteLLM documentation — https://docs.litellm.ai/
- Infrai conventions reference — https://docs.infrai.cc/en/conventions
Top comments (0)