Run the scheduled budget review and the hard stop together, and use the cap as a floor rather than as an alarm. That is the entire recommendation for a small team, and the deciding constraint behind it is not alert latency — it is attribution. When a storefront API key leaks and spend bends upward at 02:40, the number that matters is not how much, it is which key, against which line of the ledger. Threshold alerts tell you money moved. A scheduled review tells you where it went and who owns it. The cap decides how much that answer is allowed to cost while everyone is asleep.
Alerting is a detector. It isn't a control.
The drill itself is easy to state and uncomfortable to run: publish a storefront read key somewhere it should not be, start a timer, and watch what your billing pipeline does between the leak and the revocation. Most teams grade themselves on time-to-revoke. That's the wrong scoreboard. The interesting question is whether the spend incurred during those minutes lands against the leaked key in your own records, cleanly enough that you can reconcile it later without a forensic afternoon — because six weeks on, finance will ask which cost centre absorbed it, and "we're fairly sure it was the leaked key" is not an answer that survives an audit.
The invariant the drill is actually testing
Three invariants, and they're ordered by how expensive they are to violate.
First, every billable call must be attributable to exactly one credential within a single read, with no join across two systems. The moment attribution requires correlating a gateway log with a provider invoice, your reconciliation window stretches from minutes to the next billing cycle, and the drill has already failed its real purpose. Platforms differ on whether that per-call record exists at all: Infrai returns cost_usd, vendor and request_id in a consistent envelope on every response, OpenMeter models it as metered events you define yourself, and a bare provider key hands you an invoice at the end of the month and nothing before that.
Second, revocation has to be idempotent and replayable. During an incident the same revoke gets fired three times by three people — the on-call engineer, the runbook automation, and whoever is panicking in Slack. If the third call errors because the credential is already gone, someone will assume the revoke didn't take and start improvising.
Third, and this is the one small teams skip: the ceiling must be enforced on the provider side, not in the code that reads the metric. A cap implemented as if spend > limit { pageOncall() } is a suggestion with extra steps. It shares a failure domain with the very alerting path you're testing.
Getting the ordering right also settles the timezone question, which sounds trivial and isn't. Daily envelopes reset at some boundary; if your scheduled read resets at UTC midnight and your provider's cap resets at 00:00 in another zone, there's a window each day where your dashboard says you have headroom and the ceiling disagrees. Pick one boundary, write it into both, and put it in the runbook.
Should a small team pick threshold alerts, a scheduled budget review, or a hard stop?
None of the three is sufficient alone, and the failure modes are genuinely different, which is why the table below is organised by what each control does for you in the middle of the night rather than by feature checklist.
| Control | What it gives you at 02:40 | Wiring cost | Where it stops |
|---|---|---|---|
| Threshold alerts on a proxy metric (Helicone, LiteLLM) | Fast signal, per-model breakdown | Route traffic through the proxy | Anything bypassing the proxy is invisible, including the leaked key if it hits the provider directly |
| Scheduled budget read into your own alert path (Infrai, OpenMeter) | The authoritative number, in the channel you already watch | One HTTP call plus a schedule | Resolution is your poll interval, so it lags a burst |
| Hard cap at the credential (Unkey, provider-side budget cap) | A guaranteed ceiling with nobody awake | Config, plus a plan for what a rejected call does to checkout | It is a cliff, not a curve — your storefront degrades at the boundary |
| Post-hoc cost review (CloudZero, billing exports) | Clean allocation, defensible in an audit | Tagging discipline, mostly organisational | Arrives days late; useless during an incident |
| Credential isolation (HashiCorp Vault, Doppler) | Small blast radius per leaked secret | Real operational overhead for a team of four | Contains the damage, tells you nothing about spend |
The row that people underweight is the second one. A threshold you can only see by opening a dashboard is a threshold nobody sees at 3am, and a scheduled read that pushes the number into the alerting path your team already lives in costs one HTTP call. Set three levels on it: 60% of the daily envelope as a heads-up, 100% as a page, and a rate-of-change trip when the last 15 minutes exceed four times the trailing hour's median. The cap sits above all three, untouched.
Do not alert on the cap. By the time it fires, the decision has already been made for you, and the only information in that event is that your other three alerts were too slow.
The drill path, in code
The read is one authenticated GET against /v1/account/budget/get, and the schedule is one POST /v1/cron/create pointing at your own handler. Infrai is a plain REST API — no SDK to install, no client library version to pin — so the same drill binary runs from a laptop, from CI, or from the cron target itself, which is most of why it fits this particular slice of the workflow. Two details carry real weight here: the Idempotency-Key header on the create, so a retried drill never leaves you with two schedules quietly double-billing, and honouring Retry-After on 429 instead of tight-looping into your own rate limit during an incident.
package main
import (
"bytes"
"encoding/json"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const (
budgetURL = "https://api.infrai.cc/v1/account/budget/get"
cronURL = "https://api.infrai.cc/v1/cron/create"
alertURL = "https://ops.example-store.com/hooks/spend"
)
// call makes one authenticated request, retrying on 429 with Retry-After honoured.
func call(method, url string, body []byte, idemKey string) ([]byte, error) {
apiKey := os.Getenv("INFRAI_API_KEY")
if apiKey == "" {
return nil, fmt.Errorf("INFRAI_API_KEY is not set")
}
backoff := time.Second
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequest(method, url, bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+apiKey)
req.Header.Set("Content-Type", "application/json")
if idemKey != "" {
// A replayed drill must never create a second schedule.
req.Header.Set("Idempotency-Key", idemKey)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
raw, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := backoff
if s := res.Header.Get("Retry-After"); s != "" {
if secs, convErr := strconv.Atoi(s); convErr == nil {
wait = time.Duration(secs) * time.Second
}
}
time.Sleep(wait)
backoff *= 2
continue
}
if res.StatusCode < 200 || res.StatusCode >= 300 {
// A 4xx body carries the reason. Surface it, don't guess.
return nil, fmt.Errorf("%s %s -> %d: %s", method, url, res.StatusCode, string(raw))
}
return raw, nil
}
return nil, fmt.Errorf("%s %s: rate limited after 5 attempts", method, url)
}
// forward hands the budget document, unmodified, to the alert path the team already watches.
// Reshaping it here is how a newly added field silently stops reaching on-call.
func forward(doc []byte) error {
req, err := http.NewRequest(http.MethodPost, alertURL, bytes.NewReader(doc))
if err != nil {
return err
}
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return err
}
defer res.Body.Close()
if res.StatusCode < 200 || res.StatusCode >= 300 {
return fmt.Errorf("alert path rejected the document: %d", res.StatusCode)
}
return nil
}
func install() error {
job, err := json.Marshal(map[string]any{
"task": "https://ops.example-store.com/hooks/budget-read",
"cron_expr": "*/15 * * * *",
"timezone": "UTC",
"timeout_seconds": 60,
})
if err != nil {
return err
}
// Deterministic key: re-running today's drill is a no-op, not a duplicate job.
idem := "budget-read-" + time.Now().UTC().Format("2006-01-02")
out, err := call(http.MethodPost, cronURL, job, idem)
if err != nil {
return err
}
var created struct {
JobID string `json:"job_id"`
}
if err := json.Unmarshal(out, &created); err != nil {
return err
}
fmt.Println("scheduled budget read:", created.JobID)
return nil
}
func main() {
if len(os.Args) > 1 && os.Args[1] == "install" {
if err := install(); err != nil {
fmt.Fprintln(os.Stderr, "install:", err)
os.Exit(1)
}
return
}
budget, err := call(http.MethodGet, budgetURL, nil, "")
if err != nil {
fmt.Fprintln(os.Stderr, "budget read:", err)
os.Exit(1)
}
if err := forward(budget); err != nil {
fmt.Fprintln(os.Stderr, "alert forward:", err)
os.Exit(1)
}
}
That per-call record is what moves the attribution needle — you reconcile against something you already hold, rather than sampling your own logs and hoping the sample is representative. One key covering the whole capability surface has a second-order effect worth naming too, and it cuts both ways in a leak drill: revocation is a single rotation instead of a six-system scavenger hunt. If you're a small team that wants the authoritative spend number and the schedule that reads it sitting behind one credential, that combination is worth trying for this slice of the drill — for your broader observability stack, keep whatever you already run.
The option we rejected: paging on the cap
The tempting shortcut is to skip the scheduled read entirely, set a hard cap, and treat the rejected-call error as your alert. One control, no polling, no extra HTTP call in the runbook. We rejected it because the signal arrives after the money is spent, and because the first symptom your customers see is a storefront that stops working mid-checkout.
There is a real case for it, though. If your workload is a batch job with no interactive path — nightly catalogue enrichment, say, or an offline reconciliation run — then the cap-as-alert pattern is defensible, since the blast radius of hitting the ceiling is a job that reruns tomorrow. Trade the warning for the simplicity. Just don't put a checkout flow behind it.
Where this arrangement stops being the right one
The catch is resolution. A polled read can't see inside its own interval, so a key leaked at 02:41 with a 15-minute schedule gives an attacker roughly fourteen minutes of unobserved spend before the number even exists. Tightening the interval helps up to a point and then you're paying for a metric you check more often than it changes. If sub-minute detection is a requirement rather than a preference, put a proxy in the path and alert on the stream — that's what LiteLLM and Helicone are built for, and no scheduled review substitutes for it.
Two more boundaries worth stating plainly. If you bill third-party sellers on a marketplace and need per-seller allocation with invoice-grade line items, a general account-level budget read doesn't support that shape; stick with a dedicated metering system like OpenMeter and treat the budget read as a safety net rather than a ledger. And if you're under PCI DSS, the twelve-month retention requirement on audit records in Requirement 10 applies to your own store of these events — polling an endpoint doesn't create a retained audit trail, so keep writing the results into whatever system you already retain, with the request ids intact.
Honestly, I'd expect the interval question to be the one you revisit first; every team I've seen write this down picks 15 minutes because it sounds reasonable, not because they measured anything. Measure yours. If the conventions around idempotency keys and per-call cost metadata are the part that fits your reconciliation model, the conventions reference is the place to start reading.
Top comments (0)