Use the alert threshold to wake somebody up, and use the hard cap to stop the money — they are different controls and only one of them actually stops a runaway workload. An alert is a promise that a human will act inside your response window, which at 02:00 on a Sunday is a generous assumption. In a multi-tenant customer support platform, where every tenant gets its own scoped API key for ticket summarization and auto-replies, the cap has exactly one enforcement point that holds under load: the credential the tenant's workload authenticates with. Revoke or downgrade that key and spend stops at the next request, with a record of who stopped it, when, and against which usage number.
The threshold tells you. The cap does something about it.
The constraint that decides the design is metering lag. Usage is aggregated asynchronously, so any cap you build is an approximation of a number that is already stale by the time you read it; you will overshoot by whatever was in flight between the last aggregation and the revoke. Budget for that overshoot explicitly instead of pretending it's zero.
Which one actually stops a runaway workload: the hard cap or the alert threshold?
A budget alert is telemetry with an opinion. It fires at 80% of the monthly budget, pages whoever is on call, and the workload keeps running at full rate the entire time — through acknowledgement, through the login to the console, through the argument about whether this tenant is allowed to be throttled. Mean time to acknowledge is a real number in your incident data, and it is the number that multiplies your overspend.
The Node.js side of this is where it usually gets expensive. A support automation takes an inbound ticket webhook, enqueues a job, and a worker calls a model API to draft a reply; the HTTP client has a 30s timeout and three retries, the queue's visibility timeout is 30s as well, and the model call occasionally takes longer than either. Now the queue redelivers a job that is still running, the client retries a request that already reached the provider, and a single ticket bills three or four times. That is the same shape as the duplicate-delivery pages I associate with badly tuned job infrastructure, except each duplicate now has a price attached.
That isn't a pricing problem. It's a delivery-semantics problem wearing a pricing costume.
So run both controls, with different owners and different expectations. The threshold belongs to whoever watches the budget; the cap belongs to whoever owns access.
| Control | What it changes at request time | Evidence it leaves | Stops a runaway workload? |
|---|---|---|---|
| Budget alert threshold | Nothing | A notification, maybe an ack | No — it depends on a human |
| Caller-side rate or concurrency limit | The caller's own request rate | Local metrics only | Only if every caller ships the fix |
| Hard cap enforced on the tenant's scoped key | The authorization decision | Revocation record plus key status | Yes, from the next request onward |
| Account-wide hard cap | Authorization for every tenant | Account audit log | Yes, and it also takes down tenants that did nothing wrong |
The per-tenant scoping in row three is what keeps a cost control from becoming an availability incident for everybody else. One noisy tenant, one key, one blast radius.
Why a hard cap belongs at the scoped key rather than in the billing dashboard
Once you accept that the cap must be enforced in the request path, it stops being a billing feature and becomes an authorization decision — and authorization decisions are the ones your platform already knows how to log, review, and explain to an auditor six months later. OAuth 2.0 gives you the vocabulary for scope, and RFC 7009 gives a standard shape for retiring a credential. Neither cares what the reason was, which is precisely why the reason has to live in your own audit record.
For auditability of access, the record that matters carries the non-secret key identifier, the tenant, the billing period, the metered spend observed at decision time, the cap value it crossed, the actor (automation or a named human), a reason code, and a pointer to whatever event can reverse it. The secret value never appears — not in the log, not in the ticket, not in the notification, which is the boring, unglamorous discipline that OWASP's secrets guidance keeps repeating because people keep skipping it.
One detail that costs real money if you get it wrong: return 403 with a machine-readable reason, not 429. A 429 means "you are being rate limited, come back later" and a well-behaved client honours Retry-After and does exactly that, hammering a wall it can never get past and filling your queue with jobs that will never succeed. "Not this month" is a terminal condition. Say so.
The enforcement itself should be a reconciler, not a callback hanging off the metering pipeline. Desired key state is a pure function of usage and cap; a loop compares that to actual key state and fixes the difference. Re-running it is free, which means a retried job, a duplicated event, or an on-call engineer running the runbook twice cannot produce two revocations or two audit entries.
package spendguard
import (
"context"
"errors"
"fmt"
"time"
)
var ErrAlreadyRevoked = errors.New("key already revoked")
type Decision string
const (
Allow Decision = "allow" // under the alert threshold
Notify Decision = "notify" // over the alert threshold, still spending
Deny Decision = "deny" // over the hard cap, the key must stop authenticating
)
// Usage is the metered spend for one tenant in one billing period. It is
// always a little stale; treat AsOf as part of the evidence, not decoration.
type Usage struct {
Tenant string
Period string // "2026-09"
SpentMicros int64
AsOf time.Time
}
// Cap references the tenant's scoped key by identifier. The secret itself
// never enters this package.
type Cap struct {
Tenant string
KeyID string
AlertMicros int64
HardMicros int64
}
func Evaluate(u Usage, c Cap) Decision {
switch {
case u.SpentMicros >= c.HardMicros:
return Deny
case u.SpentMicros >= c.AlertMicros:
return Notify
default:
return Allow
}
}
// Directory is whatever issues and revokes scoped keys for you: an internal
// service, a secrets manager, an identity provider.
type Directory interface {
Revoke(ctx context.Context, keyID, reason string) error
}
type AuditLog interface {
Seen(ctx context.Context, idempotencyKey string) (bool, error)
Append(ctx context.Context, ev Event) error
}
type Event struct {
IdempotencyKey string
Tenant string
KeyID string
Action string
ReasonCode string
SpentMicros int64
CapMicros int64
ObservedAt time.Time
DecidedAt time.Time
Actor string
}
// Enforce is safe to call on every reconcile tick. One cap crossing produces
// one revocation and one audit record, no matter how often it runs.
func Enforce(ctx context.Context, dir Directory, log AuditLog, u Usage, c Cap, now time.Time) error {
if Evaluate(u, c) != Deny {
return nil
}
idem := fmt.Sprintf("cap-deny:%s:%s:%d", c.Tenant, u.Period, c.HardMicros)
seen, err := log.Seen(ctx, idem)
if err != nil {
return fmt.Errorf("audit lookup %s: %w", idem, err)
}
if seen {
return nil
}
if err := dir.Revoke(ctx, c.KeyID, "hard_spend_cap"); err != nil && !errors.Is(err, ErrAlreadyRevoked) {
return fmt.Errorf("revoke key %s: %w", c.KeyID, err)
}
return log.Append(ctx, Event{
IdempotencyKey: idem,
Tenant: c.Tenant,
KeyID: c.KeyID,
Action: "revoke",
ReasonCode: "hard_spend_cap",
SpentMicros: u.SpentMicros,
CapMicros: c.HardMicros,
ObservedAt: u.AsOf,
DecidedAt: now,
Actor: "spendguard/reconciler",
})
}
Note what the idempotency key is made of: tenant, period, and the cap value that was crossed. Raise the cap mid-period and the key changes, so a deliberate new decision can be recorded — while a replayed usage event cannot.
Verification, rollback, and the audit trail that survives a dispute
You don't know a cap works until you have fired it on purpose, on a schedule, in an environment that resembles production. Create a synthetic tenant, set the hard cap two orders of magnitude below anything real, drive traffic until it crosses, then assert three things: the key's status is revoked, the audit record exists and carries the usage number that triggered it, and a request with that key comes back 403 with the reason code rather than 429.
A cap that has never fired is a hypothesis.
{
"idempotency_key": "cap-deny:tenant_4471:2026-09:250000000",
"tenant": "tenant_4471",
"key_id": "k_9f2c1a",
"action": "revoke",
"reason_code": "hard_spend_cap",
"spent_micros": 251480000,
"cap_micros": 250000000,
"observed_at": "2026-09-11T04:18:02Z",
"decided_at": "2026-09-11T04:20:37Z",
"actor": "spendguard/reconciler"
}
Rollback is where teams reach for the wrong primitive. There is no un-revoke — reversal means raising the cap, issuing a fresh scoped key, distributing it, and writing a second audit record that references the first. Keep the emergency override behind a named human, a reason code, and an expiry, otherwise the override quietly becomes the permanent cap and you find out in the next invoice review.
The observability that makes all of this defensible is small: a counter of decisions labelled by tenant and outcome, a gauge of headroom against each cap, and — the one people forget — an alert on the reconciler's own freshness. A metering pipeline that silently stops delivering makes the threshold and the cap equally useless, because both read the same stale number and conclude everything is fine.
When a hard cap is the wrong control
The catch is that revoking a key converts a cost incident into an availability incident, and for some tenants that trade is unacceptable. If a support queue is covered by a contractual response time, a hard cap is not a good fit; stick with a soft degrade — fall back to a cheaper model path or a templated acknowledgement, keep writing the same audit records, and page a human with the headroom number attached.
Key-scoped caps also only cover what flows through that key. Storage, egress, seats, and anything billed outside the authenticated request path will keep accruing after the revoke, so a cap presented to the customer as a total spend limit is a promise your architecture can't keep unless every billable path sits behind a credential you control.
And if you need cent-exact enforcement, no key-level cap will deliver it. I'm not sure that guarantee exists anywhere outside a prepaid balance that is decremented synchronously before the work runs, which buys precision at the cost of latency on every request. Most teams should pick approximate-but-fast and write the overshoot into the runbook.
Two controls, one owner each, and a drill on the calendar before the invoice teaches you the same lesson.
References
- OWASP Secrets Management Cheat Sheet
- RFC 6749 — The OAuth 2.0 Authorization Framework (access token scope)
- RFC 7009 — OAuth 2.0 Token Revocation
- RFC 6585 — Additional HTTP Status Codes (429 Too Many Requests)
- RFC 9110 — HTTP Semantics (403 Forbidden, idempotent methods)
- NIST SP 800-92 — Guide to Computer Security Log Management
- OpenTelemetry Metrics specification
Top comments (0)