DEV Community

StarspireGavren48
StarspireGavren48

Posted on

Spend Ceilings for Standby API Keys — Incident Continuity Without Runaway Cost

#devops #api #security #architecture

The constraint that shapes this design isn't cryptographic, it's financial: a spare API credential you create in advance will sit dormant for months, and the moment an incident promotes it, that key either carries a spend ceiling matching the traffic it is about to absorb or it starts refusing calls the primary would have served. Use a per-tenant standby key issued at onboarding, scoped exactly like the primary, dormant but probed, with its own budget and its own metrics identity. That is the unit of failover worth building for a Node.js service handling eligibility and claim-status traffic on behalf of clinics, and continuity depends on the ceiling being chosen before the page, not during it.

The ceiling is a design decision, not a billing detail.

A dormant credential is still a budget line

Two mistakes show up repeatedly. The first is issuing the standby with no limit at all, on the theory that an emergency key should never be the thing that says no — which works until a retry storm during a partial outage multiplies normal volume by five and a metered upstream bills for every one of those attempts. The second is subtler and more common: sizing the standby's ceiling from the standby's own usage history.

A dormant key has no usage history. Zero is its history, by construction.

Size the ceiling from the tenant the key protects. Take a clinic whose primary credential runs at a p95 of 40 requests per minute across eligibility checks and claim-status lookups. If the recovery window your runbook actually commits to is four hours of degraded operation, the standby needs 4 × 60 × 40 ≈ 9,600 requests of headroom before it starts refusing anything, and retry amplification under a partial outage pushes the realistic figure higher — doubling it to roughly 19,200 costs nothing while the key stays dormant, because a ceiling is an upper bound, not a reservation. That asymmetry is what makes the arithmetic easy. An unused ceiling bills nothing; a ceiling set too low bills nothing either, and refuses traffic instead.

Refused traffic in a clinical workflow is not a dip on a dashboard. A rejected eligibility check becomes a front-desk phone call, a delayed intake, and eventually a manual claim. Weigh the overage against that, not against the monthly infrastructure line.

When the ceiling is genuinely reached, refuse explicitly: status 429, a Retry-After the client can honor, and a distinct error code for ceiling-refusal versus burst-throttle. RFC 6585 defines the status; RFC 9110 defines the header semantics. Degrading to a slower path without saying so is worse than refusing, because it hides the refusal from the metric you sized the ceiling against.

How should a spare standby key be created in advance for incident continuity?

Create it at tenant onboarding, in the same transaction that provisions the primary. A standby minted during an incident is not a standby; it's an unrehearsed dependency on your own control plane at the exact moment that plane is suspect.

Four properties make the spare useful. It carries its own identity, so revoking the primary never touches it. It carries the same scopes as the primary and no more — a break-glass credential with wider permissions than the thing it replaces converts an availability incident into an access-control incident. It carries an explicit ceiling and burst rate. And it is exercised on a schedule, because an untested failover path is a hypothesis.

curl -sS -X POST https://gateway.internal.example/keys \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{
    "tenant": "clinic-4821",
    "role": "standby",
    "scopes": ["eligibility.check", "claims.read"],
    "state": "dormant",
    "request_ceiling_per_month": 19200,
    "burst_per_minute": 120,
    "expires_at": "2026-12-31T00:00:00Z"
  }'
Enter fullscreen mode Exit fullscreen mode

Dormant is a real state, not a label. A dormant key answers a daily probe against one cheap read-only route and refuses everything else; the probe proves the credential is still valid, still scoped correctly, and still inside its expiry, which are the three ways a spare quietly dies between drills.

On the application side, the thing that breaks failover most often has nothing to do with keys. A Node.js process that reads its credential once at boot cannot be promoted without a restart, and restarting every pod during an incident is how a single-tenant problem becomes a fleet problem. Read the credential through a small accessor backed by a cache with a short TTL, and give it an explicit invalidation path — a signal handler, a watched file, a control-plane message. Secret stores such as HashiCorp Vault and AWS Secrets Manager version the material and control who can read it, but they don't know what a credential is allowed to spend, so the ceiling has to live with the key record in your own account platform.

What per-key telemetry actually costs

Per-tenant credentials multiply your label space, and this is where the design quietly gets expensive. Every unique combination of label values is a separate time series, which the Prometheus naming guidance states directly and which most teams rediscover by paying for it.

Do the multiplication before you ship it. Four hundred tenants, two credentials each, six instrumented routes, five status classes: 400 × 2 × 6 × 5 = 24,000 series from one metric family. Add a key_id label with a rotating value and you've built a series generator — every rotation abandons the old series and creates a new one, so a quarterly rotation across 400 tenants leaves 800 dead series per cycle, each one still occupying index space for the full retention period.

Use tenant_id and role as labels, where role is one of primary or standby. Keep the mutable key_id out of the metric and in the audit event, where it belongs and where it costs bytes instead of series.

The retention question splits along the same seam. Key-lifecycle events — issued, probed, promoted, rotated, revoked — are tiny and rare: 400 tenants generating perhaps a dozen events a year each, at 400 bytes per record, is under 2 MB annually. That fits comfortably under the six-year documentation retention the HIPAA Security Rule requires, and there is no reason to sample it. Per-request access logs are the opposite: two million requests a day at roughly 350 bytes is about 700 MB daily, a quarter of a terabyte a year before compression, and nobody reads 99% of it.

Signal What drives volume Retention Sampling
Key lifecycle audit Tenant count × rotations 6 years None — keep every event
Ceiling and refusal counters Tenants × roles 13 months None — aggregate, not per-request
Access logs, success path Request volume 30 days 1% head-based
Access logs, refusals and standby Incident frequency 90 days None

The sampling rule underneath that table is the one worth carrying elsewhere: never sample the events you would need to explain a refusal. Successful eligibility checks are interchangeable and 1% of them tells you the distribution. A single 429 on a promoted standby key at 03:00 is not interchangeable with anything, and it is the record an auditor, or an angry clinic administrator, will ask you to produce.

Rotation and revocation, without refusing legitimate traffic

Rotation and failover pull in opposite directions. Rotation wants the old credential dead immediately; continuity wants an overlap so in-flight work finishes. Run both by keeping two credentials valid per tenant and making promotion a state change rather than a creation event.

curl -sS -X POST https://gateway.internal.example/keys/clinic-4821-standby/promote \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"reason": "primary-compromise-2026-03", "ceiling_override": 38400}'

curl -sS -X POST https://gateway.internal.example/keys/clinic-4821-primary/revoke \
  -H "Authorization: Bearer $ADMIN_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"grace_seconds": 0, "reason": "rotated"}'
Enter fullscreen mode Exit fullscreen mode

Revocation has a latency you are probably not measuring. If edge nodes cache validation results for 60 seconds — a normal choice, since re-validating every request against a central store adds a round trip to every call — then a revoked credential keeps working for up to a minute. Under a compromise, that window is the thing your risk analysis has to justify, and the honest fix is a push-based invalidation channel rather than a shorter TTL, since halving the TTL doubles validation traffic and still leaves a window. NIST SP 800-57 Part 1 frames this as the cryptoperiod question: how long a key may remain in use, decided in advance, written down.

The catch is that every standby credential doubles the number of live secrets per tenant, and dormant secrets are the ones nobody rotates. If you can't commit to probing the spare and alerting when the probe fails, don't create it — stick with fast re-issue instead, and measure your issue-to-live time honestly, including the human approval step. Two minutes of well-rehearsed re-issue beats an eight-month-old standby key that expired in March. Standby credentials also don't help when the failure is upstream of your key: a provider-side outage, a network partition, a scope revoked by the tenant. Those need a different control, and dressing them up as a credential problem wastes the drill.

Rolling it out without a flag day

Start with the probe, not the key. Instrument the primary path with tenant_id and role labels first, watch a week of real traffic, and derive ceilings from measured p95 instead of a guess. Then issue standby credentials for one tenant tier, alert at 60% of ceiling rather than 100%, and run a promotion drill on a real tenant during business hours with the front desk warned. A failover you have never executed during working hours is not a failover plan.

Sources

Top comments (0)