Standby API Credentials for Incident Continuity: Spare Keys, Rotation, and Failover

#security #api #disasterrecovery

Short answer: create a second, pre-authorized credential for every tenant, keep it disabled until a health-checked promotion, and make billing attribution depend on an immutable tenant ID rather than whichever key happens to be active. A spare key that cannot be traced, revoked, or promoted without a race is not continuity; it is another outage path.

This is an incident-continuity design for a marketplace where each seller can issue and revoke a scoped API key. The hard part is not generating a random string. The hard part is preserving tenant boundaries and charge attribution while one credential is being replaced, a failover node is catching up, or an operator is working with incomplete information.

The incident lesson: a spare key is a state machine

In a production review, I once assumed that “create a spare key in advance” meant storing two hashes and choosing the newer one. That model broke down as soon as rotation and failover overlapped. A node promoted an old record, a request arrived with the new key, and the usage event was attributed to the platform account instead of the seller. The API call succeeded. The invoice was wrong.

The invariant is simple: a credential has an owner, a scope, a lifecycle state, and an audit identity. Those fields must travel with every request and every metering event. A key value is only a lookup secret; it is not a billing identity.

Use an explicit state machine such as standby -> active -> retiring -> revoked. A tenant may have at most one active key for a given scope, but can have one or more standby keys with an activation deadline. Promotion is a transaction guarded by the tenant ID and a monotonically increasing version. Revocation is idempotent, so an operator can repeat it after a timeout without guessing what happened.

Three short rules prevent most of the damage:

Store only a salted hash and a visible key ID; show the secret once.
Resolve the key to a tenant before authorization, then attach that tenant ID to the usage record.
Replicate lifecycle changes before promoting a failover node, and reject stale versions.

That last rule is easy to skip. It is also where disaster plans usually become fiction.

How should a marketplace create a standby API credential for rotation and failover?

Start with a control-plane record that is independent of the serving node. A minimal schema has tenant_id, key_id, scope, state, version, created_at, activate_after, and revoked_at. The secret hash belongs in a secrets store with an access log; the metadata belongs in a strongly consistent database or a replicated log. Keeping both in one mutable cache makes an apparently healthy failover unsafe.

The create operation should be boring. Generate 256 bits from a cryptographically secure source, hash with a memory-hard password function, and write the metadata under a unique (tenant_id, scope, key_id) constraint. Mark it standby, then emit an audit event containing the tenant and key IDs but never the secret. A worker can later test promotion in a staging tenant using the same path as production.

Here is the core of a Go service. The repository and database details are deliberately plain; the important part is that attribution is resolved once and passed into metering.

package credentials

import (
    "context"
    "crypto/rand"
    "encoding/base64"
    "fmt"
    "time"
)

type KeyState string

const (
    Standby  KeyState = "standby"
    Active   KeyState = "active"
    Retiring KeyState = "retiring"
    Revoked  KeyState = "revoked"
)

type Key struct {
    TenantID string
    KeyID    string
    Scope    string
    State    KeyState
    Version  int64
}

type Store interface {
    CreateKey(ctx context.Context, tenantID, keyID, scope, hash string) error
    Resolve(ctx context.Context, presented string) (Key, error)
    RecordUsage(ctx context.Context, tenantID, keyID string, units int64) error
}

func CreateStandby(ctx context.Context, store Store, tenantID, scope string) (string, error) {
    raw := make([]byte, 32)
    if _, err := rand.Read(raw); err != nil {
        return "", fmt.Errorf("generate credential: %w", err)
    }
    secret := base64.RawURLEncoding.EncodeToString(raw)
    keyID := "k_" + secret[:12]
    // HashSecret represents a memory-hard hash with a per-key salt.
    hash, err := HashSecret(secret)
    if err != nil {
        return "", fmt.Errorf("hash credential: %w", err)
    }
    if err := store.CreateKey(ctx, tenantID, keyID, scope, hash); err != nil {
        return "", err
    }
    return keyID + "." + secret, nil
}

func Meter(ctx context.Context, store Store, presented string, units int64) error {
    key, err := store.Resolve(ctx, presented)
    if err != nil || key.State == Revoked {
        return fmt.Errorf("credential rejected")
    }
    return store.RecordUsage(ctx, key.TenantID, key.KeyID, units)
}

The returned secret should go directly to the tenant's secure delivery channel and never enter application logs. In Node.js, the caller can wrap this service over HTTPS; the protocol does not change the ordering requirement. Resolve, authorize, and attribute before performing the billable operation, then publish a usage event with a deduplication key such as (request_id, key_id).

Rotation and failover need different clocks

Rotation is a planned transition. Failover is an uncertainty event. Treating them as the same workflow creates a race between a human deadline and replication lag.

For rotation, create the standby key, distribute it, and run a canary request that proves scope and tenant attribution. Only then promote it. Keep the previous key in retiring for a bounded overlap window; two minutes may be enough for a short-lived worker, while a batch marketplace integration may need an hour. The correct value comes from observed token cache and job retry lifetimes, not a universal recipe. After the window, revoke the old key and verify that both the authorization path and the metering path reject it.

For failover, the promoted node must read the latest credential version from the control plane. If it cannot establish freshness, it should fail closed for writes and expose a clear readiness failure to the orchestrator. Serving with an unknown key map may keep HTTP green while charging the wrong tenant. My capacity-planning reflex is to budget this check explicitly: at 2,000 requests per second, a 30-second replay window represents up to 60,000 usage events that need idempotent reconciliation, even before a retry storm is counted.

The SLO should cover attribution, not just availability. A useful objective is “99.99% of accepted billable requests have a tenant and key ID attached within five minutes.” Alert on missing or conflicting identity fields, stale control-plane versions, and promotion attempts that exceed the expected replication lag. A dashboard showing only 2xx rate will miss the expensive failure.

Compare the boundaries before you buy or build

A managed account platform, a gateway, and a self-hosted secrets system solve different portions of this problem. Stripe Billing can provide established invoice primitives, but it does not by itself define your per-request credential lifecycle. Unkey focuses on API key management and usage controls, while a gateway such as Kong Gateway or Tyk can centralize enforcement at the edge. HashiCorp Vault or AWS Secrets Manager can protect secret material, but you still own tenant attribution, rotation orchestration, and the audit join between requests and charges.

Approach	Strength	Boundary to test	Operational question
Gateway plus internal control plane	Consistent edge enforcement	Billing identity may be lost in async consumers	Who owns the tenant ID after a queue hop?
Dedicated key-management service	Clear lifecycle and audit model	Extra service and replication path	Can it meet the failover freshness SLO?
Self-hosted secrets store	Policy control and portability	You operate upgrades, backups, and unseal access	Is there a staffed on-call for recovery?
Billing platform with custom auth	Mature invoices and tax workflows	Credential semantics remain application work	Can usage events be replayed without double charge?

The catch is that none of these choices removes the state machine. Choose a gateway when uniform edge policy is the dominant need. Choose a dedicated control plane when tenant-level attribution and rotation are the differentiators. Stick with a self-hosted secrets store when regulatory policy requires it and the team can carry the on-call load. A managed option is not suitable when its export or replication model cannot prove which key version authorized a charge.

Test the disaster path, then keep it small

A continuity drill should create a standby key for a disposable tenant, cut traffic to a node with an intentionally stale snapshot, and verify that writes stop until freshness returns. Then promote the standby, send duplicate requests, revoke the old key, and reconcile usage totals. Capture timestamps for creation, replication, promotion, first accepted request, and revocation. Those timestamps are more useful than a screenshot of a green health check.

Keep the runbook short enough to use at 03:00. It should name the tenant, scope, key IDs, last known version, operator, and rollback action. Never paste the secret into the ticket. Record a hash of the runbook inputs so the incident record can be compared with the eventual billing export.

I'm not sure any team can predict the perfect overlap window on its first attempt; your mileage may vary with queue depth and token caching. Measure it. The design is sound when a failover changes where requests are served without changing who gets charged, and when a revoked credential becomes unusable everywhere within the stated SLO.

DEV Community