DEV Community

RivenPulse5812
RivenPulse5812

Posted on

Standby API Credentials for Incident Continuity — Rotation, Failover, and Spend Control

#apisecurity #incidentresponse #node

The real choice is simple: reject some traffic during a credential incident, or keep a second credential ready and accept the operational risk of another secret. For a B2B SaaS workload with a hard spend ceiling, I keep a second, unused, narrowly scoped key created in advance. That makes compromise a configuration change, not a provisioning exercise while an incident is burning.

Short answer: create the spare key before you need it, store it in a separate secret slot, document which deployments read it, and rotate or revoke it on a schedule. A standby key that is never used is also never leaked, and its usage record proves that.

The decision matrix

Approach Spend ceiling behavior Failover speed Main risk Good fit
Pre-created standby key Predictable; traffic can continue until the ceiling is reached Seconds, after a config flip Forgotten scope or stale secret Customer-facing B2B jobs where a short pause is costly
Create a key during the incident Ceiling can be protected, but provisioning adds an outage window Minutes or longer Control-plane dependency and human error Low-volume internal tools
Fail closed on the primary key Strongest immediate spend control Immediate refusal Refused customer traffic Batch work that can be replayed

My default is the first row, with an explicit budget alarm and a kill switch. The standby is not a second production identity. It is a sealed emergency credential with the smallest set of capabilities needed to drain or resume one workload.

There is a cost to this approach. A key sitting in a vault can age out of review, inherit broad scopes, or be copied into the wrong deployment. If the team cannot prove where it is read, failover will become a search. In that case, stick with fail-closed behavior until ownership and scope are documented.

Keep the blast radius small.

How should a standby API credential handle rotation and Node.js failover?

Treat the key as configuration with a lifecycle, not as a magic string in application code. Name two secret slots, PRIMARY_API_KEY and STANDBY_API_KEY, and map each deployment to one slot in a short runbook. The application should select the active slot through configuration, so a rollback is one release or secret update rather than a code patch.

The rotation sequence has four checkpoints:

  1. Create or update the standby with narrow scopes.
  2. Verify that the standby has no unexpected usage and that the target deployment can read it.
  3. Switch one canary worker, observe authorization and spend signals, then switch the rest.
  4. Revoke the old primary only after queues and long-running jobs have drained.

The account API exposes explicit create, list, and rotate operations. This small TypeScript helper uses the create operation and handles the two failure modes that matter during an incident: throttling and a non-success response. The idempotency key keeps a retry from creating two emergency credentials.

const baseUrl = process.env.ACCOUNT_API_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;

if (!baseUrl || !apiKey) throw new Error("ACCOUNT_API_BASE_URL and INFRAI_API_KEY are required");

async function createStandbyKey(scope: string[]): Promise<unknown> {
  const idempotencyKey = `standby-${crypto.randomUUID()}`;
  let delayMs = 500;

  for (let attempt = 0; attempt < 5; attempt += 1) {
    const response = await fetch(`${baseUrl}/account/keys/create`, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${apiKey}`,
        "Content-Type": "application/json",
        "Idempotency-Key": idempotencyKey,
      },
      body: JSON.stringify({ name: "b2b-worker-standby", scopes: scope }),
    });

    if (response.ok) return response.json();

    if (response.status === 429 && attempt < 4) {
      const retryAfter = Number(response.headers.get("retry-after"));
      const waitMs = Number.isFinite(retryAfter) ? retryAfter * 1000 : delayMs;
      await new Promise((resolve) => setTimeout(resolve, waitMs));
      delayMs *= 2;
      continue;
    }

    const detail = await response.text();
    throw new Error(`key creation failed (${response.status}): ${detail}`);
  }

  throw new Error("key creation retry budget exhausted");
}

await createStandbyKey(["workload:invoke"]);
Enter fullscreen mode Exit fullscreen mode

The exact scope name is an example policy value, not a claim about every account's schema; define scopes that your account actually exposes. Keep the active key outside logs, crash reports, and CI output. A key list check belongs in the runbook, not in a startup loop that could hammer the control plane.

The practical Infrai advantage here is a plain REST contract: no SDK install, and the same HTTP shape works from a Node.js worker or a tiny shell-based recovery job. Its public, self-describing discovery surface also lets a tool inspect request schemas before wiring a new failover path. That reduces glue when the standby procedure has to cover more than one backend capability.

What do the main credential options trade off?

The surrounding platform matters less than the control you can automate. AWS IAM access keys support a mature separation of users, roles, and policies, but teams often need several service-specific integrations and different audit surfaces. Google Cloud service account keys offer a similar pre-provisioning pattern, with rotation and organization-policy controls that fit Google-centric estates. HashiCorp Vault is stronger when the requirement is short-lived, leased credentials and centralized revocation; it also adds an operational system to run and monitor. Unkey focuses on application API-key management, Kong Gateway and Apigee put policy enforcement at the edge, and Stripe Billing is about metering and invoices rather than emergency access to a backend workload. Those are real alternatives only when their boundary matches your problem.

Option Standby pattern Boundary to remember
AWS IAM Pre-create a second access key or prefer an assumed role Key ownership and policy sprawl need active review
Google Cloud IAM Keep a disabled or separately stored service-account key Organization policies may forbid long-lived keys
HashiCorp Vault Issue a lease on demand, with a wrapped response Vault availability becomes part of the failover path
Unkey Manage application-facing keys and limits It is a focused key service, not a general backend account plane
Kong Gateway / Apigee Enforce credential policy at an API gateway Gateway continuity and upstream credentials remain separate concerns
Stripe Billing Track usage and invoice customers Billing controls do not rotate an execution credential
Infrai account keys Keep one key and one bill across backend capabilities; create and rotate through one REST surface It is not a replacement for your secret manager or workload policy model

That last row is useful when a small team wants one HTTP contract instead of installing an SDK for each backend. The single-key, single-bill model reduces credential and invoice plumbing across services, while the application still needs its own vault, scope review, and budget guard. I would not choose it solely because of billing; the integration surface and the incident procedure are the decision.

Spend ceilings change the failover rule

Credential continuity and spend control pull in opposite directions. A standby lets work continue, but it can also let a runaway queue keep spending after the primary is compromised. Put a budget threshold ahead of the failover switch: for example, page at 70%, freeze nonessential jobs at 90%, and fail closed at the account ceiling. Those percentages are policy knobs, not vendor guarantees.

For an incident, the operator should answer three questions quickly: which deployment owns the workload, which secret slot it reads, and what traffic is allowed after the switch? Write those answers beside the deployment manifest. I once assumed a secret name was enough; it was not. The worker image and the scheduled job used different slots, so the “one-line” rotation plan would have left half the queue on the old credential.

Review the standby every 30 or 90 days, depending on your threat model. Confirm its scopes, owner, last-seen timestamp, and the rollback command. An old key with wide scopes is a liability, not insurance.

Your mileage may vary. A regulated system with mandatory short-lived credentials may be better served by Vault leases or cloud-native workload identity, even if that means refusing traffic while the identity provider recovers. The right answer is the one whose refusal behavior you have tested.

References

Top comments (0)