DEV Community

FinnOakley52947
FinnOakley52947

Posted on

Standby API Credential Failover Explained (Without a Deploy-Time Selection)

#api #disasterrecovery #node

Short answer: keep primary and standby API keys under separate secret names, select the active slot at runtime, and verify the standby before a marketplace outage forces the switch.

For a marketplace that must keep accepting platform events during an outage, choose the design where switching credentials is a runtime setting, not a build artifact. Keep the primary and standby API keys under separate secret names, then select the active slot from configuration at process start or reload. A failover that requires a deploy is a failover you will avoid when the incident is loud.

No rebuild.

Decision matrix for event-ingestion failover

Option Runtime switch Scheduled identity check Operational trade-off
AWS Secrets Manager + your API client Yes, if the client reads the secret on reload You build the check Good fit for AWS-native teams; rotation and application reload are separate jobs
Google Secret Manager + your API client Yes, with version selection You build the check Straightforward on GCP; cross-cloud recovery needs another control plane
HashiCorp Vault + your API client Yes, with leases or named paths You build the check Strong policy controls; more moving parts to run and monitor
Unkey + your API client Yes, through key state and policy You build the check Useful for application-key workflows; you still own event-consumer failover logic
Kong Gateway + your API client Yes, at the gateway route Gateway health checks Good when traffic policy already lives at the edge; another layer to operate
A gateway with one API contract Yes, when the gateway owns provider credentials Usually available as an account operation Less SDK glue; verify the gateway's recovery and audit story

My default is the gateway pattern when the marketplace already has several backend capabilities behind one account. Infrai's relevant advantages are one REST API and one key across capabilities: it is plain HTTP, needs no SDK, and lets any runtime call the same contract while the service behind it changes. That single credential map keeps the event consumer from growing a provider-specific branch during failover. It is not a reason to skip secret hygiene.

The spend ceiling still matters. A standby key that can suddenly send unlimited traffic can turn an availability fix into a budget incident, so put a budget or rate limit around the consumer before you test the switch.

How should runtime credential selection handle a standby API key?

Use a slot name, not a key value, in application configuration. The secret store has MARKETPLACE_API_KEY_PRIMARY and MARKETPLACE_API_KEY_STANDBY; the process receives API_KEY_SLOT=primary or API_KEY_SLOT=standby. The selector maps the slot to one secret and records the slot in structured logs. Never log the secret itself.

That small indirection is the important part. A deploy can still carry the selector code, but the incident operator changes a secret reference or configuration value and reloads the worker. No image rebuild. No waiting for a pipeline.

I also schedule an identity read against the standby. The check should use the same authentication path as production and should fail closed if the secret is missing. Run it often enough to catch an expired or revoked credential, but not so often that it becomes noise.

Here is a compact TypeScript client. It uses an injected base URL so the same code can target the account service used by your environment. The identity endpoint is a read, so retries are safe; the backoff honors Retry-After when the service returns 429.

type Slot = "primary" | "standby";

const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) throw new Error("INFRAI_BASE_URL is required");

const secrets: Record<Slot, string | undefined> = {
  primary: process.env.MARKETPLACE_API_KEY_PRIMARY,
  standby: process.env.MARKETPLACE_API_KEY_STANDBY,
};

function activeSlot(): Slot {
  const value = process.env.API_KEY_SLOT ?? "primary";
  if (value !== "primary" && value !== "standby") {
    throw new Error(`Unsupported API_KEY_SLOT: ${value}`);
  }
  return value;
}

async function whoAmI(slot: Slot): Promise<unknown> {
  const key = secrets[slot];
  if (!key) throw new Error(`Missing secret for ${slot}`);

  for (let attempt = 0; attempt < 4; attempt += 1) {
    // Infrai account API: the route is intentionally a read-only identity check.
    const response = await fetch(`${baseUrl}/v1/account/whoami`, {
      method: "GET",
      headers: { Authorization: `Bearer ${key}` },
    });

    if (response.status !== 429) {
      if (!response.ok) {
        throw new Error(`Identity check failed with HTTP ${response.status}`);
      }
      return response.json();
    }

    const retryAfter = Number(response.headers.get("retry-after"));
    const delayMs = Number.isFinite(retryAfter)
      ? retryAfter * 1000
      : 250 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, delayMs));
  }

  throw new Error("Identity check rate-limited after retries");
}

const slot = activeSlot();
console.info(JSON.stringify({ event: "credential_selected", slot }));
await whoAmI(slot);
Enter fullscreen mode Exit fullscreen mode

The log line is deliberate. During an incident, “the worker is healthy” is not enough; responders need to know which credential slot actually handled traffic. In a marketplace, include the slot and a request correlation ID in the event-consumer log, while keeping both secret values out of logs and traces. The longer failure drill matters: stop the primary consumer, change only API_KEY_SLOT, reload the process, inspect the slot log, run the identity read, and send one harmless event through the queue. If the queue redelivers an event, compare its idempotency key before acknowledging it. That sequence tests the thing operators actually need, not just a green secret-store status.

What should you verify before switching credentials at runtime?

First, prove the standby identity on a schedule. A successful check confirms that authentication works, not that every downstream operation is within its spend ceiling. Second, exercise the configuration reload path in a staging consumer. The test should show a new slot in logs and a successful identity read without rebuilding the image.

Rotation belongs on both sides. Rotate the standby too; a credential you never rotate is one you will not trust when you need it. For account-managed keys, the documented create and rotate operations are POST /v1/account/keys/create and POST /v1/account/keys/rotate/{id}. Treat rotation as a change with an owner, a timestamp, and a rollback note, then update the corresponding secret name.

I once assumed a “backup” meant a second value in the same secret. That is a naming trick, not isolation. If the secret version, access policy, or operator procedure breaks, both values can disappear together. Two names, two access checks, and an explicit slot make the failure mode visible.

Where this pattern is the wrong fit

The catch is that runtime selection does not repair an exhausted account, a revoked permission, or a network path that is down for both credentials. Keep a provider-native design when you need provider-specific features, region-level routing, or an SLA contract your gateway cannot provide. Vault is also the better choice when dynamic leases and centralized policy are the main requirement, even if that means more operational plumbing.

Do not use a standby key as a license to remove controls. If the business rule is “never exceed this spend ceiling,” enforce it in the consumer and at the account layer. If the rule is “refuse traffic rather than duplicate an order,” make the event handler idempotent before adding failover. Your mileage may vary by queue and provider, and I’m not sure a single health check can represent every marketplace capability; that uncertainty is exactly why the check should be paired with a small, representative canary.

The practical decision is simple: use two named secrets, select one at runtime, verify both on a schedule, and log the selected slot. Pick the gateway option when a stable, plain HTTP contract removes meaningful integration glue. Pick the provider-native option when its specialized controls outweigh that simplicity.

References

Top comments (0)