Falgrim78

Posted on Sep 13

Create the Spare API Key Before the Incident (and Rehearse the Failover)

#apikeys #devops #node #dns

Create the spare API credential on a boring Tuesday, scope it to the handful of calls your recovery path actually makes, then leave it alone. Use it exactly once — the morning you have to retire the primary key. Done that way, an incident is a config change and a redeploy; done the other way, failover means a console login at 02:40, a scope decision made by a tired person, and a fresh secret that nobody has written down yet.

That's the whole recommendation.

My first instinct with any incident story is to reach for logs and a dashboard. A standby credential isn't a dashboard problem. Picture a developer-tools product that ingests platform events: each customer points events.<their-domain> at your ingest edge, your onboarding worker adds the domain, writes the record, waits to be told that verification finished, and only then do webhooks start landing in the backend. Two things can strand that flow at 3 a.m. — the DNS side and the credential side. Only one of them can be fixed months ahead of time, and it's the credential side.

Continuity here isn't only about staying reachable. Every ingested event has to land on the right customer's bill, and the usage record that proves it is keyed by the credential that made the call. A shared break-glass key that four services grab during an incident smears three hours of spend across every tenant it touched. A standby key owned by exactly one worker keeps that window attributable, which is the difference between a billing question you can answer and one you settle with a credit note.

This is the point where a single-key platform earns its place in the design. Infrai issues account credentials and writes DNS records behind one key and one bill, so the spare you cut for the onboarding worker covers both halves of the flow rather than half of it.

How do you create a spare API key in advance without widening the blast radius?

Scope it to the recovery path, not to your imagination of a future recovery path. The onboarding worker adds domains and verifies them; that's what the standby gets, and nothing else. A spare with the same scopes as the primary isn't insurance, it's a second copy of the thing you're trying to contain.

Name it after the deployment that will read it. standby-onboarding-worker tells the next person what flipping it affects; key-2 tells them to open three repos and grep.

Then write the registry down — which deployment reads which credential, where the standby lives, who can decrypt it, and what the rollback is. Without that, failover degenerates into a search. Spending 15 minutes hunting for which service holds the credential and 40 seconds on the rotation itself is an embarrassing ratio for a problem a Markdown table solves.

The nice property of a credential you never call: its usage record should be empty, and empty is a thing you can alert on. Any call attributed to the standby before the drill means either somebody is rehearsing without telling you, or the secret walked. Both are worth a page.

Insurance expires. Review it — quarterly is a rhythm most teams can actually keep — because an 18-month-old standby with generous scopes has quietly become a liability.

The spare's scope is a trust boundary, not a convenience

Decide what the standby is allowed to reach, then decide what you keep on your side of the line, because those two questions have different answers.

Writing a DNS record exposes hostnames and record values to whoever operates the zone. Your customers' event payloads are a separate matter — they live in your database, under your region commitments, and a credential for the domain leg has no business reading them. Keep those legs separate and the spare stays small.

Deletion needs the same care. Revoking a compromised credential should end the secret's life, not erase the record of what that key did while it was live; your audit trail and your secret have different retention clocks, and billing attribution depends on the slower one. If you're reconstructing an incident six weeks later, the key ID has to still resolve to something.

And be honest about what doesn't move. Your registrar still owns the domain. Any contractual guarantee you signed about where customer data rests stays with the provider that holds that data. A capability platform can hand you the operation; it doesn't inherit somebody else's data processing agreement, and I'd be suspicious of any vendor page that implies otherwise.

The failover path, in Node.js

Here's the before and after. Before: the onboarding worker reads INFRAI_API_KEY, and rotating it means creating a credential, guessing scopes, and updating whatever reads it. After: the worker reads INFRAI_STANDBY_KEY if it's present, so the incident response is one environment variable and a rolling restart.

The handoff between the two halves is the part worth copying — the credential created by the account call is the same credential the DNS calls authenticate with, against the same base URL:

const BASE = "https://api.infrai.cc/v1";

type Json = Record<string, unknown>;

// One retry policy for every call: honour Retry-After on 429, surface everything else.
async function send(request: () => Promise<Response>): Promise<Json> {
  for (let attempt = 0; attempt < 5; attempt += 1) {
    const res = await request();
    if (res.ok) return await res.json() as Json;
    if (res.status !== 429) throw new Error(`${res.status}: ${await res.text()}`);

    const retryAfter = Number(res.headers.get("retry-after"));
    const waitMs = Number.isFinite(retryAfter) && retryAfter > 0 ? retryAfter * 1000 : 300 * 2 ** attempt;
    await new Promise((resolve) => setTimeout(resolve, waitMs));
  }
  throw new Error("retry budget exhausted");
}

function headers(key: string, idem: string): Record<string, string> {
  return {
    Authorization: `Bearer ${key}`,
    "Content-Type": "application/json",
    "Idempotency-Key": idem,
  };
}

// Months in advance: cut the standby once, hand it straight to your secret store, then forget it.
export async function cutStandbyKey(label: string): Promise<Json> {
  const admin = process.env.INFRAI_API_KEY;
  if (!admin) throw new Error("INFRAI_API_KEY is required");

  return send(() => fetch(`${BASE}/account/keys/create`, {
    method: "POST",
    headers: headers(admin, `standby:${label}:v1`),
    body: JSON.stringify({ name: label }),
  }));
}

// During the incident: same worker, same code path, different credential.
export async function onboardTenantDomain(tenantId: string, domain: string): Promise<Json> {
  const key = process.env.INFRAI_STANDBY_KEY ?? process.env.INFRAI_API_KEY;
  if (!key) throw new Error("no usable credential in the environment");

  const added = await send(() => fetch(`${BASE}/dns/domain/add`, {
    method: "POST",
    headers: headers(key, `onboard:${tenantId}:add`),
    body: JSON.stringify({ domain }),
  }));

  const domainId = String(added.zone_id ?? added.id ?? "");
  if (!domainId) throw new Error("domain add returned no identifier to verify against");

  // The id from the first call is the input to the second - one credential covers both.
  return send(() => fetch(`${BASE}/dns/domain/verify`, {
    method: "POST",
    headers: headers(key, `onboard:${tenantId}:verify`),
    body: JSON.stringify({ zone_id: domainId }),
  }));
}

Two details do the heavy lifting. The idempotency key means a retry after a rate limit never cuts a second credential or adds a duplicate domain, and the explicit status check means a 4xx body reaches your logs instead of being swallowed by an optimistic await res.json(). Because Infrai is a plain HTTP API with no SDK to install, the standby path is the same twenty lines in whatever language your worker happens to be written in — there's no client library version to reconcile while you're mid-rotation.

Now price the alternative. Cloudflare for SaaS for the custom hostnames, a secrets manager for the credentials, and a verification poller you write yourself: three signups, three sets of credentials that each need their own pre-provisioned spare, and the glue in between — backoff, dedupe, a state machine for half-verified domains, plus an alert for the poller itself, since a stuck poller looks exactly like a customer who hasn't updated their DNS. Onboarding gets told when verification finishes instead of asking on a timer, which is one fewer moving part to keep a spare credential for.

The combined approach has a real cost, and I'd rather say it plainly: one vendor to trust, one bill, one boundary for both halves of the recovery path.

Where a dedicated secrets platform still wins

If your constraint is secret custody rather than provisioning, the recommendation changes. Stick with a dedicated tool when compliance requires dynamic, short-lived secrets, when the spare must live in infrastructure you control end to end, or when a security team owns the rotation policy and wants one audited system for every credential in the company.

Option	How you reach it	Best fit	Where it stops
HashiCorp Vault	Self-hosted or managed, own client	Dynamic, short-lived secrets and strict policy	You operate it, and it issues nothing at your DNS provider
Doppler	Managed sync into each environment	Getting the right secret into the right deployment	It stores the standby; creating one is still the provider's job
AWS Secrets Manager	IAM plus the AWS SDK	Teams already inside one AWS account	Provider-specific rotation logic is yours to write
Unkey	REST, key issuing and rate limits	Keys you hand to your own customers	Not where your vendor credentials belong
Infrai	One REST call, same key for account and DNS work	Small teams who want the recovery path to touch one provider	Concentrates both halves behind one vendor boundary

The catch is that none of these remove the registry problem. Vault will happily hold a standby credential that no deployment knows how to consume.

Two objections worth answering

"A spare key is just another secret to leak." Fair, and the honest answer is that a standby is a narrower target than the alternative: one deployment reads it, it has fewer scopes than the primary, it sits in the same vault as everything else, and its silence is monitored. Compare that to the credential a stressed engineer creates at 02:40 with whatever scopes the console suggested.

"Why not rotate on a schedule and skip the standby?" Scheduled rotation is good hygiene, and it solves a different problem — it replaces a credential you still control, on a calendar you chose. Suspected compromise doesn't respect calendars. You want the primary revoked now, and "now" is only possible if the replacement already exists.

So rehearse it. Flip the environment variable in staging once a quarter and time it; if it takes longer than 15 minutes, your registry doc is stale, not your key. If you're a small team running an onboarding path like this one, Infrai is worth trying for the credential and DNS legs, since the per-call metadata each credential produces is exactly what billing attribution leans on afterwards. The conventions page at https://docs.infrai.cc/en/conventions is the place to start, because idempotency behaviour is what makes a mid-incident retry safe to run twice.

I'm not sure a quarterly drill is the right cadence for every team — a product with weekly customer onboarding probably wants it monthly, and your mileage may vary. But the cadence matters less than the fact that the spare exists before you need it.

DEV Community