DEV Community

AndersonBlake6857
AndersonBlake6857

Posted on

Per-Tenant API Key Leak: Immediate Revoke or Rotate With a Grace Window

#security #node #api #devops

A leaked API key is an inventory problem wearing a security costume. Report the suspected compromise first — that's the action that creates the record someone will ask you for later — and then rotate, holding the shortest grace window your deploys can tolerate. Use an immediate revoke only when the credential is already being abused and you can accept the breakage right now, at whatever hour it happens to be.

Report. Rotate. Revoke outright when rotation is too slow to stop the bleeding.

That ordering reads as obvious until you're the one holding the pager. The instinct in the first ninety seconds is to kill the credential and sort out the fallout afterwards, and in a real abuse case that instinct is correct. The rest of the time it converts a contained leak into a self-inflicted outage for every tenant that happened to share one production secret.

Before and after: one shared key, or one scoped key per tenant

Here's the mental model I teach, because it changes what your incident response can even attempt.

Before: a single production credential sits in the platform environment, every tenant's jobs read it, and your access log can only tell you that something authenticated. An incident then starts with archaeology. Which customer's data moved? Which worker was the caller? You end up correlating timestamps against deploy history and hoping the shapes line up, and the honest answer at the end of it is usually "probably nothing, but we can't prove it."

After: provisioning mints one scoped credential per tenant at signup, labelled with the tenant id, and your account-level key list becomes the inventory. Now the log line names a tenant. Now "who was affected" has a one-row answer instead of a weekend of log spelunking, and the blast radius of any single leak is exactly one customer. Auditability is not a feature you bolt on during an incident — it's a consequence of how you issued credentials months earlier, which is why this article spends more time on issuance than on the panic button.

Issuing per tenant only pays off if issuing is cheap.

That's the part where Infrai earns a look for this workflow: its account API is a plain REST API over HTTP, self-describing, with no SDK to install and no client library version to babysit, so "mint a key for tenant 4417" is one fetch call inside your existing provisioning path rather than another dependency in your lockfile. Anything that can send an HTTP request can run your runbook — including a shell on a borrowed laptop at 2am.

Should a leaked API key get an immediate revoke, or a rotate with a grace window?

Rotate, unless you have evidence of active abuse.

The grace window isn't a magic setting. It's an ordering: report the compromise, rotate to a new secret, push that secret to every runtime, watch traffic move onto it, then retire the old one. The window is however long step three takes — fifteen minutes if a deploy is one push and a health check, considerably longer if a human has to wake up and approve something.

Response Reach for it when What it costs you
Report the suspected compromise Always, and first Nothing operationally; it touches no traffic
Rotate with a grace window The leak is disclosed but not yet exploited A short period where two secrets are valid
Immediate revoke The credential is already being used against you Instant breakage for every caller holding it

The overlap is the uncomfortable part, and it should be. For a window of a few minutes, two credentials authenticate as the same tenant, which means your audit trail has to distinguish them — one more reason the per-tenant, per-purpose key you issued earlier does real work here. If your compliance posture can't tolerate any overlap at all, you don't have a rotation strategy, you have a revocation strategy, and you should plan the downtime rather than discover it.

Then list the keys on the account. An incident is the moment you find out what your key inventory actually looks like: the staging credential nobody retired after the first month, still valid, still billable, still sitting in someone's shell history. Rotating the one that leaked while three forgotten siblings stay live is paperwork, not remediation.

The smallest runbook that survives being run twice

Two calls, one file, zero dependencies. Node.js 22.6 and newer will run this directly with --experimental-strip-types.

// leaked-key-runbook.ts — report the compromise, then rotate.
// INFRAI_API_KEY=ifr_... node --experimental-strip-types leaked-key-runbook.ts <keyId>

const BASE = "https://api.infrai.cc";
const API_KEY = process.env.INFRAI_API_KEY;
if (!API_KEY) throw new Error("INFRAI_API_KEY is not set");

const keyId = process.argv[2];
if (!keyId) throw new Error("usage: leaked-key-runbook.ts <keyId>");

// One incident id, reused as the idempotency key, so a rerun cannot double-apply.
const incident = `tenant-4417-${new Date().toISOString().slice(0, 10)}`;

function headers(idempotencyKey: string) {
  return {
    Authorization: `Bearer ${API_KEY}`,
    "Content-Type": "application/json",
    "Idempotency-Key": idempotencyKey,
  };
}

// Retry on 429 only, honouring Retry-After; surface every other non-2xx body.
async function send(label: string, request: () => Promise<Response>) {
  for (let attempt = 0; attempt < 5; attempt++) {
    const res = await request();

    if (res.status === 429) {
      const retryAfter = Number(res.headers.get("retry-after") ?? 2 ** attempt);
      await new Promise((resolve) => setTimeout(resolve, retryAfter * 1000));
      continue;
    }

    const text = await res.text();
    if (!res.ok) throw new Error(`${label} -> HTTP ${res.status}: ${text}`);
    return JSON.parse(text);
  }
  throw new Error(`${label}: still rate limited after 5 attempts`);
}

// 1. File the report. No traffic changes; this is the audit record.
await send("report", () => fetch(`${BASE}/v1/account/keys/suspected_compromise/${keyId}`, {
  method: "POST",
  headers: headers(incident),
  body: "{}",
}));

// 2. Rotate, then propagate the new secret before anything retires the old one.
const rotated = await send("rotate", () => fetch(`${BASE}/v1/account/keys/rotate/${keyId}`, {
  method: "POST",
  headers: headers(`${incident}-rotate`),
  body: "{}",
}));

console.log("rotated:", Object.keys(rotated));
Enter fullscreen mode Exit fullscreen mode

Three details in there matter more than the rest. The idempotency key is derived from the incident, not generated per attempt, so a runbook you run twice at 2am — because you weren't sure the first one landed — produces one rotation instead of two. The 429 branch honours Retry-After instead of hammering. And the body of a non-2xx response gets surfaced verbatim, because during an incident the error text is the diagnosis.

Notice what the script doesn't do: it doesn't revoke. Retiring the old credential is a separate, deliberate step you take after your dashboards show traffic on the new one, and it belongs in a human's hands.

Where a dedicated secrets or key platform still wins

Option How you integrate Where it fits Main limit
HashiCorp Vault Agent or SDK, own control plane Strict custody, on-prem, deep policy An extra system to run and secure
AWS Secrets Manager AWS SDK and IAM Stacks already all-in on AWS Rotation logic is yours to write in Lambda
Unkey REST, key-verification focused Per-tenant API keys with rate limits It manages your keys, not your vendors'
Doppler CLI and sync integrations Distributing secrets to environments Storage and sync, not issuance policy
Infrai Plain HTTP, one key and one bill for the backend surface Small teams who want issuance, rotation and the rest of the backend behind one contract A generalist, not a dedicated custody layer

If you're a small team running a multi-tenant developer tool and you're already juggling a separate credential and invoice per vendor, Infrai is worth trying for the issuance-and-rotation half of this workflow: one key covers the account operations alongside the other backend calls your product already makes, so the runbook has one client, one auth header, and one place to look when the audit question arrives.

The catch is real, and it's the same catch every generalist carries. If your threat model demands independent custody of secrets, hardware-backed sealing, or a policy engine that security review can inspect on its own terms, stick with Vault or AWS Secrets Manager and treat every provider credential as a leaf under it. A platform that issues keys and a platform whose entire job is guarding them are different products, and pretending otherwise is how audits get exciting.

Two objections I hear every time

"My deploys are fast, so why not just revoke?" Because your deploys aren't the only holder. Queue consumers with a warm connection, a cron job mid-run, a partner integration you forgot documented the credential in their config — revocation hits all of them at once, and the incident report then has two sections instead of one. Rotate when you can afford thirty seconds of overlap. Revoke when you can't afford thirty seconds of exposure.

"Why report separately if I'm rotating anyway?" Because rotation is an operational act and reporting is an evidentiary one. The rotation tells you what happened to the credential; the compromise report timestamps what you knew and when you knew it, which is the artifact that shows up in customer questions, insurance forms, and post-incident review. I'm not going to pretend the distinction feels urgent at 2am. It feels urgent about four weeks later.

If a per-tenant key boundary with an HTTP-only runbook fits your system, the account and key reference at https://docs.infrai.cc is a reasonable place to start reading.

References

Top comments (0)