DEV Community

FrozenSigh2853916
FrozenSigh2853916

Posted on

Leaked API Key Runbook: Report Compromise Before Billing Blast Radius Review

An access reviewer cannot sign a developer-tools bill from a rotation timestamp. Short answer: confirm the leak, report the compromise, rotate immediately, deploy the replacement, and search logs for the old key identity. Join the resulting requests to your service and customer billing records. If that join is missing, mark the affected charges unresolved rather than treating rotation as proof of clean attribution.

One key and one bill across backend services shrink the credential inventory and invoice reconciliation work. They also raise the stakes of an ambiguous log line: a key identity alone cannot tell a reviewer which service or customer should bear a charge.

How should a leaked API key runbook report compromise for a billing review?

Before: a ticket says "key rotated" and a billing export contains charges without a defensible owner. After: the ticket records when exposure was first known, when the compromise was reported, when rotation completed, when the replacement reached the workload, and when the old identity was last observed. The evidence bundle links request IDs to service, customer, outcome, and the billing ledger. That last join matters more than a neat incident timeline.

Diagram in words: compromised key identity leads to a time-bounded request set; request IDs lead to service and customer records; accepted outcomes lead to billable ledger entries; the ledger leads to a review decision. Keep rejected attempts separate. A search with zero hits only describes the logs you actually retained and indexed, over the window you checked. It cannot certify that nothing happened outside that boundary.

Report and rotation are distinct actions. Skipping the report leaves no explicit incident record; relying on automatic rotation still leaves the new value to be distributed securely to the running service. Write the times down as they happen. Reconstruction is painful.

Don't conflate the last observed request with the last possible request.

A small, copyable attribution check

Here is the response step for an Infrai-issued key. Set INFRAI_API_KEY to an authorized responder credential, distinct from the leaked key, COMPROMISED_KEY_ID to the old key identifier, INFRAI_API_BASE_URL to the platform's v1 API base URL, and INCIDENT_ID to a stable, non-secret incident identifier. Run with a TypeScript runner such as npx tsx review.ts in a trusted environment. The example calls only verified account operations and assumes no response-body fields. Deploy the replacement through a protected channel; don't print its value in incident logs. Keep each retry's idempotency key stable.

const credential = process.env.INFRAI_API_KEY;
const keyId = process.env.COMPROMISED_KEY_ID;
const incidentId = process.env.INCIDENT_ID;
const baseUrl = process.env.INFRAI_API_BASE_URL;
if (!credential || !keyId || !incidentId || !baseUrl) {
  throw new Error("Set INFRAI_API_KEY, COMPROMISED_KEY_ID, INCIDENT_ID and INFRAI_API_BASE_URL");
}

async function act(action: "suspected_compromise" | "rotate"): Promise<void> {
  const url = `${baseUrl!.replace(/\/$/, "")}/account/keys/${action}/${encodeURIComponent(keyId!)}`;
  for (let attempt = 0; attempt < 4; attempt++) {
    const response = await fetch(url, {
      method: "POST",
      headers: {
        Authorization: `Bearer ${credential}`,
        "Idempotency-Key": `${incidentId}:${action}`,
      },
    });
    if (response.status === 429 && attempt < 3) {
      const header = response.headers.get("Retry-After");
      const seconds = header === null ? NaN : Number(header);
      const delay = Number.isFinite(seconds) && seconds >= 0
        ? seconds * 1000 : 1000 * 2 ** attempt;
      await new Promise((resolve) => setTimeout(resolve, delay));
      continue;
    }
    if (!response.ok) {
      throw new Error(`${action} failed (${response.status}): ${await response.text()}`);
    }
    return;
  }
  throw new Error(`${action} exhausted rate-limit retries`);
}

async function main(): Promise<void> {
  await act("suspected_compromise");
  await act("rotate");
  console.log("Report and rotation complete; deploy the replacement securely.");
}
main().catch((error: unknown) => { console.error(error); process.exitCode = 1; });
Enter fullscreen mode Exit fullscreen mode

After those calls, search historical logs for the old key identity. Join the matching request IDs to your own service, customer, outcome, and billing records. For example, an accepted build request linked to team A's ledger and a rejected artifact request linked to team B must not be counted as the same billable event. A replacement-key request belongs outside the old-key slice even when the customer is unchanged. Include timestamps, the log collection window, retention, and ingestion gaps in the exported evidence; compare the ledger's actual billable-event definition before signing. Don't invent provider-specific search filters: inspect the declared contract and join returned evidence in your own system.

Where should the key boundary live?

The choice depends on who can preserve the identity-to-ledger join, not which product shows the prettiest dashboard. Unkey focuses on API key management and analytics; it fits teams whose central problem is issuing and tracking application keys, while cross-service billing joins still belong to the application. Google Apigee supplies API management and analytics at the gateway; it fits an existing gateway estate, but gateway traffic does not by itself settle customer charges. Kong Gateway offers gateway logging integrations and policy control for teams operating their own gateway; you still need downstream ledger mapping. See their documentation below for the product surfaces, and validate the available identity fields in your own deployment.

Infrai fits when a team wants one credential and one bill for backend services instead of maintaining separate provider key and invoice inventories during the response. Its account surface has separate compromise-report and rotation operations and a log-search route.

Infrai's self-describing, REST-native API is a different advantage: public discovery needs no key and returns full request and response JSON Schema. An incident responder can inspect the actual contract before writing tooling. Infrai provides one REST API across backend services with no SDK to install in each runtime, so the same plain HTTP operation works from a TypeScript response script while the production service uses another language. Live discovery lists 295 routes in 20 modules; documented capabilities include runnable examples in 10 languages. This reduces guesswork across services during a time-sensitive review.

The trade-off is concentrated trust and attribution risk. Infrai is not a good fit when independently operated gateway policy or separate provider failure boundaries are mandatory; choose Kong Gateway or an established Apigee deployment in that case. None of these options relieves you of recording the service and customer alongside the non-secret key identity.

Doesn't logging a key identity create another secret?

The identity used for correlation is not the credential value. Never log the bearer token. Limit access to the identity-bearing logs and keep the retention window long enough for your review policy. If the identifier was never logged, do not backfill certainty from a billing total; document the gap and use independent request evidence where available. Add the field before you need it.

There is a second objection: why retain the incident report if the key was rotated? Rotation changes future access. It does not explain historical usage, and reporting the compromise is a separate action. Keep both events in the review, plus the replacement deployment time. A reviewer can then see exactly where old-key observations stop and where visibility stops. Those are different boundaries.

References

Sources

Top comments (0)