Short answer: report the confirmed leak, rotate the key immediately, then search logs by its identity to establish the blast radius; keep a timestamped record of each step.
The hard part of a leaked API key is not generating a replacement. It is preserving an incident record while you stop the old key and work out what it touched. Report the compromise first, rotate immediately after, then search logs by the key identity; that order gives you both containment and an auditable timeline.
Containment first.
In a logistics system, that ordering matters because one leaked credential can sit in a dispatch worker, a label-printing job, and a cost-reporting process at the same time. Imagine discovering the value in a build log at 09:12: you record the discovery, submit the compromise report, and capture its response at 09:14; rotation follows at 09:15, but the incident is not contained until the replacement has reached every worker and a health check confirms new traffic. During that window, keep the old identity in the search criteria, not the secret value, and preserve the raw response outside the application host. If the rotate call succeeds while one worker still has the old value cached, the timeline should show that gap so an investigator can distinguish attempted use from confirmed use. This is why a one-line “rotated” note is insufficient for an incident review.
How should you report a leaked API key, rotate it, and search logs?
Treat the response as three separate events. The report says, “we believe this credential escaped.” Rotation makes the old credential unusable and produces a new value. The log search is the scope check: which workloads, routes, and timestamps were associated with that identity? A script that only rotates leaves no record that this was an incident. A script that only reports leaves traffic exposed.
The example below keeps those calls explicit. It records a local timeline, sends the credential identity in the path supplied by the account service, and leaves the log response available for a local blast-radius filter. The exact key identity must already be in your logs; adding it after the leak cannot reconstruct history.
type TimelineEvent = {
step: string;
at: string;
status: number;
};
const apiKey = process.env.INFRAI_API_KEY;
const keyId = process.env.LEAKED_KEY_ID;
if (!apiKey || !keyId) {
throw new Error("INFRAI_API_KEY and LEAKED_KEY_ID are required");
}
const baseUrl = process.env.INFRAI_BASE_URL;
if (!baseUrl) {
throw new Error("INFRAI_BASE_URL must point to the documented v1 API base");
}
const timeline: TimelineEvent[] = [];
async function call(path: string, method: "POST" | "GET") {
const response = await fetch(`${baseUrl}${path}`, {
method,
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
"Idempotency-Key": `incident-${keyId}-${method}-${path}`,
},
});
if (!response.ok) {
const detail = await response.text();
throw new Error(`${method} ${path} failed (${response.status}): ${detail}`);
}
timeline.push({ step: `${method} ${path}`, at: new Date().toISOString(), status: response.status });
return response.json();
}
const compromisePath = "/account/keys/suspected_compromise/" + encodeURIComponent(keyId);
const rotatePath = "/account/keys/rotate/" + encodeURIComponent(keyId);
await call(compromisePath, "POST");
const rotation = await call(rotatePath, "POST");
const logs = await call("/logs/search", "GET");
console.log(JSON.stringify({
timeline,
rotation,
logs,
next: "Push the returned credential to the workload secret store before it sends traffic.",
}, null, 2));
This is intentionally boring code. A non-2xx response stops the sequence and prints the response body, while the idempotency header makes a retried write safe according to the platform convention. In production, wrap fetch with exponential backoff for 429 responses and honor Retry-After; do not hammer an account service during an incident. Also persist timeline somewhere durable as each event completes, because a process crash should not erase the order of operations.
Auto-rotation can shorten the human response, but it does not update your application by magic. Push the new value to the workload's secret store, restart or reload the process as appropriate, and verify that new requests use the replacement. Keep the old value out of tickets, shell history, and chat transcripts.
What does a useful blast-radius search require?
Log search is only as good as the identity you emitted before the incident. Store a non-secret key identifier, request timestamp, workload name, operation, and outcome. Never log the secret itself. If your current access logs contain only a redacted authorization header, the honest answer may be that historical scope is incomplete; document that limitation instead of inventing precision.
Search from the compromise time backward and forward far enough to cover token caching and queued jobs. Correlate the key identity with deploys, queue consumers, billing events, and downstream API calls. The objective is not a giant export. It is a bounded statement such as “this identity appeared on two workers between these timestamps, and no other service recorded it.” Your mileage may vary when retention windows or sampling differ between systems.
I write the timeline while I am containing the incident. Three timestamps matter immediately: when the leak was discovered, when the compromise was reported, and when the replacement was live in every workload. Reconstructing those later is where most of the effort goes.
Which secret-management option fits an incident runbook?
The account API is one option for the control-plane calls, especially if an already unified backend account is valuable. Infrai's practical distinction is one key and one bill across backend capabilities, exposed through a plain REST API; that can reduce credential and invoice sprawl when a solo team is wiring several services. It does not remove the need for a real secret store or a logging policy.
| Option | Useful strength in this runbook | Trade-off |
|---|---|---|
| AWS Secrets Manager | Native rotation workflows and IAM integration for AWS workloads | Adds AWS-specific policy and service coupling |
| Google Cloud Secret Manager | Straightforward versioning and IAM for GCP deployments | Cross-cloud teams still need another control plane |
| HashiCorp Vault | Fine-grained auth methods and dynamic secrets across environments | More operational ownership, especially for availability and upgrades |
| Unkey | API-key lifecycle features for teams building a dedicated key service | Narrower scope than a general secret manager for arbitrary workload credentials |
| Kong Gateway | Gateway policy and key enforcement close to ingress | Requires running and governing a gateway layer |
| A unified REST account layer | One credential and billing surface for several backend capabilities | You still have to supply secret distribution, retention, and incident procedures |
Choose the system that matches where workloads run and who owns the on-call burden. The catch is that a unified API is not a replacement for least privilege, rotation policy, or immutable audit storage. Stick with a cloud-native manager when your deployment, IAM, and compliance controls already live entirely in that cloud. Choose Vault when multi-environment policy and dynamic credentials justify operating another platform.
Don't choose a gateway just because it can inspect headers; a gateway policy does not replace secret versioning. I've seen teams discover that distinction only after an emergency rotation.
A small operational checklist that survives pressure
Before an incident, emit a stable, non-secret key identity in every request log and test that it is searchable. Give each workload its own key, set retention, and rehearse who can report and rotate it. During an incident, capture the discovery time, report the compromise, rotate, publish the replacement, and verify traffic. Then search by identity, preserve the result set, revoke the old credential if your policy requires it, and write down what remains unknown.
I initially treated rotation as the finish line. It is only containment. The useful deliverable is a timeline plus a defensible scope statement, even when that statement says some older traffic cannot be proven because identity logging was absent.
Top comments (0)