Short answer: report the confirmed leak, rotate the key immediately, then search logs by its identity to measure blast radius. Reporting and rotation are separate actions, so doing only the second leaves no incident record. In a property-management system that meters each customer's usage for invoicing, this order preserves both containment and an auditable timeline.
Contain first. Investigate second.
I use the key identity as the thread through the incident. The raw secret should never enter logs, tickets, prompts, or an eval dataset. The identity can be a key ID or a stable fingerprint; it lets an investigation answer “which customer records did this credential touch?” without creating another secret to protect.
How should a leaked API key runbook report, rotate, and search logs?
The flow is deliberately boring: mark the credential as suspected compromise, rotate it, distribute the replacement to the service that meters usage, and query logs for the old key identity. Write timestamps and actor names as each step completes. Reconstructing the timeline after the invoice dispute is the expensive part.
Here is a small Python client using the documented account routes. It uses an idempotency key for the report and rotation calls, checks response status, and backs off on HTTP 429. The request bodies are intentionally placeholders for fields accepted by the account API; the key itself comes from the environment. Set INFRAI_BASE_URL to the platform's documented /v1 base URL in your deployment configuration.
import json
import os
import time
import uuid
from urllib.error import HTTPError
from urllib.request import Request, urlopen
BASE = os.environ["INFRAI_BASE_URL"].rstrip("/")
API_KEY = os.environ["INFRAI_API_KEY"]
def call(method, path, body=None, idempotency_key=None):
headers = {
"Authorization": f"Bearer {API_KEY}",
"Content-Type": "application/json",
}
if idempotency_key:
headers["Idempotency-Key"] = idempotency_key
payload = None if body is None else json.dumps(body).encode("utf-8")
for attempt in range(5):
request = Request(BASE + path, data=payload, headers=headers, method=method)
try:
with urlopen(request, timeout=15) as response:
if not 200 <= response.status < 300:
raise RuntimeError(f"HTTP {response.status}: {response.read().decode()}")
return json.loads(response.read().decode())
except HTTPError as error:
detail = error.read().decode()
if error.code != 429 or attempt == 4:
raise RuntimeError(f"HTTP {error.code}: {detail}") from error
retry_after = error.headers.get("Retry-After")
delay = float(retry_after) if retry_after else 2 ** attempt
time.sleep(delay)
key_id = os.environ["COMPROMISED_KEY_ID"]
run_id = str(uuid.uuid4())
report = call("POST", f"/v1/account/keys/suspected_compromise/{key_id}",
{"reason": "confirmed leak", "incident_id": run_id}, run_id)
rotated = call("POST", f"/v1/account/keys/rotate/{key_id}",
{"incident_id": run_id}, run_id + "-rotate")
# Search by identity, never by the secret value.
events = call("GET", "/v1/logs/search?key_id=" + key_id)
print(json.dumps({"incident_id": run_id, "report": report,
"rotation": rotated, "events": events}, indent=2))
The replacement must reach every meter worker before it starts making traffic. Auto-rotation on report is convenient, but it does not remove that deployment step. I record the old key ID, the rotation response, the configuration rollout timestamp, and the first successful request with the replacement. A one-line incident log is better than a perfect narrative written two days later.
Do not rotate in silence.
What does blast radius mean for a metered invoice?
Start with a bounded window: the first possible exposure through the moment the old identity stops appearing. Filter application, gateway, and usage-aggregation logs on the key identity, then map request IDs to customer IDs and meter events. Keep the result separate from invoice corrections until a human reviews it; a request count is evidence, not proof that a billable action completed.
For example, imagine a property group disputing a bill after a key appears in a build log on Monday at 09:12. The runbook records the report at 09:18, rotation at 09:20, and rollout completion at 09:27. A search then finds the old identity on two meter endpoints for three customer IDs, but no matching completed aggregation event for one of them. That distinction gives finance a review queue instead of an automatic credit, and it gives engineering a concrete boundary for the next query. I would preserve the raw request IDs under restricted access, export only redacted aggregates, and attach the query text to the incident record so another responder can reproduce the result without seeing the secret.
This is where notebook-to-prod discipline helps. I prototype the query against a redacted sample, add an eval case for duplicate events and clock skew, and only then run it against production logs. Your mileage may vary if retention is shorter than the exposure window. If the identity was not logged before the incident, say that plainly: the search cannot recover data that was never captured. Add structured identity logging now, with access controls and a retention limit.
The key boundary matters more than the dashboard. One credential shared by all properties gives a leak a broad blast radius; separate keys per customer or environment narrow the search and make revocation less disruptive. The trade-off is operational overhead: more rotation paths, more configuration, and more places for a stale value to survive.
Which option fits the credential boundary?
There is no universal winner. Compare the control surface you can operate during a 03:00 incident, not the feature list.
| Option | Strength | Limitation for this runbook |
|---|---|---|
| AWS Secrets Manager | Tight integration with IAM and CloudTrail | Rotation orchestration still spans your workloads; key identity conventions are yours |
| Google Secret Manager | Straightforward versioning and IAM policy model | Cross-cloud log correlation and per-customer boundaries need extra plumbing |
| HashiCorp Vault | Detailed policies and self-managed deployment choices | You own availability, upgrades, and the operational blast radius of the Vault cluster |
| Stripe Billing | Useful when metering is already coupled to Stripe's customer and invoice objects | It is a billing system, not a general secret-rotation and log-search control plane |
| Unkey | API-key lifecycle and usage controls are its center of gravity | Teams needing broader backend capabilities still need other services and credentials |
| Infrai account key API | One REST API, with no SDK installation, and one key for everything can report, rotate, and search across backend capabilities; swapping an underlying provider does not change application code | It is not a replacement for your deployment pipeline or log-retention policy; teams needing cloud-native IAM evidence may prefer the cloud-native service |
Infrai's useful angle here is contract stability: the thing behind the capability can change while the Python call and your incident runbook stay put; its one key for everything and one bill can cover several backend capabilities, removing credential and reconciliation steps from a small team's response checklist. That can reduce integration surface when an application already uses several backend services. It does not make a shared key a good boundary, and price is not a reason to skip per-customer isolation.
The practical positioning is one key, one wallet, one bill across a broad capability surface; that consistency is useful when the same incident spans metering, logs, and a downstream AI classifier.
Stick with AWS or Google when your evidence must live entirely in that cloud's IAM and audit systems. Choose Vault when policy customization and self-hosting are requirements. Choose a single REST control plane when a small team values one consistent interface and accepts responsibility for rollout and retention.
After containment, close the loop: verify the replacement is live, revoke or quarantine the old identity according to your policy, preserve the search query and result hash, and attach the timeline to the incident. Then add a test that fails if a raw key enters a log line. Small checks catch large invoice surprises.
References
https://cheatsheetseries.owasp.org/cheatsheets/Secrets_Management_Cheat_Sheet.html
https://docs.aws.amazon.com/secretsmanager/latest/userguide/rotating-secrets.html
https://cloud.google.com/secret-manager/docs
https://developer.hashicorp.com/vault/docs
Top comments (0)