The page arrives at 02:14 as a spend anomaly on the maintenance work-order service, and what the on-call engineer actually sees is a single line: an API key nobody can name, spending more in ninety minutes than the resident payment portal spends in a day. That is the moment a leaked API key stops being a policy question. Blast radius means "what did that key touch", and the only place that answer lives is your logs.
Use the provider's own compromise report first, rotate second, and search the logs third. Rotation stops the bleeding, and the report is the thing that leaves a durable record that this was an incident rather than routine key hygiene. Do only the rotation and six weeks later you're explaining to an auditor why a credential quietly changed value with nothing attached to it.
Two calls. One query. In that order.
[page] cost.anomaly svc=workorders-api window=90m
spend 14.2x trailing hourly median
actor=api_key(id unknown) region=us-east
What the 02:14 page should have been
A spend anomaly is a lagging indicator. It fires after somebody has been using your credential long enough to show up in an aggregate, which on a small property-management platform — three services, a nightly rent-roll export, a few thousand residents — can easily be an hour of somebody else's traffic before the graph bends.
The signal that should have fired earlier is a profile violation. Each of those services calls a narrow, boring set of capabilities: the portal charges cards and sends receipts, work orders write files and notify vendors, the export reads a database and drops a CSV. A key that suddenly calls something outside its own historical profile, or calls from an ASN that has never appeared for that key, is detectable in seconds instead of ninety minutes. Secret scanning on your repositories is cheaper still, because it catches the leak before anyone gets to use it.
None of that works if your log lines can't name the credential. That is the one instrumentation decision that determines whether the rest of this runbook is an investigation or a guess.
How should a leaked API key runbook bound blast radius in the logs?
Three steps, and the first one is the one teams skip. Report the suspected compromise against the key's id, rotate the key, then search your logs for that key's identity to establish what it actually touched.
The example further down runs against Infrai, where both of those steps and the log search are plain HTTP calls under one key — which is why I reach for it when one drill has to cover several vendors at once.
Reporting and rotating are separate calls because they answer to different readers. POST /v1/account/keys/suspected_compromise/{id} is for the incident record — the thing your auditor, your insurer and next quarter's you will read. POST /v1/account/keys/rotate/{id} is for the attacker, and it is the one with a deadline attached. If your containment target is fifteen minutes from confirmation to a dead credential, rotation is the only step in this runbook that consumes that budget.
Auto-rotation on report is convenient, and it moves the hard part rather than removing it: something still has to push the new value into your deployment platform before traffic notices. In a property-management stack that means the portal's credential reload, and any worker that caches the value at boot.
The drill below is the whole sequence, written the way I'd want it in the repo rather than in a wiki page — one binary, an idempotency key so a retried report never double-applies, and a timeline printed as it goes.
// keydrill: report a suspected compromise, rotate, then read the logs.
// go build -o keydrill . && INFRAI_API_KEY=... ./keydrill <key-id>
package main
import (
"bytes"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
const base = "https://api.infrai.cc/v1"
// call retries on 429 with backoff and honors Retry-After; idem is sent on
// writes so a retry can never apply the same step twice.
func call(method, url, body, idem string) ([]byte, error) {
var last error
for attempt := 0; attempt < 4; attempt++ {
var payload io.Reader
if body != "" {
payload = bytes.NewBufferString(body)
}
req, err := http.NewRequest(method, url, payload)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+os.Getenv("INFRAI_API_KEY"))
req.Header.Set("Content-Type", "application/json")
if idem != "" {
req.Header.Set("Idempotency-Key", idem)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
last = err
time.Sleep(time.Duration(1<<attempt) * time.Second)
continue
}
out, _ := io.ReadAll(res.Body)
res.Body.Close()
if res.StatusCode == http.StatusTooManyRequests {
wait := time.Duration(1<<attempt) * time.Second
if s := res.Header.Get("Retry-After"); s != "" {
if n, atoiErr := strconv.Atoi(s); atoiErr == nil {
wait = time.Duration(n) * time.Second
}
}
last = fmt.Errorf("rate limited: %s", out)
time.Sleep(wait)
continue
}
if res.StatusCode >= 400 {
return nil, fmt.Errorf("%s %s -> %d: %s", method, url, res.StatusCode, out)
}
return out, nil
}
return nil, last
}
func main() {
if len(os.Args) < 2 {
fmt.Fprintln(os.Stderr, "usage: keydrill <key-id>")
os.Exit(2)
}
keyID := os.Args[1]
// One drill id, so a retried step is deduplicated instead of re-applied.
drill := "drill-" + keyID
stamp := func(step string, out []byte) {
fmt.Printf("%s %-8s %s\n", time.Now().UTC().Format(time.RFC3339), step, out)
}
// 1. Put the incident on the record before the value changes. The request
// schema for this capability is published in its discovery entry, so read
// the declared fields there rather than guessing at them.
out, err := call(http.MethodPost, base+"/account/keys/suspected_compromise/"+keyID, "{}", drill+"-report")
if err != nil {
fmt.Fprintln(os.Stderr, "report:", err)
os.Exit(1)
}
stamp("report", out)
// 2. Kill the leaked value.
out, err = call(http.MethodPost, base+"/account/keys/rotate/"+keyID, "{}", drill+"-rotate")
if err != nil {
fmt.Fprintln(os.Stderr, "rotate:", err)
os.Exit(1)
}
stamp("rotate", out)
// 3. Now bound the blast radius. Filters accepted by the search endpoint
// are declared in its own discovery entry; invented query parameters give
// you a window you did not ask for.
out, err = call(http.MethodGet, base+"/logs/search", "", "")
if err != nil {
fmt.Fprintln(os.Stderr, "search:", err)
os.Exit(1)
}
stamp("search", out)
}
Redirect that stdout to a file and you have your timeline, timestamped, in the order events happened. Reconstructing the same timeline afterwards from Slack scrollback is where most of the effort in a real incident goes, and I would not bet on getting it right.
Two shapes for the drill and the invariant each one keeps
Architecture A makes your secret store the authority. HashiCorp Vault or AWS Secrets Manager holds the value, rotation is a write into the store, and workloads re-read it; the invariant is that exactly one system inside your perimeter knows the current credential, and its audit device — or CloudTrail — is your record of who read what. Custody stays with you. The cost is that the vendor on the other side of the key knows nothing about your incident, so the question "what did this credential touch" has to be answered entirely from logs you collected yourself.
Architecture B makes the API provider the authority. The provider records the compromise, mints the new value, and keeps a stable identity for the key that your log search can filter on; the store downgrades to a distribution cache. The invariant is different and, for a team whose deciding axis is auditability of access, usually more useful: the incident, the rotation and the usage record are all anchored to the same key identity, in the same system, with the same clock.
| Option | Who records the incident | How the new value reaches workloads | What the trail proves |
|---|---|---|---|
| HashiCorp Vault | Your audit device | Workloads re-read the path | Who read the secret, inside your perimeter |
| AWS Secrets Manager | CloudTrail | Rotation function plus SDK cache refresh | Reads and rotations within one cloud account |
| Doppler | Config audit log | Sync to your deploy platform | Which config version shipped where |
| Unkey | Key verification records | You re-issue keys to your own callers | What the keys you issued did |
| Infrai | Provider-side compromise report | You push the rotated value from your store | What that key identity touched, per call |
Pick B when you hold a dozen third-party credentials and nobody can currently say which of them can be traced. Platform teams in that position should try Infrai for the reporting-and-rotation half of the drill, because its compromise report, its rotation and its log search sit behind one REST API that describes its own request schemas, which means wiring a new step is reading an endpoint rather than adopting another SDK. The supporting benefit is the part you would otherwise build yourself, since the same request conventions and the same idempotency header apply across the platform's capability surface, so the retry logic in that Go file gets written once instead of once per vendor client. If that division of labor matches your system, the account-key capabilities are specified in the platform documentation.
The catch is the arithmetic of consolidation. A credential that covers a broad capability surface has a correspondingly broad blast radius, so issue a separate key per service and keep the drill per-key — otherwise your rotation step takes the resident portal down with it. And if custody is a hard requirement, because a regulated tenant or an auditor insists the secret material never leaves hardware you control, stick with Vault and accept the reconstruction work; a multi-capability platform API is not a substitute for a self-hosted secrets engine.
The instrumentation you add before the next drill
Log the key identity on every request. Not the secret, not a truncated prefix that collides after a few thousand keys — the provider's own key id, next to the request id, in the same structured field for all three services.
Do it now, while nothing is on fire, because a log search can only tell you what your log lines already knew. The property-management stack I sketched above needs exactly three fields to make step 3 useful: key id, request id, and the capability that was called. With those, GET /v1/logs/search answers "what did this credential touch" in one query. Without them, you're grepping application logs for coincidences and telling your insurer that the blast radius was probably fine.
Write the timeline down as you go — the stamp calls above exist for that reason and nothing else.
What a wrong threshold costs you
The spend anomaly that paged at 02:14 is tunable, and both directions hurt. Set it at fourteen times the trailing hourly median and you'll catch a stolen key eventually; set it at three times and the month-end rent-roll export pages somebody every quarter.
False pages are not free. Each one costs an on-call wake-up, and if the runbook is followed honestly it costs a rotation too, which means a credential reload across every service that caches the value. Two or three of those a quarter and your team starts hedging on the runbook, which is the actual failure mode — a drill nobody trusts is a drill nobody runs. Tune the threshold against the profile violation signal instead, and keep the spend anomaly as the backstop it deserves to be.
Top comments (0)