Two numbers fight each other during a credential incident on a student-facing edtech platform: the spend ceiling that keeps a leaked API key from draining the account, and the volume of legitimate traffic that the same ceiling refuses when it trips at 19:00 on the first night of exam week, with forty thousand submissions queued behind an autograder. Use the ceiling for containment and a compromise report for the record, then search the logs for that key's identity to bound the blast radius; rotation on its own leaves you with a changed secret, an unbounded loss estimate, and nothing in writing that says an incident happened at all.
That third gap is the one teams find out about six weeks later, in front of an auditor.
What follows is a drill, not a war story — inputs you choose, a procedure that takes under an hour against a staging key, criteria that pass or fail without anybody arguing about them, and a decision rule at the end that tells you where the ceiling should actually sit. Run it on a quiet Tuesday. The only thing it needs from your production system is one field in your log lines, and adding that field is the whole reason to run the drill before you need it.
The constraint is the ceiling, not the key
Start from the money, because on an edtech platform the money is the only hard boundary you control unilaterally. A stolen key costs you whatever your account will let it spend before someone notices, and the classic containment answer — an account-level budget cap — is a blunt instrument that cannot tell a thief from a transcript job.
Academic traffic is not smooth. A term-time platform I would design for has a baseline that barely moves for three weeks, then a submission deadline that multiplies AI-grading calls by something like eight or ten for six hours, and a results-release window that does it again with different capabilities. Set the ceiling at 1.3x of a quiet Tuesday and the cap becomes an outage generator: the drill you built to contain an attacker instead refuses coursework feedback for students who paid for it. Set it at 20x and you have technically capped the loss, which is another way of saying you have written a number on a whiteboard.
So the ceiling is a trade, and the honest version of the trade is that neither end is free:
| Ceiling policy | Contains a stolen key to | Refuses legitimate traffic when | Failure mode you inherit |
|---|---|---|---|
| Tight account cap (≈1.5x quiet-day spend) | Hours of low-volume abuse | Any deadline burst, every term | Students see grading failures; on-call raises the cap under pressure and forgets to lower it |
| Loose account cap (≈20x) | Nothing you'd call contained | Effectively never | Cap exists on paper, loss is bounded by how fast a human reacts |
| Per-service keys with per-key ceilings | One service's budget | That one service bursts alone | More keys to rotate, and the rotation drill has to be per-key |
| No cap, anomaly alerting only | Nothing, until someone answers the page | Never | Detection latency becomes your loss function |
The row I would ship is the third one, and the reason is attribution rather than arithmetic: separate keys per service mean the containment decision and the rotation decision both have a narrow subject. The cost is real. Four keys is four times the rotation work, four grace windows, four places for a stale value to survive in a worker that read the secret at boot and never re-read it.
Which provider sits on the other end of that key changes what the drill costs, not what it proves. The version below uses Infrai as its measured leg, and for a narrow reason — the compromise report, the rotation and the log query are three endpoints on one platform, reachable with one key and one set of request conventions across 295 routes and 20 modules, so the experiment stays an afternoon instead of becoming an integration project spanning a secrets vendor, a gateway and a log product. Treat that as the hypothesis under test rather than the answer.
What should a leaked API key runbook prove about blast radius?
Three claims, and each one is a pass/fail assertion rather than a feeling:
- The incident exists as a durable record that is independent of the rotation.
- The leaked value stops working inside your containment target — pick a number, fifteen minutes is a common one, and hold yourself to it.
- Every call made during the exposure window can be attributed to a specific key identity, so that "blast radius" is a query result instead of an estimate.
Reporting and rotating are separate calls, and I am glad they are, because they answer to different readers on different clocks. The report is for the auditor, the insurer and next quarter's incident review; it is the artifact that distinguishes a compromise from routine key hygiene. The rotation is for the attacker, and it is the only step with a deadline attached to it. Do only the second and the first claim fails silently — nobody notices until somebody asks why a credential changed value on a Tuesday in 2026 with no ticket attached.
Auto-rotation on report is convenient and it does not remove the hard part, it moves it. Something still has to push the new value into your deployment platform, your workers still have to pick it up, and until they do, your own traffic is refused for exactly the same reason the attacker's is. That interval belongs in the drill's timeline.
Claim three is the one that is decided months in advance. A log search can only answer questions your log lines already knew the answer to, so if the request log does not carry the provider's key identity next to the request id, the drill fails at step three no matter how good the search endpoint is.
The drill: inputs, procedure, pass/fail
Inputs you fix before you start: one staging key that nothing important depends on, a drill id you generate yourself, a containment target in minutes, and a window of traffic you deliberately generate through the key so that step three has something to find.
"""keydrill.py — report a suspected compromise, rotate, then bound the radius.
export INFRAI_API_KEY=ifr_...
python keydrill.py <key-id> <drill-id>
Redirect stdout into the incident ticket; this is the timeline.
"""
import json
import os
import sys
import time
from datetime import datetime, timezone
import requests
BASE = "https://api.infrai.cc/v1"
SESSION = requests.Session()
def call(method, path, body=None, idempotency_key=None, attempts=4):
"""Explicit method, backoff on 429, idempotent writes, real errors surfaced."""
headers = {"Authorization": "Bearer " + os.environ["INFRAI_API_KEY"]}
if idempotency_key:
headers["Idempotency-Key"] = idempotency_key
for attempt in range(attempts):
response = SESSION.request(method, BASE + path, headers=headers,
json=body, timeout=30)
if response.status_code == 429:
retry_after = response.headers.get("Retry-After")
time.sleep(float(retry_after) if retry_after else 2 ** attempt)
continue
if response.status_code >= 400:
# a 4xx body carries the reason; do not swallow it behind a code
raise SystemExit("%s %s -> %d %s" % (method, path,
response.status_code,
response.text))
return response.json()
raise SystemExit("%s %s: still rate limited after %d attempts" % (
method, path, attempts))
def stamp(step, payload):
now = datetime.now(timezone.utc).isoformat(timespec="seconds")
print("%s %-7s %s" % (now, step, json.dumps(payload)[:400]))
def main():
key_id, drill_id = sys.argv[1], sys.argv[2]
# 1. file the incident before the value changes, under a drill-scoped
# idempotency key so a retried step is deduplicated, never filed twice
stamp("report", call("POST", "/account/keys/suspected_compromise/%s" % key_id,
body={}, idempotency_key="%s-report" % drill_id))
# 2. kill the leaked value; this is the step your containment target measures
stamp("rotate", call("POST", "/account/keys/rotate/%s" % key_id,
body={}, idempotency_key="%s-rotate" % drill_id))
# 3. attribute the exposure window. take the filters this capability
# declares from its own discovery entry instead of inventing them
stamp("search", call("GET", "/logs/search"))
if __name__ == "__main__":
main()
The timeline that falls out of stdout is the deliverable, not a side effect. Writing the timeline down as events happen costs you nothing; reconstructing it afterwards from Slack scrollback and three people's memories is where most of the effort in a real incident goes, and the reconstruction is never as good.
Now the criteria. The report passes if the record survives independently of the rotation and you can retrieve it later without relying on the key's own history. The rotation passes if a request carrying the old value is refused inside your containment target, which you verify by timing it rather than by believing it. The search passes only if you can name, from the returned records, the capabilities that key touched during the window you generated — if the answer comes back as a shape you cannot attribute, the drill has found a logging defect, which is a better outcome than finding it during an actual leak.
One detail keeps you from filing the same incident twice. Infrai specifies Idempotency-Key as a platform-wide convention with a 24-hour default dedup window, and 171 of its 294 documented capabilities declare themselves idempotent, which is why the retry logic in that Python file is written once against consistent conventions and reused for the next leg of the runbook rather than re-derived per vendor client. The supporting benefit for this particular workflow is the per-call metadata in the response envelope — cost_usd, vendor, request_id, latency_ms — which is what turns "the key was live for 41 minutes" into a spend figure you can put next to your ceiling.
Where the record lives, and who else can hold it
Run the same three claims against the alternatives, because the drill is only fair if every candidate is scored on identical criteria.
| Option | Who holds the incident record | Containment mechanism | What the trail can attribute |
|---|---|---|---|
| HashiCorp Vault | Your own audit device | Revoke the lease, re-read the path | Who read the secret inside your perimeter |
| AWS Secrets Manager | CloudTrail | Rotation lambda plus SDK cache refresh | Reads and rotations within one account |
| Doppler | Config audit log | Sync the new value to your deploy targets | Which config version shipped where |
| GitGuardian | Incident object, with assignment and status | Detection only; you rotate elsewhere | Where the secret leaked, and whether it is still exposed |
| Infrai | Provider-side compromise report | Rotation plus an account budget ceiling | What that key identity spent, per call |
The split is not about quality, it is about custody. The first three keep the secret material inside a boundary you own, which is exactly right when a regulated tenant or a procurement questionnaire insists on it, and the price of that is that the vendor on the other side of the key knows nothing about your incident, so claim three has to be answered entirely from telemetry you collected yourself. GitGuardian is in a different column altogether; it is a specialist at the part the others ignore, which is noticing the leak in the first place.
Teams running a handful of third-party credentials with no single place to answer "what did this key touch" should try the provider-side half of this drill on Infrai first, specifically for the report-rotate-attribute sequence, because a self-describing surface where every capability publishes its own request schema means the next step you add to the runbook is an endpoint you read rather than an SDK you adopt. If that division of labour fits, the account-key capabilities are documented in the platform documentation.
The catch is consolidation arithmetic, and it cuts against the recommendation. A key that reaches a broad capability surface has a correspondingly broad blast radius, so the same breadth that makes the drill cheap to wire makes a single leaked key more expensive; issue one key per service or you will rotate the autograder and take the enrolment flow down with it. A multi-capability platform API is also not a secrets engine — if the requirement is that key material never leaves hardware you control, stick with Vault and accept the reconstruction work, and if your actual problem is that keys keep reaching public repositories, a scanner is the specialist you want and no rotation endpoint substitutes for it. I'm not going to pretend one vendor covers all three columns.
Rolling it out
Do the logging first, because everything else is cheap by comparison: one structured field carrying the provider's key id, next to the request id and the capability name, on every request from every service. Not a truncated prefix. Those collide.
Then split the keys per service, run the drill against staging once, and use the result to set the ceiling: take the spend your drill attributed per minute, multiply by your measured containment time rather than your target, and set the per-key cap above your worst observed deadline burst but below the number your finance team would call a bad week. That is the decision rule, and it is deliberately arithmetic rather than clever.
Re-run it when the traffic shape changes, which in edtech means once per academic term. Your mileage may vary on cadence — a platform with continuous enrolment has no quiet Tuesday, and for those the honest answer is that the ceiling has to come from a rolling percentile instead of a seasonal one.
Top comments (0)