An API key leaked: the first-hour runbook that avoids the self-inflicted outage

A key with broad scopes hit a public repo. Nobody panicked, nobody got fired, and the team was back to normal inside an hour — because they already knew the order of operations. Here it is, so your team knows it too.

Bots scrape public repos and paste sites for sk_, AKIA, and ghp_ strings within minutes. That means two things: revocation is the real fix (history cleanup is hygiene), and the order you rotate in decides whether you get a scare or an outage.

The safe order (the part everyone gets wrong)

The classic self-inflicted outage is revoking first and discovering production was using that key. The safe dance:

1. Create the replacement key first — most providers allow two live keys during a rotation window.
2. Deploy the new key everywhere the old one lives: secrets manager, server config, CI variables, cron jobs, the two dotfiles your best engineer still edits by hand.
3. Verify the new key with a real call, then revoke the old one. Revoke — don't just delete the file.
4. Can't stage it? A 10-minute outage beats a drained cloud account. Revoke, fix, restart.

While that's running: check for real abuse

• Usage dashboard: calls you didn't make — unfamiliar endpoints, regions, user agents, a spike since the leak date.
• If spend occurred: revoke, then contact the provider's abuse desk with your timeline. Providers treat scraped-key spend fairly when you reported fast.
• Cloud keys: new IAM keys, new resources in odd regions, forwarded email rules. Assume persistence until proven otherwise.
• Preserve the audit logs before you delete anything. The logs are your evidence.

The copies that outlive the repo

Deleting the file fixes nothing on its own: git history, CI logs, error-tracker events, Slack messages, tickets, screenshots, and "quick test" scripts all still hold the string. Assume it was scraped, rotate regardless, and purge where you can.

And close the hole, not just the instance: the missing .gitignore, the hardcoded default, the verbose logger. Fix the process in the same week or next week it happens again.

The 30-minute prevention pass

• Secret-scanning in CI (provider-native or a pre-commit hook) — catches the leak at push time.
• .env out of the repo, .env.example committed so onboarding still works.
• Least-privilege scopes + expiry on every new key. An expired key can't leak next year.
• CI variable masking so tokens print as ***.
• A quarterly two-minute audit: grep for sk_, AKIA, ghp_; every live key gets an owner and an expiry.

Takeaway: scope the blast radius, rotate in the safe order, check for real abuse, purge the quiet copies — in that order, in one hour.

---

The full write-up (with the blast-radius table and the mistakes list) is on our notes site:

👉 API Key Leaked: the First-Hour Runbook for Small Teams

If you want the surrounding system — fillable incident plan, severity matrix, comms templates — that's the Ops Starter Kit ($14), and Vol. 2 ($27) adds the full DR plan and evidence log. Launch week: 30% off any paid kit with code HIVE-LAUNCH30.