Uptime monitoring from outside your own infrastructure that alerts you the minute your site goes down — upordown.online
Rotating secrets is good hygiene. Keys leak, people leave, contractors finish, and a password that's been the same for four years is a liability. Every security checklist says "rotate regularly".
What the security checklist doesn't say is that rotation is one of the most reliable ways to take your own site down. You change a password, and a service you forgot about (a cron job, a staging box that turns out to be load-bearing, a Lambda nobody's touched since 2023) can no longer connect. Sometimes immediately. Sometimes at 3am, when its connection pool recycles.
Here's a checklist for making rotation boring. It assumes you have some way of storing secrets (a secrets manager, encrypted environment variables, a vault), but the steps apply regardless of the tool.
Before you rotate
1. Find every consumer of the secret.
This is the step that prevents most rotation outages, and the one most often skipped. Search your infrastructure code, CI configs, deployment manifests and secrets store for references to the secret's name. Then look at the other side: the database or API provider's own logs of which clients, IPs or users are connecting with this credential. Anything in the logs that isn't in your list is a surprise waiting to happen.
2. Check whether the provider supports two valid credentials at once.
The safest rotation pattern is overlap: create the new credential, move every consumer to it, then revoke the old one. Many API providers allow multiple active keys; many databases let you create a second user or set a new password while existing sessions continue. If your provider allows only one credential at a time, you need a short, coordinated cutover instead, and the rest of this checklist matters even more.
3. Know how each consumer loads the secret.
Some services read secrets at startup only. Some re-read on every request. Some cache them in a connection pool that lives for hours. A service that loaded the old password at boot will keep working until it restarts, then fail. Write down, for each consumer, what it takes for it to pick up a new value: a redeploy, a restart, a config reload, or nothing.
4. Make sure you can roll back.
Before revoking anything, know exactly how to restore the old credential if something breaks. With the overlap pattern, rollback is just "don't revoke yet". With a hard cutover, you may need the old value somewhere safe for the duration of the change.
5. Pick a time when someone is watching.
Not Friday at 17:00. Not right before a holiday. Do it when the people who own the consumers are around and when traffic is normal enough that you'll notice errors.
During the rotation
6. Create the new credential with the same permissions, no more and no less.
Rotation is a bad moment to also "tidy up" permissions. If the new key has fewer scopes than the old one, you've combined two changes, and when something fails you won't know which caused it. Change permissions in a separate, later step.
7. Update consumers one at a time, and verify each.
For each consumer: deploy or reload with the new secret, then confirm it's actually using it and working. Confirmation means a real request succeeding, a health check that exercises the dependency, or the provider's logs showing the new credential in use, not just "the deploy went green".
8. Watch the old credential's usage drop to zero.
If the provider shows per-key or per-user usage, this is the best signal you have. When the old credential has had no traffic for a while (longer than your longest cron interval, ideally a full day for anything batch-related), you've probably found every consumer.
9. Revoke the old credential deliberately.
Revoke it, then immediately watch error rates and logs for anything failing to authenticate. If something breaks now, you've found a consumer that wasn't in your list. Restore if necessary, fix, try again.
The rotation isn't done until the live site still works for everyone. upordown.online checks the public URL from outside, so a missed config reload shows up as an alert, not a customer ticket.
After the rotation
10. Monitor the things that run rarely.
The consumers most likely to break are the ones that don't run often: weekly reports, monthly billing, quarterly exports, disaster recovery scripts. Make sure each has a heartbeat or a failure alert, so a broken credential on day 30 is noticed on day 30, not on day 90.
11. Write down what you found.
Update your list of consumers for this secret. Note which ones needed a restart, which picked it up automatically, and any surprises. Next rotation starts from this list instead of from scratch.
12. Schedule the next one.
Rotation that happens once every few years is always painful, because nobody remembers how. Rotation that happens every few months becomes routine, and routine is what you want. Some teams automate it entirely; even a calendar reminder is better than nothing.
The pattern underneath
Almost every rotation outage comes from the same root cause: a consumer nobody knew about. The checklist is mostly a set of ways to find those consumers before revoking, rather than after.
Two habits make that easier year-round:
- One secret per consumer where practical. If each service has its own key, rotating one can't break the others, and usage logs tell you exactly who's using what.
- Monitoring that exercises dependencies. A health check that only returns "process is alive" won't notice the database credential stopped working. A check that runs a trivial query will.
A quick check you can run today
Pick your most important secret, maybe the production database password, and answer three questions:
- Can you list every service that uses it?
- If you changed it right now, which of those would fail first, and which would fail only after a restart?
- Would your monitoring tell you within minutes?
If the answers are "not really", "no idea" and "probably not", you've found the work to do before the next rotation. External monitoring that notices when your site or API starts failing, whatever the cause, is what we do at upordown.online.
Related reading
- Two developers, one outage, no postmortem? That's how the same outage happens again
- No deploys on Friday: wise rule or a sign your monitoring is broken?
- Your uptime alerts have never been tested. A 12-point checklist to prove they'll actually wake you
- Your health check returns 200 while users get errors: 7 health check mistakes
- Adding rate limiting to your API? A 12-point checklist so it doesn't block your own users (or your monitor)
What's the most surprising thing that broke the last time you rotated a credential?
Top comments (3)
Step 8 says to watch the old key's usage drop to zero, and the AI providers do show usage per key. OpenAI's Usage API groups by api_key_id in 1-minute buckets. Anthropic's usage_report filters by api_key_ids[], though Console playground use has no key id. OpenRouter's key list shows last_used_at.
That's really handy, thanks. Per-key usage from the provider is the cleanest way to know the old key is truly idle, much better than guessing from your own logs. The playground gap is a good warning too, since someone testing in the Console wouldn't show up against any key at all.
Thanks, glad it's useful!