Creating a new database secret is the beginning of a rotation—not the end.
The old credential may still be held by an MCP agent, connection pool, deployment revision, test client, or forgotten worker. A successful login with the new secret proves only that the new path works.
A safer runbook is:
- inventory every consumer and active session
- create a new version or replacement database role
- preserve or narrow the reviewed grants
- open a short, explicitly bounded overlap window
- canary the new credential through the real network path
- drain pools and stop old revisions from creating connections
- test positive and negative authorization cases
- revoke the old credential
- attempt a fresh login with the old secret and require failure
Track time to distribute, connect, drain, revoke, and prove revocation—not merely the secret's “updated at” timestamp.
And keep rollback narrow. Leaving both credentials valid indefinitely is not rollback; it is a second production access path.
Full runbook: MCP database credential rotation
Top comments (2)
Step 9 is the one most rotation docs skip, and it is the only step that proves anything. You cannot confirm a state change by asking the system that performed it. You confirm it by running the thing that has to fail now.
One case worth adding, from running an MCP server in production: with agent-held credentials the dangerous state is rarely "the old secret still works". It is "the credential still authenticates, and still carries the authority it was minted with, while the policy behind it moved." That state survives step 9 intact, because the login does not fail. Nothing about the connection is stale. The grant is. So the negative cases in step 7 have to be re-run after revocation, asserting on the narrowed grant rather than on the connection.
Step 6 has a similar asymmetry. Draining a pool is a server-side action against a known set of consumers. An agent session is held by a client you do not control and cannot enumerate, and it reconnects on its own. Revocation there has to invalidate the session, not just the credential.
Which leads to a real question, since the runbook stops at the database boundary: in a relay architecture, where does the step 9 proof live for the agent-facing credential? Rotating the role behind the relay does not touch what the client holds. Two credentials, two revocation proofs, and only one of them has a login you can actually go try.
Good call on treating the 'updated at' timestamp as meaningless. The real insight is that rotation is a distributed systems problem, not a key management problem. You can inventory your own pools and deployments, but the MCP agents and services you haven't thought about yet are exactly where old credentials leak and linger.
The structural fix is to stop handing out long-lived credentials to every consumer. If agents pull credentials from a single vault at call time instead of caching them, rotation becomes a one-line change at the source. The old value dies immediately, without a canary window or drain ceremony, and you don't have to hunt down forgotten workers.
Nice writeup. The overlap window and negative authorization test are the two steps most teams skip.