Rotating Secrets Without Breaking Production: Cloud KMS + Secret Manager at Solo-Builder Scale

I once rotated a database password by hand at 11 p.m. and spent the next hour watching a service fail to start because one Cloud Run revision still held the old value in an environment variable. Nothing about the rotation itself was hard. The hard part was that I had no system: no versioning, no rollout order, no way to answer the simple question of which revision was reading which secret.

That night is why I now treat secret rotation as a deployment problem, not a clipboard problem. This article walks through the setup I use today on Google Cloud: Secret Manager for the secrets themselves, Cloud KMS for the encryption keys underneath them, and a rotation pattern that keeps old and new versions alive at the same time so a deploy can never strand a running revision. Everything here is sized for a small team or a solo builder. No enterprise ceremony, but also no shortcuts that fall apart the first time you rotate under pressure.

The mental model: three layers, three different jobs

It helps to separate three things that people often blur together:

1. The secret — a database password, an API key, a signing key. This lives in Secret Manager.
2. The encryption key — the Cloud KMS key that encrypts the secret at rest. Secret Manager uses Google-managed encryption by default, and you can bring your own customer-managed encryption key (CMEK) if you want control over rotation and destruction of the key itself.
3. The access path — how your workload reads the secret at runtime: as a mounted volume, as an environment variable injected by Cloud Run, or fetched in code through the Secret Manager API.

Rotation can happen at any of these layers, and they are independent decisions. Rotating a secret value is not the same as rotating its encryption key, and neither one automatically rotates the credentials held by downstream systems. When I plan a rotation, I write down which layer I am rotating and what breaks if I get the order wrong. That one habit has prevented more incidents than any tool.

Setting up the pieces

Start with a KMS key ring and a key with a rotation schedule. The rotation schedule matters because it is the one part of this system that works while you forget about it:

gcloud kms keyrings create app-keyring --location=us-central1

gcloud kms keys create app-secrets-key \
  --keyring=app-keyring \
  --location=us-central1 \
  --purpose=encryption \
  --rotation-period=90d \
  --next-rotation-time="2027-02-01T00:00:00Z"

A 90-day rotation period is a reasonable default. Shorter periods sound more secure, but every rotation is a chance for something to go wrong, and a schedule you cannot operate reliably is worse than a longer schedule you actually follow. The trade-off is honest: KMS key rotation re-encrypts data with the new key version going forward; old key versions stay available to decrypt older ciphertext, so rotation is not an emergency procedure. That is a feature, not a weakness. It means a bad rotation does not brick your data.

Next, a secret that uses that key for encryption:

gcloud secrets create db-password \
  --replication-policy="automatic" \
  --kms-key="projects/PROJECT_ID/locations/us-central1/keyRings/app-keyring/cryptoKeys/app-secrets-key"

One decision to make deliberately is the replication policy. Automatic replication is simpler and fine for most small workloads. If you have a data-residency requirement, user-managed replication with explicit locations is the right call, but it adds operational surface. Do not pick the complex option by default.

The pattern that actually works: dual-version rotation

Here is the core pattern, and it is worth stating plainly because the naive version — change the secret, redeploy everything at once — is how you get a midnight outage:

1. Add the new version; do not disable the old one. Secret Manager versions are cheap and explicit. Create the new value as a new version while the old version stays enabled.
2. Make the consumer accept both. For a database password, that means creating the new database user or updating the credential so both the old and new passwords authenticate during the transition window. For an API key, it means the receiving system has both keys registered. This is the step people skip, and it is the step that makes rotation safe.
3. Roll the workload forward. Deploy a new Cloud Run revision that reads the new version. Cloud Run can mount a secret as a volume or inject it as an environment variable pinned to a specific version:

# service.yaml (excerpt)
spec:
  template:
    spec:
      containers:
        - image: gcr.io/PROJECT_ID/app:latest
          env:
            - name: DB_PASSWORD
              valueFrom:
                secretKeyRef:
                  name: db-password
                  # Pin to "latest" only if your deploy pipeline
                  # controls exactly when a revision goes live.
                  # Pinning a version number is safer for rollbacks.
                  version: "2"

Pinning a version number versus using latest is a real trade-off. latest means a new secret version flows into the next deploy automatically, which is convenient and also means a bad secret version ships the moment you deploy for an unrelated reason. A pinned version means rotation is a deliberate deploy, and rollback is just redeploying the previous revision. I pin versions. The extra deploy is cheap; the ambiguity is not.

1. Verify the new revision, then disable the old version. Wait until traffic has fully shifted and the old revision is gone — Cloud Run keeps old revisions around, so check that no revision still references the old version before you disable it. Disabling is reversible; destroying is not. I disable first, wait a full deploy cycle, and only destroy much later, if ever.

The whole sequence can be scripted, and it should be, because a manual rotation done twice a year is a manual rotation done wrong twice a year.

Automating the schedule

Rotation that depends on someone remembering is rotation that does not happen. A small Cloud Run Job triggered by Cloud Scheduler can handle the mechanical parts: generate the new value, add it as a new Secret Manager version, and open the follow-up work (the dual-acceptance step and the deploy) as a tracked task rather than trying to automate the parts that need judgment.

# rotate_secret.py (excerpt) — runs as a Cloud Run Job
from google.cloud import secretmanager

client = secretmanager.SecretManagerServiceClient()
parent = f"projects/{PROJECT_ID}/secrets/db-password"

# Add the new version. The old version stays enabled until
# a human confirms the new revision is serving traffic.
response = client.add_secret_version(
    request={"parent": parent, "payload": {"data": new_password_bytes}}
)
print(f"Added version: {response.name}")
# Deliberately NOT disabling the old version here.
# That step belongs to the deploy checklist, after verification.

Notice what the script does not do: it does not disable the old version, and it does not deploy. Those are the two steps where a human verifying traffic is worth more than automation. Automate the boring generation; keep a human on the irreversible step.

Grant the job the minimum IAM it needs and nothing more:

gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:rotator@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretVersionAdder"

# The workload itself only needs accessor, and only on the
# secrets it actually reads — never secretmanager.admin.
gcloud secrets add-iam-policy-binding db-password \
  --member="serviceAccount:app@PROJECT_ID.iam.gserviceaccount.com" \
  --role="roles/secretmanager.secretAccessor"

This is where Workload Identity Federation earns its keep if you rotate from CI: the CI job gets a short-lived token scoped to exactly this secret, instead of a long-lived service-account key that itself becomes a rotation problem. Every long-lived credential you create is a future rotation ticket. Count them before you create them.

What breaks, honestly

A few limitations worth naming before you adopt this:

• Dual acceptance is not always possible. Some third-party APIs allow only one active key at a time. In that case the transition window is a real outage risk, and the honest answer is to schedule the rotation, accept a brief window, and have a rollback ready — not to pretend the pattern covers it.
• Secret Manager versions are not free of cost at large scale, and neither is KMS. For a solo builder the bill is small, but if you rotate hundreds of secrets weekly, count versions and key operations before you automate aggressively.
• Rotation does not revoke leaked values. If a secret was exposed, rotating it is necessary but not sufficient: you also need to check logs for use of the old value, revoke sessions or tokens derived from it, and treat the exposure as an incident, not a housekeeping task.
• Destroying a secret version is irreversible. Disabled versions can be re-enabled. Destroyed ones cannot. I have never once regretted waiting an extra month before destroying.

The checklist I actually use

1. Which layer am I rotating — value, key, or both?
2. Can the consumer accept both values during the transition? If not, what is the planned window?
3. Is the new version pinned in the deploy, not floating on latest?
4. Has traffic fully shifted off every revision using the old version?
5. Old version disabled — not destroyed — and the disable date recorded.

That is the whole system. It is not clever, and that is the point. Secret rotation should be boring. The night I rotated by hand taught me that the goal is not a faster rotation; it is a rotation so uneventful that nobody remembers it happened.

Takeaways

• Separate the secret, the encryption key, and the access path; rotate each deliberately.
• Dual-version rotation — new version live, old version still accepted, deploy, verify, then disable — is what makes rotation safe.
• Pin secret versions in Cloud Run deploys; latest trades safety for convenience.
• Automate generation, keep a human on the irreversible step.
• Disable before you destroy, and never confuse rotation with incident response for a leaked secret.