Rotate a leaked API key without stopping your services: dotvault hands out the new value first

This is part 11 of the series "dotvault, Start to Finish: One Command from Your Laptop to CI, Your Team, and Your AI Agents." dotvault will be released free of charge when this series ends.

In the demo, a script kept calling with the key at 15-second intervals during the whole rotation, and in all 111 calls that got a response, the caller's value was accepted. The caller moved to the new value on its own, and the leaked value was rejected from then on.

That matters because rotating a key between two of your services has an ordering problem. Switch the service that checks the key first, and the caller still holding the old value gets rejected. Switch the caller first, and the checker rejects a value it has never seen. Either way, calls fail until the other side catches up, and the usual way around it is to stop both and switch them together. This part shows how dotvault rotate --reason leak avoids that.

This part also covers a second feature: rotating a key automatically at an interval you choose.

Rotation is not only for leaks. It is part of routine key management. Set an interval when you register a key, and dotvault keeps rotating it on its own, even when nothing has leaked. Nobody has to rotate keys by hand any more, so you can keep each key's lifetime short.

This is part 11 of 12.

The short version

• Register the key for rotation management first. Registering it names the holder (the service token that sends the key) and the verifier (the service token that checks it).
• When it leaks, run dotvault rotate <KEY> --reason leak. dotvault gives the new value to the verifier first, then to the holder, and revokes the leaked value once both have read the new one, so you rotate without stopping your services. In the demo, all 111 calls that got a response were accepted.
• Routine rotation can run on its own, too. Set an interval (interval_seconds) when you register the key, and as long as the holders and verifiers read from dotvault as part of their normal work, dotvault keeps rotating it at that interval, so you can keep the key's lifetime short.
• Too long to wait: run --revoke-now once every holder has the new value. Retire, which stops the key at once, is the last resort.

Before you read on

This follows part 4 (overwriting and rolling back values) and part 7 (creating service tokens). You can follow along without them.

1. Register the key for rotation management

The demo: order-worker processes orders and calls payment-api, the payment API. It sends PAYMENT_API_KEY, and payment-api checks that the value is right. Assume PAYMENT_API_KEY has leaked.

The terms used here:

Term	Meaning
Rotation management	dotvault generates the key's values and decides who receives them. Only registered keys are covered
Holder	The side that holds the value and calls the other service. In the demo, the order-worker token
Verifier	The side that checks the value it receives. In the demo, the payment-api token
Rotate	Hand out a new value, then revoke the old one
Retire	Revoke the value at once, without handing out a new one

Rotation works for keys that both sides receive from dotvault, which means keys between your own services. dotvault generates the new value. An API key issued by another company's service has to be replaced in that service's dashboard.

In the demo project shop-backend, the first value of PAYMENT_API_KEY went into production with dotvault set. order-worker and payment-api each got a Read only service token. A service token is the string, starting with est_, that CI or a service uses instead of logging in. Part 7 shows how to create one.

Registering takes a token of the Credential rotation type. In Service Tokens, click Create token and pick "Credential rotation" as the Role.

As the dialog says, this token can issue tokens and register keys only inside one project and the environments you pick. It is not needed once the key is registered, so it expires in 1 day.

With that token, send one HTTP PUT, the request type that stores data at a URL. The body is JSON naming the holder and verifier tokens by Token ID.

HTTP 201 (created) and "created":true mean the key is registered. The fields:

Field	What it holds
bindings	Who receives the key. tokenId is the service token's Token ID, role is holder or verifier, and label is a name you choose
residual_seconds	How long to wait, after every holder has moved to the new value, before revoking the old one. 60 seconds in the demo
generator (in the response)	How new values are made. random-base64url-32 is 32 random bytes written as letters, digits and symbols

key= in the URL is the key name, and project=, env= and file= say where it lives. $DOTVAULT_TOKEN in the Authorization header holds the Credential rotation token.

A rotation cannot start in the first 5 minutes after registering. Register service-to-service keys when you store them, not after a leak.

2. Only the holder's pull gets the value

After registering, pull the same project as a person, with the holder token and with the verifier token.

Only the holder, order-worker, gets the value. This screen was captured after the rotation in section 4; a registered key is handed out the same way before and after a rotation. The person (an org admin) and the verifier see the key name in a "withheld 1 rotation-managed key(s)" warning, with no value. If you need a key with that name for local development, store a development value under a different key name.

The verifier does not pull. It reads the set of values it should accept from GET /v1/env/key-set. Each entry has the value, an ID for the value (keyId) and a state (state). The verifier accepts a value that is in this set.

3. Start the rotation with --reason leak

To see whether calls keep working during the rotation, a small script, service.mjs, ran at 15-second intervals. Each round, it:

1. Pulls PAYMENT_API_KEY with the order-worker token
2. Reads the accepted set with the payment-api token, and prints accepted if the value is in it, REJECTED if not
3. Checks the leaked value (the value before the rotation) against the same set
4. Reads how far the rotation has got (phase)

It prints only the first 8 characters of each keyId, never the value. With the script running, an org admin signed in to the CLI runs:

dotvault rotate PAYMENT_API_KEY -E production --reason leak

started rotation means it is running. generation 2 means this is the key's second rotation; the first was the routine rotation in section 7, which was aborted.

A leak rotation cannot be stopped. --abort returns ABORT_REFUSED_LEAK with exit code 7, so there is no path back to the leaked value.

--status shows how far the rotation has got. In the audit log, the rotation reached STAGED in the same second it started.

4. The verifier gets the new value first, then the holder

The rotation goes in this order.

phase	What happens
STAGED	dotvault generates a new value and adds it to the verifier's set. The holder still gets the old value
PROMOTED	About 15 minutes after the verifier is seen reading the new value. The holder's pull starts returning the new value. The leaked value stays in the set
HOLDERS_OK	About 15 minutes after the holder is seen reading the new value
DONE	residual_seconds (60 seconds) after HOLDERS_OK, the leaked value is revoked and leaves the set

The table leaves out LOCKED, the phase at the start (the audit log shows STAGED in the same second as the start), and VERIFIERS_OK, the mark that the verifier has read, which is recorded in the same second as PROMOTED.

The 15-minute waits exist because holders and verifiers may keep a value or a set they read for up to 5 minutes. Several instances can share one token and they do not all read at the same moment, so 10 more minutes are added. --reason leak does not skip these waits on its own. In a leak rotation, if one side never reads, the rotation moves on anyway after 30 minutes (plus the 5 minutes a read may be kept).

Here is the script's output around each change of phase. The first two ABORTED lines are the state left by the routine rotation aborted in section 7.

Three things show up:

• The verifier learns the new value first. At STAGED, the new value (k_iVota-, state staged) joins the set, while the holder keeps using the old value (k_Z0Njns) and is still accepted.
• The holder moves over without noticing. From the first pull after PROMOTED, the holder's value is k_iVota-. The verifier already has it, so it is accepted. The leaked value stays in the set as retiring.
• The leaked value leaves last. After DONE, the set holds only the new value, and the leaked value is REJECTED.

Over the whole run, the script made 124 rounds. In all 111 that got a response, the holder's value was accepted. The other 13, error fetch failed, were rounds where the script's network request failed, so no response from dotvault came back.

The console shows the rotation too. In the project's Keys, select PAYMENT_API_KEY and a Rotation section appears on the right.

Besides Phase, the Rotation panel shows when the rotation is next due to move on (Next due) and, in a leak rotation, when it moves on even if a side has not read the new value (Deadline). The figure leaves those two lines out.

5. After the rotation, the leaked value cannot be written back

After DONE, check the state and try writing the leaked value back.

In dotvault versions, the new v3 is state=current and the leaked v1 is state=revoked. v3 and v2 were made by system:rotation-driver, which is dotvault itself. v2 belongs to the routine rotation aborted in section 7.

Writing the leaked value with dotvault set, or rolling back to v1 with dotvault rollback --to 1, both fail with ROTATION_MANAGED: a key under rotation management changes only through rotation. As part 4 showed, an ordinary key can be rolled back to an old value after an overwrite. A key under rotation management cannot go back to the leaked value, even if someone tries with good intentions.

6. The rotation's log is in the audit log

In the console's Audit Log, filter with shop-backend:production:env:PAYMENT_API_KEY as the Target ID: project, environment, file and key name joined by :.

The leak rotation's rows, from the bottom up. 14582408-… is the user ID of dev-e, the demo user.

Action	Actor	Meaning
secret.rotation.started	14582408-…, cli	dev-e started the rotation from the CLI
secret.rotation.staged	system:rotation-driver	A new value was generated and added to the verifier's set
secret.rotation.verifiers_ok	system:rotation-driver	The 15-minute wait after the verifier read the new value is over
secret.rotation.promoted	system:rotation-driver	The holder started receiving the new value
secret.rotation.holders_ok	system:rotation-driver	The 15-minute wait after the holder read the new value is over
secret.rotation.revoked	system:rotation-driver	The leaked value was revoked
secret.rotation.completed	system:rotation-driver	The rotation finished

A person started the rotation and dotvault did the rest. That the leaked value was revoked is one row you can point to later.

7. In a routine rotation, the old value stays usable for 24 hours

When nothing has leaked, a routine rotation runs without --reason (the same as --reason manual).

Two differences from a leak rotation:

• The old and new values both stay valid for at least 24 hours. For 24 hours after the holder starts getting the new value, the old value is still accepted. Every service has time to move over.
• It can be stopped. Before the holder gets the new value, --abort stopped it in a few seconds: the new value became revoked and the old value stayed current.

waiting on verifier payment-api means payment-api has not read the new value yet. last ok is when it last read the set. The phase deadline of a routine rotation (24 hours after the start) is how long it waits for a side that has not read the new value. Past it, the rotation does not move on; it stops (STALLED).

8. Set an interval and let dotvault rotate on its own

Even when nothing has leaked, you may not want to keep one key for a long time. Add interval_seconds (the rotation interval, in seconds) when you register the key, and dotvault starts a routine rotation on its own at that interval. Rotation stops being a person's job.

In the demo, a new key, WEBHOOK_SECRET, was registered with the same holder and verifier tokens as in section 1, with interval_seconds set to 600 (10 minutes). Ten minutes is only for the recording; in real use you would pick something like 30 days (2592000).

Right after registering, --status says managed but has never run: the key is registered but has never been rotated. Nobody typed a rotate command after that (the only thing running was the script that reads values). Once the 10-minute interval had passed, --status showed reason scheduled (dotvault started this rotation itself because the interval came round), and the rotation had already added the new value to the verifier's accepted set (STAGED).

The audit log agrees. After the registration (rotation_provisioner.config_created), both secret.rotation.started and secret.rotation.staged have the Actor system:rotation-driver. In the leak rotation in section 6, a person was the Actor of started; here dotvault started the rotation too.

A scheduled rotation is the same as the routine rotation in section 7: the old and new values both stay valid for at least 24 hours, and the new value goes to the verifier first, then to the holder. The next rotation starts once the interval has passed after the previous one finished. Right after you register a key, or shorten its interval, nothing starts for 5 minutes.

A rotation only reaches the end when the holders and verifiers read from dotvault as part of their normal work. If a service never reads again, the rotation waits on it with the waiting on … line from section 7, and past the deadline it stops (STALLED).

9. Too long to wait: --revoke-now. Last resort: retire

A rotation accepts the leaked value until the new one is everywhere. Even after the holder has read the new value, there is a 15-minute wait and the 60-second residual. To make the leaked value stop working sooner, skip the remaining waits with --revoke-now.

dotvault rotate REFUND_API_KEY --revoke-now --confirm REFUND_API_KEY -E production

This was recorded after PAYMENT_API_KEY had been retired (later in this section), on a second key, REFUND_API_KEY, registered the same way. One of the script's calls already carried the new value (PROMOTED). Then --revoke-now: in the audit log, revoke_now, revoked and completed share the same second, and from the next call that got a response, the leaked value was REJECTED. The holder's value stayed accepted. The script ran 74 times during this recording, and in all 57 calls that got a response, the holder's value was accepted (the other 17 failed on the script's side before reaching dotvault).

Run --revoke-now once every holder has the new value, that is, after PROMOTED. It skips every wait from that point on, so if you run it earlier, dotvault switches values and revokes the old one without waiting for the verifier or the holder to read the new one, and a holder still on the old value is rejected.

The waiting on holder … line in --status is per token. When several services share one token, the line disappears as soon as one of them reads. Any other service still holding the old value is rejected after --revoke-now; that is what 10 of the 15 minutes of waiting are for. Run it only when you know every service has pulled again, for example after restarting them. In the demo the holder's value stayed accepted because there was one holder and it pulled on every call.

One more thing: revoking takes the value out of dotvault's accepted set. While a verifier keeps its copy of the set (up to 5 minutes), it may still accept the leaked value. The demo's verifier read the set on every call, so the leaked value was REJECTED from the next call.

--confirm repeats the key name; one wrong character stops it with VALIDATION_ERROR and nothing is sent. Only a signed-in person can run it; a service token cannot.

If you cannot even wait for the holder to move to the new value, retire the key.

dotvault rotate PAYMENT_API_KEY --retire --confirm PAYMENT_API_KEY -E production

The retire ran on the same key after the PAYMENT_API_KEY rotation was recorded. revoked version(s): 3 means it revoked v3, the current value from the rotation; v1 and v2 were already revoked. The verifier can no longer read the set (HTTP 403, NOT_BOUND), and writing the leaked value back with set fails with VALUE_REVOKED: a revoked value can never be written again.

Retire hands out no new value, so the holder and the verifier both stop at once. --confirm repeats the key name, and one wrong character means nothing happens. Only a signed-in person can retire; a service token cannot. Use it only when the leaked value has to stop working right now, even at the cost of stopping your services.

10. Revoke, overwrite, rotate or retire

What to use depends on what leaked.

What leaked	What to use	What happens	Covered in
A service token (est_…)	Revoke in Service Tokens	The token stops working with dotvault. The values do not change	Parts 7, 8 and 10
The value of a key that is not registered	Overwrite it with dotvault set	The new value goes out. The old one stays in the history and rollback can restore it	Part 4
The value of a registered key	dotvault rotate --reason leak	The new value goes out first, then the leaked value is revoked. Services keep running as long as the holders read the new value	This part
The same, once every holder has the new value and you want to skip the remaining waits	dotvault rotate --revoke-now	The leaked value leaves dotvault's accepted set within seconds. Any service still holding the old value is rejected	This part
The same, when it must stop right now	dotvault rotate --retire	Revoked at once. The holder and the verifier both stop	This part

For a routine rotation when nothing has leaked, use dotvault rotate (--reason manual); the old value stays usable for at least 24 hours. Add interval_seconds to the registration, and dotvault starts that routine rotation on its own at the interval you set.

Wrapping up

Rotating a key between two services fails calls whichever side you switch first, until the other side catches up. Register service-to-service keys for rotation management when you store them, and when one leaks, run dotvault rotate <KEY> --reason leak. dotvault gives the new value to the verifier, then to the holder, and revokes the leaked value only after both have read the new one, so the leaked value stops working without stopping your services. Set an interval when you register the key, and routine rotation before any leak stops being a person's job too.

---

dotvault stores the secrets in your .env and config files server-side and hands them to any authorised developer, CI job or AI agent with a single command (https://dotvault.io). It will be released free of charge when this series ends, and the steps in these articles are the steps you will use on day one.

If you find this useful, please like and follow.

Our products are listed at https://www.eln.ne.jp/products

---

EarthLink Network builds its own products to run the whole company on AI — each one born from a need along the way. See the full index of what we build:

→ The 18 products EarthLink Network builds in-house

For more about the company and each product, visit www.eln.ne.jp.