A Credential Rotation Without an Overlap Is Just an Outage You Scheduled

TL;DR — I shipped managed S3-compatible storage servers today, and the most interesting thing in the whole feature wasn't the provisioning pipeline. It was discovering that the storage daemon has no "rotate key" command — and that this absence is a better design than the setPassword() I'd already written for databases.

---

The rotation I got wrong first

My database provisioning has had credential rotation for a while. The contract looks about how you'd expect:

interface DatabaseAdminContract
{
    public function setPassword(string $username, string $password): void;
}

One call. The old password stops working, the new one starts working, done. Clean API, satisfying to write.

Here's the thing: between the moment that call returns and the moment every application holding the old password has been redeployed with the new one, every one of those applications is broken. Not degraded — broken, authentication-failure broken. The rotation didn't secure anything during that window. It just caused an outage, and the outage's length is however long your slowest deploy takes.

We call that a rotation. It's a cutover wearing a rotation's clothes.

The absence that turned out to be the answer

When I wired up the S3-compatible daemon, I went looking for its rotate command. There isn't one. You can create an access key, you can delete an access key, and that's the vocabulary.

My first reaction was "fine, I'll emulate it — delete then create." My second reaction, about ten seconds later, was that emulating it would reproduce exactly the window I just described, on purpose, in a system that didn't have it.

So rotation stopped being a call and became a workflow:

1. Issue a replacement key on the same server.
2. Re-attach it to the same buckets with the same privileges.
3. Let the application cut over on its own schedule.
4. Delete the outgoing key.

Both keys work for the whole span between 1 and 4. There is no moment where nothing authenticates. The cost is that a one-liner became four acts, two of which belong to a human.

Modelling "both keys are valid" as normal

This is the part that bit me in review. If two live credentials for the same bucket look like a fault in your UI, the next operator who sees it will "fix" it — by deleting one, probably the wrong one, probably during a deploy.

So the overlap has to be a first-class state, not an anomaly. In Laravel terms that's an enum carrying its own presentation:

enum AccessKeyStatus: string
{
    case Pending  = 'pending';
    case Active   = 'active';
    case Retiring = 'retiring';   // rotating out — still valid, on purpose
    case Deleted  = 'deleted';

    public function label(): string
    {
        return match ($this) {
            self::Retiring => __('Retiring — still valid until the overlap ends'),
            // …
        };
    }

    public function color(): string
    {
        return match ($this) {
            self::Retiring => 'amber',   // attention, not alarm
            self::Active   => 'emerald',
            // …
        };
    }
}

Amber, not red. The distinction matters more than it sounds: red means something went wrong, amber means something is in progress and will need you later. A retiring key is the second thing.

Two actions, because the second one is a decision

Starting a rotation is safe and can be one click. Finishing one is not, so it isn't the same action:

final class RotateAccessKeyAction
{
    public const DEFAULT_OVERLAP_DAYS = 7;

    public function execute(User $user, AccessKey $outgoing, int $overlapDays = self::DEFAULT_OVERLAP_DAYS): AccessKey
    {
        if ($overlapDays < 1) {
            throw ValidationException::withMessages([
                'overlapDays' => __('The overlap must be at least a day — a rotation with no window is a cutover, and this exists to avoid the outage a cutover causes.'),
            ]);
        }

        // Resume, don't start a second one. Two live replacements is worse
        // than none: the recorded window would cover the wrong pair.
        $existing = $outgoing->replacements()
            ->whereIn('status', [AccessKeyStatus::Pending, AccessKeyStatus::Active])
            ->first();

        if ($existing instanceof AccessKey) {
            return $existing;
        }

        // …issue the replacement, mirror the bucket permissions,
        //   mark the outgoing key Retiring with an expiry.
    }
}

Two details in there I'd argue for in any rotation implementation.

The idempotency check isn't defensive coding, it's correctness. Double-clicking "Rotate" shouldn't produce two replacement keys. Not because two keys is untidy, but because the compliance record says "credential X is being replaced by credential Y, and X stays valid until Z" — and that sentence has no meaning with two Ys in it.

The validation message explains the rule, not the constraint. The overlap must be at least a day is a form error. The sentence after the dash is the reason someone can act on. Error copy is where your architectural decisions actually reach the people using the thing.

Completing the rotation is separate, and it refuses by default:

final class CompleteAccessKeyRotationAction
{
    public function execute(User $user, AccessKey $outgoing, bool $force = false): void
    {
        $replacement = $outgoing->replacements()
            ->where('status', AccessKeyStatus::Active)
            ->first();

        if (! $replacement instanceof AccessKey) {
            throw ValidationException::withMessages([
                'key' => __('The replacement has not been created on the daemon yet. Withdrawing this one now would leave nothing able to authenticate.'),
            ]);
        }

        if (! $force && ! $outgoing->hasExpired()) {
            throw ValidationException::withMessages([
                'key' => __('The overlap runs until :until. Finish early only if you are certain nothing is still using this key.', [
                    'until' => $outgoing->expires_at?->toDayDateTimeString(),
                ]),
            ]);
        }

        // …audit, close the rotation record, then reuse the ordinary delete.
    }
}

Why not just delete it on a timer?

This was the tempting shortcut and I want to be honest about why I didn't take it.

The platform cannot see whether an application still holds the old secret. Nothing reports that. There's no callback, no heartbeat, no "I have cut over" signal. So a scheduled job that deletes the outgoing key when the timer expires is a platform arranging an outage for itself, at 3am, with nobody watching.

The expiry on the key is the deadline. The completion action is the decision. Keeping those two things separate is the whole point — the deadline creates the pressure, a human confirms the fact.

The trade-off is real and I'll name it: keys can now linger past their window if nobody clicks the button. That's a reporting problem (surface the overdue ones loudly), and I'd rather have a reporting problem than a scheduled outage.

The force flag is not a backdoor

Note that force doesn't skip the "does a replacement exist" check — only the "has the window elapsed" check. That's deliberate. One of those guards protects against impatience; the other protects against leaving a bucket with zero working credentials. Impatience is a judgement call. Zero working credentials never is.

If you take one thing from this post, take that shape: when you add an escape hatch, work out which of your guards it's allowed to open. A force that opens all of them isn't an escape hatch, it's an unguarded second code path.

Testing the window, not the calls

Rotation is exactly the kind of feature where mocking the daemon and asserting "createKey was called" tests nothing worth testing. What you actually care about is that both credentials work at the same time, and that the record says so:

it('keeps the outgoing key valid for the whole overlap', function () {
    $key = AccessKey::factory()->active()->create();

    $replacement = app(RotateAccessKeyAction::class)
        ->execute($this->operator, $key, overlapDays: 7);

    expect($key->fresh()->status)->toBe(AccessKeyStatus::Retiring)
        ->and($key->fresh()->hasExpired())->toBeFalse()
        ->and($replacement->status)->toBe(AccessKeyStatus::Active)
        ->and($replacement->id)->not->toBe($key->id);
});

it('refuses to start a second rotation for the same key', function () {
    $key = AccessKey::factory()->active()->create();

    $first  = app(RotateAccessKeyAction::class)->execute($this->operator, $key);
    $second = app(RotateAccessKeyAction::class)->execute($this->operator, $key->fresh());

    expect($second->id)->toBe($first->id);
});

it('refuses to complete while the window is still open', function () {
    // …rotate, then immediately try to complete
    expect(fn () => app(CompleteAccessKeyRotationAction::class)->execute($this->operator, $key))
        ->toThrow(ValidationException::class);
});

Three tests, three rules, none of them touching a real daemon. A fake admin driver behind the same contract makes that possible — which is the other half of the story and probably its own post.

The takeaway

I went in thinking the missing rotate command was a gap to paper over. It was a constraint that made me build the right thing.

If your rotation is a single call that swaps a value in place, ask what's holding the old value and how it finds out. If the answer is "we redeploy everything quickly," you don't have rotation — you have a cutover you've agreed not to talk about.

Overlap first. Deadline second. Human decides when the old one goes.

Next up: I still need to go back and give the database side the same treatment. Its setPassword() works fine, right up until the day it doesn't.