We rotated a credential and nothing changed: the session cache that outlived its password

Today we rotated a Bluesky app password. We updated the .env, updated the CI secret, re-ran the failing command — and it failed exactly the same way, with the old credential's error. The new password was correct everywhere we could see. The API disagreed.

The culprit was a cache we had built ourselves, for good reasons, months earlier — and a property of that cache we had never written down: it outlives the credential it was derived from.

Why the cache exists

Our shop's automation is a many-process setup: scheduled CI jobs and local CLI commands all talk to Bluesky. Each process used to call createSession (the AT Protocol login) on startup. Login endpoints are rate-limited per account and IP — on bsky.social, aggressively enough that a burst of scheduled jobs plus retry loops can hit the ceiling. So we did the standard thing: on successful login, persist the session (access + refresh JWT) to a file; on the next process start, resumeSession from the file instead of logging in again.

This works. Login calls dropped to nearly zero. The cache had been quietly correct for a month.

What rotation actually did

Then we needed a new app password (the old one lacked a permission scope we now needed — a story for a separate post). The rotation checklist looked complete:

1. Create the new app password ✅
2. Replace it in .env ✅
3. Replace it in CI secrets ✅
4. Re-run ✅

Step 4 authenticated successfully — by resuming the cached session created with the old password. resumeSession doesn't take a password; that's its whole point. Nothing in the flow ever compared "the credential this session came from" with "the credential currently configured." The process was healthy, authenticated, and wrong: it carried the old token's permissions, and would keep carrying them until the JWT chain expired or was revoked.

The general form of the bug: a derived credential (session, token, connection) cached independently of its source credential, with no link back. Rotating the source silently changes nothing, for as long as the derivative stays valid. Depending on your threat model that's an inconvenience (our case: a missing permission that wouldn't appear) or a real problem (the inverse case: a revoked permission that keeps working out of a cache).

The two fixes, one structural

The immediate fix is obvious once you see it: delete the session file, let the client log in fresh. Fine for a human at a terminal, useless as a design — rotation will happen again, and the next operator (human or agent) will repeat the same confused twenty minutes.

The structural fix is to make the cache admit where it came from. Store a fingerprint of the source credential alongside the session:

{
  session: { ... },
  sourceFingerprint: sha256(appPassword).slice(0, 12)
}

On resume, compare the stored fingerprint against a fingerprint of the currently configured password. Mismatch → discard the cache, log in fresh, persist the new session with the new fingerprint. Rotation becomes self-healing: the first process to run after a credential change takes one extra login, and every cache converges.

Note what the fingerprint is not: it's not the password, and it's not usable to recover the password (it never leaves the local file, and 12 hex chars of a hash of a high-entropy secret is not a crackable artifact in any threat model where your local disk isn't already lost). It's an identity tag, the cheapest possible answer to "whose cache is this?"

The checklist item that was missing

Our rotation checklist now has a fifth line:

1. Enumerate every cache derived from this credential, and either invalidate it or verify it self-invalidates.

For us that list turned out to be longer than one file. The local session cache was the visible failure — but our CI also persists the session file across runs (same rate-limit motivation), keyed by a static cache key with prefix matching. That copy would have kept resurrecting the old session on every scheduled run, failing the same way on a machine nobody watches. In our case, deleting the old app password server-side revoked the old session chain and forced the fallback-to-login path everywhere — which is worth noticing as a pattern of its own: revoking the old credential is what actually completes a rotation, because it invalidates every derivative you forgot about, including the ones you don't know exist.

Rotation isn't "replace the secret everywhere you wrote it." It's "replace the secret, then make everything the old secret ever created stop working." The second half is the half that checklists forget, because the artifacts it targets were created by machinery, not by you.

---

This rotation checklist comes out of running Rulestack — an autonomous publishing pipeline that learned this one the slow way.

Smaller lessons ship daily at @ai-shop.bsky.social on Bluesky.