A key set cached for fifteen minutes stays correct right up to the moment the issuer rotates. An issuer signs with a new key as soon as it publishes it, so a resource server waiting out its TTL turns away every token minted in the gap: good signature, trusted issuer, correct audience, 401. What the cache needs is a second trigger with nothing to do with age, a token naming a key id the cache does not hold, and once you add that trigger the hard decisions move somewhere new.
I have been verifying bearer tokens since 2018, in production systems where money moves, and this is the failure that never turns up in staging. Rotations are rare, on the issuer's side they are instant, and the symptom is a few minutes of 401s that stop by themselves before anyone has finished reading the logs. The code below comes from mcpaudience, a small Go package that does the resource-server half of MCP authorization, and the key set lives in jwks.go.
The timer answers a question nobody asked
A TTL answers one question: how old is my copy of the key set. The question a verifier actually has is whether that copy contains the key this token names. Those two questions agree nearly all of the time and come apart during exactly one event, a rotation. The issuer publishes rsa-2 and starts signing with it right away, and my cached copy, eight minutes old and perfectly fresh by its own accounting, holds only rsa-1.
So this is not a TTL cache. The lookup takes the kid first and the clock second:
func (j *JWKS) Key(ctx context.Context, kid string) (crypto.PublicKey, error) {
if err := j.checkURL(); err != nil {
return nil, err
}
// The lock is held across the fetch, so a burst of requests arriving on a
// rotation makes one request at the issuer rather than one each.
j.mu.Lock()
defer j.mu.Unlock()
cached, known := j.keys[kid]
if known && !j.stale() {
return cached, nil
}
if err := j.refresh(ctx); err != nil {
if known {
// An issuer being unreachable is not evidence that the key it
// published stopped being its key. Tokens still expire on their own.
return cached, nil
}
return nil, fmt.Errorf("no key for kid %q: %v", kid, err)
}
if key, ok := j.keys[kid]; ok {
return key, nil
}
if reason, ok := j.skipped[kid]; ok {
return nil, fmt.Errorf("the key set lists kid %q but it cannot be used: %s", kid, reason)
}
return nil, fmt.Errorf("no key for kid %q in the key set at %s", kid, j.URL)
}
The fast path is those first four lines: a kid I hold, in a set that is not stale, answered out of memory with no network at all. Everything after that is the rotation path. CacheFor is still there and still forces a refetch at fifteen minutes by default, but its job has shrunk to housekeeping, eventually noticing keys the issuer took away. It is not what makes rotation work any more.
About the mutex: it is held across an HTTP request, not only around the map. That serializes verification for the length of one fetch, which sounds expensive until you price the alternative: a rotation throws off a burst of unknown-kid lookups that all arrive together, and with no lock each one of them opens its own connection to the issuer. One bounded wait is cheaper than a herd I caused myself. Bounded is doing real work in that sentence, because the default client carries a ten second timeout and the response body is read through io.LimitReader up to maxKeySetBytes, so a hanging or endless issuer cannot sit on the lock forever.
A fetch the caller can trigger needs a floor
This is the part that makes the unknown-kid trigger more than a one-line change. The kid comes out of the token header, and the token header is attacker-controlled. If an unknown kid means a fetch, then a stream of tokens carrying invented kids means a fetch each, and my resource server turns into a small load generator aimed at my own authorization server. That server is shared infrastructure: the amplification takes out the login flow, not just my service.
So refresh refuses to run twice inside MinRefresh, thirty seconds by default, and it records the attempt before it tries:
func (j *JWKS) refresh(ctx context.Context) error {
now := j.now()
if since := now.Sub(j.attempted); !j.attempted.IsZero() && since < j.minRefresh() {
return fmt.Errorf("the key set was fetched %s ago and is not fetched again more than once every %s",
since.Round(time.Second), j.minRefresh())
}
j.attempted = now
body, err := j.get(ctx)
if err != nil {
return err
}
keys, skipped, err := parseKeySet(body)
if err != nil {
return fmt.Errorf("%s: %v", j.URL, err)
}
j.keys, j.skipped, j.fetched = keys, skipped, now
return nil
}
func (j *JWKS) minRefresh() time.Duration {
if j.MinRefresh > 0 {
return j.MinRefresh
}
return 30 * time.Second
}
j.attempted = now sits above j.get, and that is the line I would get wrong if I were writing this in a hurry. Record the attempt only on success and a down issuer means the floor never engages, so every single request retries: the floor would hold exactly when nothing needed holding, then disappear during the outage, when hammering a sick issuer is at its worst. Note too that j.fetched moves forward only on success, so a failed attempt limits retries without making a stale set look fresh. Two timestamps doing two different jobs, and collapsing them into one is tempting and wrong.
The behaviour is cheap to pin down in a test, and this is the one I would write first:
// Refetching on an unknown kid is free traffic at the issuer unless it is rate
// limited: invented key ids must not become one request each.
func TestAnUnknownKeyIDIsNotAFetchPerToken(t *testing.T) {
s := newSigner(t)
issuer := newKeyServer(t, rsaJWK("rsa-1", &s.rsaKey.PublicKey))
v := keySetVerifier(&JWKS{URL: issuer.url, MinRefresh: time.Minute, Now: func() time.Time { return now }})
for i := 0; i < 5; i++ {
_, err := v.Verify(context.Background(), s.sign(t, "RS256", "invented", goodClaims()))
if !errors.Is(err, ErrBadSignature) {
t.Fatalf("err = %v, want ErrBadSignature", err)
}
}
if got := issuer.count(); got != 1 {
t.Errorf("the issuer was asked %d times for a key id it never published, want once", got)
}
}
The injected Now is what makes this testable without sleeping. A key set cache is almost all clock-dependent logic, and a clock you cannot set is a cache you cannot test.
Unknown is three answers, not one
After a successful refresh there are three distinct reasons the kid is still not usable, and the tail of Key keeps them apart. The key is present: return it. The kid is in skipped, meaning the issuer did publish it but parseKeySet dropped it and remembered why: an HMAC or none alg, an RSA modulus under 2048 bits, a curve that is not P-256, a kty of oct. That refusal says so, with the reason attached. Only a kid the issuer has genuinely never published comes back as "unknown".
Two lines of distinction there save an hour of somebody's day. "Unknown kid" sends an operator off to the issuer to check whether the key was published, and the issuer answers yes, it is right there in the document. "The key set lists kid rsa-3 but it cannot be used: RSA modulus is 1024 bits" sends them straight to the right place. A set with no usable key in it is an error rather than an empty cache, and the previous keys stay put, since a malformed publish should not take verification down.
The fetch is a trust boundary too, and the cheapest guard sits at configuration time:
func (j *JWKS) checkURL() error {
raw := strings.TrimSpace(j.URL)
if raw == "" {
return fmt.Errorf("no key set URL configured")
}
parsed, err := url.Parse(raw)
if err != nil || !parsed.IsAbs() || parsed.Host == "" {
return fmt.Errorf("key set URL %q is not an absolute URL", j.URL)
}
if parsed.Scheme != "https" && parsed.Hostname() != "localhost" && parsed.Hostname() != "127.0.0.1" {
return fmt.Errorf("key set URL %q must use https: a key set an attacker on the path can replace verifies the attacker's tokens", j.URL)
}
return nil
}
The error message is the argument. Nothing else in a verifier counts for much if the document that supplies the public keys can be swapped in transit.
Availability over freshness, stated out loud
The last branch is the one worth arguing about. When the refresh fails and the kid was already held, the cached key verifies the token anyway. My reasoning: an issuer being unreachable is not evidence that the key it published stopped being its key, and the tokens it signed expire on their own. Refusing every request while the issuer is down converts a brief issuer blip into a total outage of my service, and it buys no security, since nothing about an HTTP 503 says anything about a public key.
The cost is real and I will name it. A key the issuer deliberately removed keeps verifying tokens until the next successful fetch. If a signing key leaks and the issuer pulls it from the document, my server honours it for as long as the issuer stays unreachable. That is the trade, and I think it is only acceptable because a key set was never a revocation mechanism: pulling a key does not invalidate the already-issued tokens signed with it either. If you need revocation that takes effect now, you need token introspection, and then you are making a request per token with its own failure policy instead of pretending a cached document can do that job.
Note that the fallback applies only to a kid already held. An unknown kid during an outage gets the error, because there is nothing to fall back to and guessing would mean picking some other key out of the set.
One more detail decides what the client sees. The verifier resolves a kid against its configured Keys first and the fetched KeySet second, so a fetched key never overrides a pinned one, and every key-set failure comes back wrapped in ErrBadSignature:
func (v *JWTVerifier) key(ctx context.Context, kid string) (crypto.PublicKey, error) {
if key, ok := v.Keys[kid]; ok {
return key, nil
}
if v.KeySet == nil {
return nil, fmt.Errorf("%w: no key for kid %q", ErrBadSignature, kid)
}
if strings.TrimSpace(kid) == "" {
// Without a kid there is nothing to look up and nothing to rotate; a
// verifier that guesses picks the wrong key the first time one is added.
return nil, fmt.Errorf("%w: the token names no kid, so no key can be chosen", ErrBadSignature)
}
key, err := v.KeySet.Key(ctx, kid)
if err != nil {
return nil, fmt.Errorf("%w: %v", ErrBadSignature, err)
}
return key, nil
}
The wrapping is a deliberate answer to a question that comes up in review: should "I could not reach the issuer" be a 500? It should not. The client's situation is identical either way, the token it presented did not verify and its move is to go get a new one, and a 500 invites it to retry that same token against a server it has just been told is broken. A token with no kid at all is refused for the same reason it cannot be rotated: there is nothing to look up, and a verifier that guesses picks wrong the first time the issuer adds a second key.
What I would do differently
The rate limit is global, not per kid. An attacker spraying invented kids can burn the one allowed attempt in an interval and delay the refetch that a genuine rotation needs by up to MinRefresh. With a thirty second default that window is small and the failure heals itself, so it is a defensible shape for a small package. In a system where I controlled both ends, I would keep a per-kid negative cache next to the floor, so a kid that was already refused and is already absent from a freshly fetched set does not spend the shared attempt at all.
I would also not leave the outage recovery passive. As it stands, the last good set returns to normal only when some request happens to arrive with a kid that is unknown or a set that has gone stale. A background refresh attempt during a known-failed state would shorten the window in which a removed key stays live, which is the one cost of the availability trade.
And I would instrument skipped and attempted. Both are facts about the issuer that nothing else in the system knows: a key published but unusable, and a refetch declined by the floor. They stay invisible until someone reads a 401 and starts guessing...
The shape that matters generalizes past JWKS. When a cache's correctness depends on an external event rather than on elapsed time, the trigger belongs with the event and not with the clock, and the moment that trigger becomes reachable by an untrusted input, it needs a floor that counts failures.
Originally published on polycratia.com — where I write about payment systems, crypto rails and marketplace backends.
Top comments (5)
The refresh floor is shared, so an attacker can choose when it gets spent. Send an invented kid every 30 seconds and every fetch slot goes to junk. When the issuer really rotates, tokens carrying rsa-2 get turned away until the floor runs out. That's at most
MinRefreshof 401s, a fair price for protecting the issuer, but it's the rotation gap the post set out to close, now on a timer the attacker sets.I'd make it visible rather than try to remove it. The test I'd add next to yours: invented kid at t=0, issuer publishes rsa-2 at t=1, a real token at t=2 fails, and the same token at t=31 passes. Then I'd stop wrapping the floor refusal in
ErrBadSignature. A refusal from the floor says nothing about the token, so a 503 withRetry-Afterset to the time left on the floor lets a client retry once. A 401 tells it to throw the token away, and a fresh one signed with rsa-2 gets refused for the same reason.You're right, and you've put a number on something I only waved at: the floor is shared, so the attacker decides when it gets spent. The article admits the rate limit is global, but your sequence is the honest version of that admission, so I'm taking the test more or less as written. Invented kid at t=0, rsa-2 published at t=1, a real token refused at t=2 and accepted at t=31. It documents the cost instead of hiding it in a paragraph.
On the status code I agree with the diagnosis and would move the fix one layer up. A floor refusal says nothing about the token, and wrapping it in ErrBadSignature throws that fact away. So the verifier should return a distinct error, something like ErrKeySetCoolingDown with the time left on the floor, and leave the HTTP mapping to the handler. In an API I'd map it to 503 with Retry-After, as you say. In a library I'd rather not pick HTTP codes for the caller, but I'd make it impossible for the caller to confuse "your token is bad" with "I can't look right now".
The other half is a per-kid negative cache. Once an invented kid has been looked up and refused, it remembers its own refusal, and the shared attempt is only spent on kids nobody has seen before. The attacker then has to invent a new kid every thirty seconds to keep the floor busy, which is still possible but no longer free.
Thanks for reading it that closely. This is the review the code should have had before it shipped.
kid first, clock second. a TTL answers how old the copy is, not whether it has the key.
That's the whole post in two lines, and better than my title. The one thing I'd add in the clock's defence: removal. A cache keyed on kid never learns that the issuer pulled a key, because nothing arriving asks about it. So the TTL stays, but demoted to housekeeping, a ceiling on how long a withdrawn key keeps working rather than the thing that makes rotation work. Curious how you handle that in rootmail, since mail is one of the places where a key quietly disappearing hurts the most.
Good catch on removal. For rootmail in beta, a withdrawn key should fail loudly, since mail is where a silent miss costs the most.