At twenty to seven on a Monday every call to our carrier's API started returning invalid grant. Not some of them. All of them, from every pod, for forty minutes, until somebody opened the partner portal and reauthorised the connection by hand with a browser and a password nobody had needed in two years.
Their platform uses refresh token rotation. Every time you exchange a refresh token you get a new one and the old one dies, and presenting a dead one is treated as evidence that it has been stolen, so the provider revokes the whole family including the access token issued a moment earlier. All of that is written in their documentation and all of it is sensible.
Our shipment poller holds the credential in memory and refreshes when its access token is within a minute of expiry. In May we scaled it from one pod to twenty to keep up with volume, and twenty pods that were deployed together hold tokens that expire together. That Monday all twenty crossed the threshold inside the same second, each read the same stored refresh token, and presented it. One of them won. The other nineteen presented a token that had been dead for a few milliseconds, the provider concluded it was being replayed, and it revoked everything, including the perfectly good token the winner had just been handed.
Nothing about the scale up looked like an authentication change. It was a replica count in a values file, reviewed and approved by people thinking about throughput.
There is one refresher now. The token pair lives in Redis, a worker that finds it close to expiry takes a short lock, refreshes, writes the new pair with its expiry, and everyone else waits and rereads rather than calling the provider. Refresh happens at three quarters of the lifetime instead of at the edge, so there is room to lose a race and recover. And a revocation error pages somebody, because recovery needs a human to give consent in a browser and no amount of retrying will ever produce that.
A credential that several processes share is shared mutable state, and every rule you already know about writing to shared state applies to it. Scaling something out changes how it authenticates, whether or not anybody meant it to.
– Sergey Shinder
Top comments (0)