Our business customers receive a monthly email with a link to each invoice, and many of them forward it to their accountants. In June the links started failing. An accountant opening one in the afternoon got an XML error page that said ExpiredToken, The provided token has expired, on a link that had been sent that morning with a seven day expiry.
The links are S3 presigned URLs. For years the invoicing job had signed them with the access keys of an IAM user, and seven days, the longest S3 accepts, meant seven days. In May we moved the job to our EKS cluster and replaced the keys with IAM roles for service accounts, which was the right change. The pod now gets temporary credentials from AssumeRoleWithWebIdentity, valid for an hour by default, and the SDK refreshes them before they expire.
A presigned URL is not a grant from S3. It is a request we sign in advance, with whatever credentials we hold at that moment, and S3 checks those credentials when the request finally arrives. With temporary credentials the URL carries the session token, and AWS documents that the URL stops working when that token expires, whatever expiry we wrote into it. The SDK refreshed our credentials for us, but nothing can refresh a link already sitting in somebody's inbox. Each link lived for whatever was left of the session that signed it, somewhere between a few minutes and an hour.
Our tests had passed because they clicked each link within seconds of creating it. Support decoded one failing link, saw X-Amz-Security-Token among its parameters, and the rest followed.
The emails now link to our own download endpoint, carrying a token we sign ourselves and accept for thirty days. The endpoint checks it, then redirects to a presigned URL signed on the spot with a five minute expiry, which the session always outlives. It also lets us revoke a link, which we could never do before. S3 server access logging is on for the invoice bucket, and a rise in requests that fail with ExpiredToken or AccessDenied raises an alert, so a customer is no longer our only monitor. Every change to how a service gets credentials now includes a search for presigned URLs.
We changed who signed the links and kept their promise of seven days. The promise was never ours to make, because a signature lasts only as long as the credentials behind it.
– Sergey Shinder
Top comments (0)