Posted on Oct 18, 2018

Auto-refresh AWS Tokens Using IAM Role and boto3

#aws #cloud #iam

The Curse of The Hour

Session management in AWS is complicated, especially when authenticating with IAM roles. A common way to obtain AWS credentials is to assume an IAM role and be given a set of temporary session keys that are only good for a certain period of time. The maximum session duration is a setting on the IAM role itself, and it is one hour by default. So if users don't specify a value for the DurationSeconds parameter, their security credentials are valid for only one hour.

A typical boto3 request to assume IAM role looks like:

response = client.assume_role(
    RoleArn='string',
    RoleSessionName='string',
    Policy='string',
    DurationSeconds=123,
    ExternalId='string',
    SerialNumber='string',
    TokenCode='string'
)

where DurationSeconds is the duration of the role session. It can go up to the maximum session duration setting for the role. So if your IAM role is only setup to go up to an hour, you wouldn't be able to extend the duration of your sessions unless you update the settings on the IAM role itself.

Long-Lasting Credentials

So an alternative must be introduced to extend IAM role sessions. This is when I found RefreshableCredentials , a botocore class acting like a container for credentials needed to authenticate requests. Moreover, it can automatically refresh the credentials! This is exactly what I need. But it's usage is poorly documented. Looking at its source code:

def __init__(self, access_key, secret_key, token, expiry_time, refresh_using, method, time_fetcher=_local_now):

The __init__ function takes several arguments, half of which I don't recognize. But there's a class method that can be used to initialize an object:

@classmethod
def create_from_metadata(cls, metadata, refresh_using, method):
    instance = cls(
         access_key=metadata['access_key'],
         secret_key=metadata['secret_key'],
         token=metadata['token'],
         expiry_time=cls._expiry_datetime(metadata['expiry_time']),
         method=method,
         refresh_using=refresh_using)
    return instance

where metadata is a dictionary containing information abound the current session, ie. access_key, secret_key, token, and expiry_time, all are things we can get from boto3's STS client's assume_role() request.

To construct the metadata response, we make a simple boto3 API call:

import boto3
sts_client = boto3.client("sts", region_name=aws_region)
params = {
    "RoleArn": self.role_name,
    "RoleSessionName": self.session_name,
    "DurationSeconds": 3600,
}
response = sts_client.assume_role(**params).get("Credentials")
metadata = {
    "access_key": response.get("AccessKeyId"),
    "secret_key": response.get("SecretAccessKey"),
    "token": response.get("SessionToken"),
    "expiry_time": response.get("Expiration").isoformat(),
}

refresh_using is a callable that returns a set of new credentials, taking the format of metadata. Remember that in Python, functions are first-class citizens. You can assign them to variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions. So I just need a function that generates and returns metadata.

def _refresh(self):
    " Refresh tokens by calling assume_role again "
    params = {
        "RoleArn": self.role_name,
        "RoleSessionName": self.session_name,
        "DurationSeconds": 3600,
    }

    response = self.sts_client.assume_role(**params).get("Credentials")
    credentials = {
        "access_key": response.get("AccessKeyId"),
        "secret_key": response.get("SecretAccessKey"),
        "token": response.get("SessionToken"),
        "expiry_time": response.get("Expiration").isoformat(),
    }
    return credentials

Now we're ready to create a RefreshableCredentials object:

from botocore.credentials import RefreshableCredentials
session_credentials = RefreshableCredentials.create_from_metadata(
    metadata=self._refresh(),
    refresh_using=self._refresh,
    method="sts-assume-role",
)

and we can use the credentials to generate a IAM role session that lasts for as long as we need:

from boto3 import Session
from botocore.session import get_session
session = get_session()
session._credentials = session_credentials
session.set_config_variable("region", aws_region)
autorefresh_session = Session(botocore_session=session)

And of course we can generate a boto client within that session, ie.:

db_client = autorefresh_session.client("rds", region_name='us-east-1')

Top comments (13)

Noelle Leigh • Jul 21 '22

For future visitors: RefreshableCredentials isn't documented, but can be found in the source code.

Kostiantyn Noha • Dec 10 '21 • Edited

@li_chastina Thanks fir the snippets!
But one question: in order to refresh credentials do we need to call autorefresh_session.client each time we use a client functions or its enough to always use db_client reference and credentials refreshed automatically?

jinyeng • Jul 24 '23

When I use this sample and print session info, looks like its generating a new credentials on every resource request !

Mandheer • Sep 3 '19

This post made my day.

jinyeng • Jul 24 '23

Thanks For this great article. I am trying to use this code for S3 bucket listing and upload. I have two issues. I see that when I try to list bucket, it seems to again refresh credentials every time.

Also when trying to list bucket contents, it just gets stuck foerver without a response.