DEV Community: Chastina Li 👩🏻‍💻

Deploying to AWS Cloud Through Azure DevOps

Chastina Li 👩🏻‍💻 — Wed, 31 Oct 2018 02:28:39 +0000

Azure DevOps is a platform for running builds and deployments pipelines for your applications. It's recently rebranded to Azure DevOps, just to jump on the bandwagon of the buzzword. It's essentially every other build system, gives you a YAML declarative pipeline DSL, and a UI if you want to build your CICD pipelines by click and drag. It's no where as popular as some of the other big build servers like Jenkins or CircleCI, therefore it's lacking quite a bit in terms of documentations, tutorials and guides.

I recently helped someone deploying AWS infrastructure using Azure DevOps, weird I know. I was initially confused about the choice, but after checking out Azure DevOps's AWS plugin, I kinda understood why a small shop would choose it to be it's CICD tool. Azure DevOps follows Jenkins in that it only provides a few basic deployment modules (called tasks in Azure DevOps) out of the box, like bash scripts, npm builds and maven builds. The rest of its power comes from a rich library of plugins, such as the AWS plugin.

YAML DSL

The YAML DSL configures which jobs to run and on which kind of servers. It also exposes some pipeline configs like what to do when a job failed, do we continue or do we quit. Below is what I wrote for my project, minimal but gets the job done:

jobs:
- job: MyJob
  pool:
    vmImage: 'ubuntu-16.04'
  displayName: My First Job
  continueOnError: true
  workspace:
    clean: outputs
  steps:
......

steps: starts a list of modules to run.

AWS Credentials

AWS Credentials can be configured via the UI (Project Settings -> Service connections -> add new AWS Service Connection). A caveat is that Service Connections only get loaded during pipeline initiation, any new connections added after the pipeline has been created won't get loaded automatically. So you'll have to delete and re-create your pipeline to use new connections. I've explained this in a Github issue that talks about this caveat. It looks like quite a few people are running into this issue, so I'm just sharing lessons learned everywhere.

Cloudformation Module

The AWS plugin comes with several pretty useful modules, like CloudFormation Update/Create, Lambda Deploy and S3 Upload. Unfortunately I don't think they published the documentation for it, so I had to look at their source code to find the docs. I've only used CloudFormation Update/Create and another module called AWS CLI. The CloudFormation module greatly saved my time because I didn't have to handle the idempotence of multiple updates after the initial creation, the module knows to update instead of create if the CloudFormation stack is already created.

  steps:
  - task: CloudFormationCreateOrUpdateStack@1
    inputs:
      awsCredentials: 'aws_tokens'
      regionName: 'us-east-2'
      stackName: 'IAMRoleStack'
      templateFile: templates/iam_role.json
      capabilityIAM: 'true'
      capabilityNamedIAM: 'true'

Here is what I used to deploy my IAM role, awsCredentials, regionName are required for all AWS modules, and stackName and templateFile are required for all CloudFormation modules. The last two are only specific to this module.

AWS CLI Module(or not)

The AWS CLI module gave me false hopes, turned out that you still have to install AWS CLI yourself in order to use the module LOL. I first had to install setuptools and wheel, yeah, pip dependencies doesn't even come for free. I then have to install it in my user space, and it took me a while find where the CLI got installed to because it's not added to my PATH. So I just used the AWS Shell script module, which seems to do exact what the AWS CLI module does, except easier to write.

- script: |
      python -m pip install --upgrade pip==9.0.3 setuptools wheel
      pip install awscli --user
    displayName: 'Install tools'
  - task: AWSShellScript@1
    inputs:
      awsCredentials: 'aws_tokens'
      regionName: 'us-east-2'
      scriptType: 'inline'
      inlineScript: |
        eval $(/home/vsts/.local/bin/aws ecr get-login --no-include-email)

Idempotently Create/Update AWS RDS Instances in Python

Chastina Li 👩🏻‍💻 — Wed, 24 Oct 2018 22:01:44 +0000

There are several ways to manage databases via code, for Amazon Web Service's Relational Database Service(RDS), you can use Terraform, Cloudformation Templates, or API libraries in your favorite language. While Terraform's statefiles are a hassle to managed and sometimes it will delete your database and start from scratch. And CloudFormation templates are too verbose to be managed properly. In this blog post, I'm talking specifically about the Python AWS API library: boto3.

Creating a Postgres database

Firstly, we need to set up the boto client:

import boto3
from botocore.config import Config

boto_config = Config(retries=dict(max_attempts=20))
client = boto3.client(
    'rds', region_name='us-east-1', config=boto_config
)

then let's invoke the create_db_instance() method with a set of configurations:

db_vars = {
    "DBName": "db_name",
    "DBInstanceIdentifier": "instance_name",
    "AllocatedStorage": 20,
    "DBInstanceClass": "db.m3.medium",
    "Engine": "postgres",
    "MasterUsername": "username",
    "MasterUserPassword": "password",
    "VpcSecurityGroupIds": [
        "sg-0007c6489efbd9bca",
    ],
    "DBSubnetGroupName": "my-subnet",
    "DBParameterGroupName": "my-parameter-group",
    "BackupRetentionPeriod": 7,
    "MultiAZ": True,
    "EngineVersion": "10.0.1",
    "PubliclyAccessible": False,
    "StorageType": "gp2",
}
client.create_db_instance(**db_vars)

Updating existing database

Updating database is a bit more complicated because it involves handling idempotency. In computing, an idempotent operation is one that has no additional effect if it is called more than once with the same input parameters. Why is it important? We don't want to update the same set of configurations on the database again and again, this causes confusion from boto and will result in errors.

Part of handling idempotency is reconciliation with existing state. This usually involve computation of differences between desired state and current state. For boto, this means whatever we give modify_db_instance must represent this difference. In Python, this can simply be done with a Dictionary comprehension:

updatable_boto_params = (
    "AllocatedStorage",
    "DBInstanceClass",
    "DBParameterGroupName",
    "DBSubnetGroupName",
    "EngineVersion",
    "Iops",
    "StorageType",
)

updated_params = {
    param: new_params.get(param)
    for param in updatable_boto_params
    if not current_params.get(param) == new_params.get(param)
}

where updatable_boto_params is parameters that you allow to update, you can obviously lock this list down to a very small set of parameters for compliance or security purposes. See the complete list of updatable configurations here.

Now that we have the change set, we can invoke the modify_db_instance:

client.modify_db_instance(**updated_params)

Waiting for it to finish

Database operations are usually long-running, when performing a series of steps sequentially (ie. create database and create a read replica), you need to wait for the synchronous process to finish. One way to tell if database has finished updating itself is by looking at the DBInstanceStatus returned by the 'describe_db_instances' method. Status of available means database has finished whatever it's doing and you can proceed to the next operation.

A better way is to use boto's RDS waiter. It polls describe_db_instances() every 30 seconds until a successful state is reached by default. But you can customize how frequently you want it to pool database status, and how long before you give up and declare the database not ready.

We can also use Python's context manager as a nice little wrapper of the waiter, we can also throw whatever exception handling logic into the context manager just to clean up the code:

from contextlib import contextmanager
from botocore.exceptions import WaiterError

@contextmanager
def wait_for_availability(instance_id, delay=120, retries=60):
    " Yield back control after database is available "
    yield True
    try:
        waiter = self.db_client.get_waiter("db_instance_available")
        waiter.wait(
            DBInstanceIdentifier=instance_id,
            WaiterConfig={"Delay": delay, "MaxAttempts": retries},
        )
    except WaiterError as exception:
        LOG.error("DBInstance: %s is not available.", instance_id)
        raise exception
    else:
        LOG.info("DB instance: %s is now available.", instance_id)
    break


with wait_for_availability(instance_id):
    client.modify_db_instance(**updated_params)

In the code snippet above, we modify our database and pool for its status every 2 minutes and for a total of 60 times. After that the database is either available and we exit, or still unavailable and we raise an exception.

Database operations are usually time consuming, and guarantee idempotence is a major complexity to deal with. Fortunately, boto provides a bunch of very useful APIs to satisfy our needs.

Auto-refresh AWS Tokens Using IAM Role and boto3

Chastina Li 👩🏻‍💻 — Thu, 18 Oct 2018 17:50:35 +0000

The Curse of The Hour

Session management in AWS is complicated, especially when authenticating with IAM roles. A common way to obtain AWS credentials is to assume an IAM role and be given a set of temporary session keys that are only good for a certain period of time. The maximum session duration is a setting on the IAM role itself, and it is one hour by default. So if users don't specify a value for the DurationSeconds parameter, their security credentials are valid for only one hour.

A typical boto3 request to assume IAM role looks like:

response = client.assume_role(
    RoleArn='string',
    RoleSessionName='string',
    Policy='string',
    DurationSeconds=123,
    ExternalId='string',
    SerialNumber='string',
    TokenCode='string'
)

where DurationSeconds is the duration of the role session. It can go up to the maximum session duration setting for the role. So if your IAM role is only setup to go up to an hour, you wouldn't be able to extend the duration of your sessions unless you update the settings on the IAM role itself.

Long-Lasting Credentials

So an alternative must be introduced to extend IAM role sessions. This is when I found RefreshableCredentials , a botocore class acting like a container for credentials needed to authenticate requests. Moreover, it can automatically refresh the credentials! This is exactly what I need. But it's usage is poorly documented. Looking at its source code:

def __init__(self, access_key, secret_key, token, expiry_time, refresh_using, method, time_fetcher=_local_now):

The __init__ function takes several arguments, half of which I don't recognize. But there's a class method that can be used to initialize an object:

@classmethod
def create_from_metadata(cls, metadata, refresh_using, method):
    instance = cls(
         access_key=metadata['access_key'],
         secret_key=metadata['secret_key'],
         token=metadata['token'],
         expiry_time=cls._expiry_datetime(metadata['expiry_time']),
         method=method,
         refresh_using=refresh_using)
    return instance

where metadata is a dictionary containing information abound the current session, ie. access_key, secret_key, token, and expiry_time, all are things we can get from boto3's STS client's assume_role() request.

To construct the metadata response, we make a simple boto3 API call:

import boto3
sts_client = boto3.client("sts", region_name=aws_region)
params = {
    "RoleArn": self.role_name,
    "RoleSessionName": self.session_name,
    "DurationSeconds": 3600,
}
response = sts_client.assume_role(**params).get("Credentials")
metadata = {
    "access_key": response.get("AccessKeyId"),
    "secret_key": response.get("SecretAccessKey"),
    "token": response.get("SessionToken"),
    "expiry_time": response.get("Expiration").isoformat(),
}

refresh_using is a callable that returns a set of new credentials, taking the format of metadata. Remember that in Python, functions are first-class citizens. You can assign them to variables, store them in data structures, pass them as arguments to other functions, and even return them as values from other functions. So I just need a function that generates and returns metadata.

def _refresh(self):
    " Refresh tokens by calling assume_role again "
    params = {
        "RoleArn": self.role_name,
        "RoleSessionName": self.session_name,
        "DurationSeconds": 3600,
    }

    response = self.sts_client.assume_role(**params).get("Credentials")
    credentials = {
        "access_key": response.get("AccessKeyId"),
        "secret_key": response.get("SecretAccessKey"),
        "token": response.get("SessionToken"),
        "expiry_time": response.get("Expiration").isoformat(),
    }
    return credentials

Now we're ready to create a RefreshableCredentials object:

from botocore.credentials import RefreshableCredentials
session_credentials = RefreshableCredentials.create_from_metadata(
    metadata=self._refresh(),
    refresh_using=self._refresh,
    method="sts-assume-role",
)

and we can use the credentials to generate a IAM role session that lasts for as long as we need:

from boto3 import Session
from botocore.session import get_session
session = get_session()
session._credentials = session_credentials
session.set_config_variable("region", aws_region)
autorefresh_session = Session(botocore_session=session)

And of course we can generate a boto client within that session, ie.:

db_client = autorefresh_session.client("rds", region_name='us-east-1')