DEV Community: Cameron Jenkinson

How to create an RDS Aurora serverless instance with CDK

Cameron Jenkinson — Sun, 02 Feb 2020 12:56:39 +0000

Amazon Aurora and Aurora Serverless is a cloud computing database service that allows applications using ephemeral computing like lambda to be able to use schema based databases like SQL and Postgres without having to worry about connection pools.

That is the main benefit for me at least alongside a straightforward SDK using HTTP based 'Data API' queries to read and write data.

Before Aurora there were limited options for database services with serverless infrastructure because managing connection pools was difficult and posed risks.

Amazon Aurora Serverless is an on-demand, auto-scaling configuration for Amazon Aurora (MySQL-compatible and PostgreSQL-compatible editions), where the database will automatically start up, shut down, and scale capacity up or down based on your application's needs.

Let's go ahead and create an Aurora Serverless instance on AWS with CDK.

Here is the entire stack upfront where I dive into each section in further detail below.

const cdk = require('@aws-cdk/core');
const rds = require('@aws-cdk/aws-rds');
const secretsManager = require('@aws-cdk/aws-secretsmanager');
const ssm = require('@aws-cdk/aws-ssm');

class DBStack extends cdk.Stack {
  constructor(app, id, { serviceName = 'movies', stage, accountId, }) {
    super(app, id);

    const databaseUsername = 'movies-database';

    const databaseCredentialsSecret = new secretsManager.Secret(this, 'DBCredentialsSecret', {
      secretName: `${serviceName}-${stage}-credentials`,
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          username: databaseUsername,
        }),
        excludePunctuation: true,
        includeSpace: false,
        generateStringKey: 'password'
      }
    });

    new ssm.StringParameter(this, 'DBCredentialsArn', {
      parameterName: `${serviceName}-${stage}-credentials-arn`,
      stringValue: databaseCredentialsSecret.secretArn,
    });

    const isDev = stage !== "production";
    const dbConfig = {
      dbClusterIdentifier: `main-${serviceName}-${stage}-cluster`,
      engineMode: 'serverless',
      engine: 'aurora-postgresql',
      engineVersion: '10.7',
      enableHttpEndpoint: true,
      databaseName: 'main',
      masterUsername: databaseCredentialsSecret.secretValueFromJson('username').toString(),
      masterUserPassword: databaseCredentialsSecret.secretValueFromJson('password'),
      backupRetentionPeriod: isDev ? 1 : 30,
      finalSnapshotIdentifier: `main-${serviceName}-${stage}-snapshot`,
      scalingConfiguration: {
        autoPause: true,
        maxCapacity: isDev ? 4 : 384,
        minCapacity: 2,
        secondsUntilAutoPause: isDev ? 3600 : 10800,
      }
    };

    const rdsCluster = new rds.CfnDBCluster(this, 'DBCluster', { dbConfig,
      deletionProtection: isDev ? false : true,
    });

    const dbClusterArn = `arn:aws:rds:${this.region}:${this.account}:cluster:${rdsCluster.ref}`;

    new ssm.StringParameter(this, 'DBResourceArn', {
      parameterName: `${serviceName}-${stage}-resource-arn`,
      stringValue: dbClusterArn,
    });
  }
}

module.exports = { DBStack };

Breaking it down

Authentication credentials

Storing the authentication credentials in secrets manager makes it easy to access the instance from different services e.g a lambda function as well as when doing manual queries from the console UI.

We define a new secret here and store the database credentials in SSM parameter store. An option has been passed to generate a random password for the password field which is pretty handy.

   const databaseCredentialsSecret = new secretsManager.Secret(this, 'DBCredentialsSecret', {
      secretName: `${serviceName}-${stage}-credentials`,
      generateSecretString: {
        secretStringTemplate: JSON.stringify({
          username: databaseUsername,
        }),
        excludePunctuation: true,
        includeSpace: false,
        generateStringKey: 'password'
      }
    });

    new ssm.StringParameter(this, 'DBCredentialsArn', {
      parameterName: `${serviceName}-${stage}-credentials-arn`,
      stringValue: databaseCredentialsSecret.secretArn,
    });

Aurora instance configuration

The main configuration for the Aurora instance takes mostly from the RDS cdk package where we can define explicit cloud formation resources. In this case we need to define a RDS Cluster and pass in the options to turn it into an Aurora Serverless instance.

There are a couple of essential options to pass including engineMode, enableHttpEndpoint and the scalingConfiguration.

When choosing between the underlying database engine you can either use Postgres or SQL, the default is SQL.

I've set a different scaling configuration depending on the deployment stage, in the case of the development is makes sense to keep the maxCapacity to a low number and higher for production.

  const dbConfig = {
      dbClusterIdentifier: `main-${serviceName}-${stage}-cluster`,
      engineMode: 'serverless',
      engine: 'aurora-postgresql',
      engineVersion: '10.7',
      enableHttpEndpoint: true,
      databaseName: 'main',
      masterUsername: databaseCredentialsSecret.secretValueFromJson('username').toString(),
      masterUserPassword: databaseCredentialsSecret.secretValueFromJson('password'),
      backupRetentionPeriod: isDev ? 1 : 30,
      finalSnapshotIdentifier: `main-${serviceName}-${stage}-snapshot`,
      scalingConfiguration: {
        autoPause: true,
        maxCapacity: isDev ? 4 : 384,
        minCapacity: 2,
        secondsUntilAutoPause: isDev ? 3600 : 10800,
      }
    };


   const isDev = stage !== "production";
   const rdsCluster = new rds.CfnDBCluster(this, 'DBCluster', { dbConfig,
      deletionProtection: isDev ? false : true,
    });

Resource ARN

The CFNCluster does not output the Resource ARN unlike other CDK constructors so we need to construct it manually.

We can then store the ARN in SSM parameter store which can be used alongside the authentication credentials ARN to execute queries through the relevant Aurora SDK.

const dbClusterArn = `arn:aws:rds:${this.region}:${this.account}:cluster:${rdsCluster.ref}`;

new ssm.StringParameter(this, 'DBResourceArn', {
  parameterName: `${serviceName}-${stage}-resource-arn`,
  stringValue: dbClusterArn,
});

Get mentored by me and succeed with serverless: https://bit.ly/2P4IgHu

A simple lambda development environment with Docker and Localstack

Cameron Jenkinson — Sat, 01 Feb 2020 09:19:30 +0000

In this article you will learn how to spin up a development environment for a simple severless app.

A Lambda would be evoked on S3 put events which would then stream and parse XML files to JSON before inserting the result into a DynamoDB table.

The first pain point I had was understanding how to interface with AWS services locally which involved setting up a AWS environment locally with localstack.

Localstack provides an easy-to-use test/mocking framework for developing applications on AWS. It provides emulation of the most popular AWS services locally which can be accessed with the same SDK's provided by AWS such as the Node aws-sdk.

Getting all the services running together was the next challenge and Docker was the next step for creating a self contained environment.

I wanted it to be fairly easy for another developer to spin up and work on the function without having to ask me for credentials or how to install Localstack.

The following will go over the configuration files I used to make this work with Localstack and Docker for a faily harmonious development environment for AWS services.

Setting up the Docker container

Set up a boilerplate Node project using NPM or Yarn then create a Dockerfile.

yarn init

Run through the Yarn prompts then

touch Dockerfile

Dockerfile:

FROM node:8.4.0

RUN apt-get update

# Official AWS documentation recommends using python3 and associated tooling.That doesn't work, or at least it does not work as easily as advertised.
RUN apt-get install python-dev python-pip -y

# The awsebcli has a dependency issue and this resolves it
RUN easy_install --upgrade six

RUN pip install awscli

WORKDIR /usr/src/app
COPY package.json yarn.lock /usr/src/app/

RUN yarn

COPY . /usr/src/app

CMD ["yarn", "dev"]

A Dockerfile is used to build images from the instructions laid out using commands where those commands are run sequentially from this file.

In our Dockerfile we start by installing Node and the development pain free version of python and pip to avoid errors when working with the aws-cli. Once the aws-cli is installed, the working directory is set where the Node dependencies are added and the project script commands can be run.

We'll define the same working directory path in the docker-compose.ym file next.

Docker-compose.yml

version: '3'
services:
  lambda-parser:
    build: .
    volumes:
      - .:/usr/src/app
      - /usr/src/app/node_modules
    environment:
      - AWS_ACCESS_KEY_ID=foobar
      - AWS_SECRET_ACCESS_KEY=foobar
      - AWS_DEFAULT_REGION=us-east-1
      - AWS_S3_ENDPOINT=http://lambda-parser-aws:4572
      - AWS_S3_PATH_STYLE=true
      - AWS_DDB_ENDPOINT=http://lambda-parser-aws:4569
    depends_on:
      - lambda-parser-aws
    command: sh -c "yarn && yarn dev"
  lambda-parser-aws:
    image: localstack/localstack:0.8.7
    ports:
      - "5000:8080"
      - "4572:4572"
      - "4569:4569"
    expose:
      - "4572"
      - "4569"
    environment:
      - SERVICES=s3,dynamodb
      - HOSTNAME=lambda-parser-aws

A Docker Compose file is a YAML file defining services, networks and volumes.

The service definition contains a configuration that is applied to each container started for that service.

We will define two service configurations, the lambda-parser and the localstack service as lambda-parser-aws.

The lambda-parser service represents the Node.js project that the lambda will be developed in. It will interact with the localstack container over a default network created automatically by docker-compose.
The lambda-parser-aws service will expose the localstack instance through the defined port 5000 and every other service we define on their respective ports listed here.

We'll need to tell localstack to expose the S3 & DynamoDB services on ports 4572 and 4569.

If you'd like to add additional services such as SQS simply add them to the SERVICES=s3,dynamodb,sqs and expose the port defined from the localstack documentation.

Makefile

.PHONY: up down reboot help

## Run the service and watch for changes
up: docker-clean-images
    docker-compose up

## Shut down the service and any associated volume
down:
    docker-compose down --volumes

## Start from scratch again
reboot: down up

docker-clean-images:
    docker image prune --force --filter "until=24h"

docker-clean-volumes:
    docker volume prune --force

docker-nuke:
    docker system prune --force --all

## Run a yarn command inside the container
%:
    docker-compose exec ern-processor yarn $@

A Makefile is a special file, containing shell commands executed on the terminal.

In our Makefile we want to be able to use Docker Compose to spin up all of the services we defined in the docker-compose.yml file which will also be in charge of running the Node script yarn dev from the Dockerfile.

I think Makefiles are great for this type of problem because it gives you a single access point for interacting with your entire application stack.

make down will spin down the Docker containers cleaning up resources and make reboot will restart all containers after spinning them down.

Setting up aws-cli shell scripts

const shell = require('shelljs');

// S3
shell.echo('Creating s3 bucket and uploading ingest...');

shell.exec('aws --endpoint-url=http://lamda-parsar-aws:4572 s3 mb s3://my-bucket');
shell.exec('aws --endpoint-url=http://lamda-parsar-aws:4572 s3 sync ingest s3://my-bucket --acl public-read');
shell.exec('aws --endpoint-url=http://lamda-parsar-aws:4572 s3api get-object-acl --bucket my-bucket --key my-xml-file.xml');

// Dynamodb
shell.echo('Creating DynamoDB table...');
shell.exec('aws --endpoint-url=http://lambda-parsar-aws:4569  dynamodb create-table \
          --table-name XmlToJson \
          --attribute-definitions \
              AttributeName=ID,AttributeType=S \
          --key-schema AttributeName=ID,KeyType=HASH \
          --provisioned-throughput ReadCapacityUnits=1,WriteCapacityUnits=1');

shell.echo('Bootstrap complete');

At this point you might be wondering how we work with the AWS services on localstack, how do I actually create a Bucket and DynamoDB table.

You have a couple of options:

write scripts that utilise the AWS Sdk to provision the services you need and seed them with data
use the aws-cli to provision the services and seed them with data

Using shelljs npm package we are going to work with the second option. Essentially you'll define terminal commands that will be executed in order using the aws-cli inside the lambda-parsar Docker container.

I've added a couple of example scripts that create an S3 bucket, upload some data and change the permissions on the created object to public. Thenwe create the DynamoDB table and define the key schema as 'ID' which will be the primary hash key used to look-up entities.

The purpose of this bootstrapping file is to simulate what your DevOps flow would look like when you actually create the services on AWS.

Doing it from within the Node project allowed me to quickly create the instances and configuration I needed to focus solely on the business logic of the Lambda itself.

Wrapping Up

From here you have most of the boilerplate needed to start developing with AWS services locally without needing to work with any live instances on the cloud meaning less headache on credential management and permissions.

Get mentored by me and succeed with serverless: https://bit.ly/2P4IgHu

How async/await works in Array reduce

Cameron Jenkinson — Sat, 01 Feb 2020 09:07:58 +0000

I found myself Googling this question a few too many times and I'd like to talk about it.

Making asynchronous calls on a collection within Reduce has a few caveats that other Array iteration methods like .map don't come with.

Let's walk through a common example

Problem

I need to make a call to a database to retrieve information to build up a new collection with that information. The call to the database is asynchronous therefore I need to await it before continuing with execution.

My example is a real scenario I worked on recently. I had a collection of media elements that each had music associated with it. The music associated with was a list of tracks that would be played on this media element when it was streamed like a playlist.

The information needed to be organised into a usage report so we could track which tracks were played when a media element was streamed.

In order to build this I had to reach out to multiple tables to get all the track information and merge them into a collection to be used for reporting.

The context is not that important but what we do inside reduce that is.

Solution

export const getTrackDetails = async ({
    db,
    lessonMediaByVendorKey
}) => Object.entries(lessonMediaByVendorKey).reduce(async (prev, [key, vendor]) => {
    // 1. Get the track information from the music associated with the media element
    // 2. Add the track to the collection for usage reporting
}, []);

Get the track information from the music associated with the media element

const getTrackFromMusicInfo = async ({
    db,
    lessonMediaByVendor
}) => lessonMediaByVendor.reduce(async (prev, lessonMedia) => {
    // 1. Get all the track information from the tables
    // 2. Merge the track information together
}, {});

The first asynchronous call is to another function called getTrackFromMusicInfo which reaches out to the two tables to gather the information for the track played.

const getTrackFromMusicInfo = async ({
    db,
    lessonMediaByVendor
}) => lessonMediaByVendor.reduce(async (track, lessonMedia) => {
    // Wait, why do we await the accumulator?
    const acc = await track;

    const { music_id: musicId } = lessonMedia;

    const musicInfo = await db
        .select('*')
        .from('workout.music_tracks')
        .where('music_id', musicId)
        .first();

    const { track_info_id: trackId } = musicInfo;

    const trackInfo = await db
        .select('*')
        .from('workout.track_info')
        .where('id', trackId)
        .first();

    return {
      ...track,
      trackInfo,
    };
}, Promise.resolve({}));

So this is the first lesson to be learned on async/await inside Reduce and that is the accumulator is always a Promise.

It is a Promise based on the previous execution.

That means we need to resolve it after each iteration to actually get to the track returned inside the function from the DB calls.

That is why we make the initial value of the reduce to be a fake resolved value in order to keep resolving the promises returned from each iteration.

This is the main thing when using async/await inside reduce. We have to await the previous accumulator value in order to actually get to the accumulator otherwise it would return a Promise and you would not be able to resolve it where it was called.

Add the track to a collection for each media

export const getTrackDetails = async ({
    db,
    lessonMediaByVendorKey
}) => Object.entries(lessonMediaByVendorKey).reduce(async (prev, [key, vendor]) => {

    const acc = await prev;

    const { name, lessonMediaByVendor } = vendor;

    const track = await getTrackFromMusicInfo({ db, lessonMediaByVendor });

    const tracks = (acc[key] && acc[key]) || [];

    tracks.push(track);

    return acc.concat({
        key,
        name,
        tracks,
    });
}, Promise.resolve([]));

Now we make the async calls to get the track information for each media element with getTrackFromMusicInfo which returns the track object when it is resolved within its own execution context.

Failing to await the Promise returned by the accumulator inside the call would result in the getTrackFromMusicInfo returning a Promise object.

Wrapping up

The initial value of the Reduce method should be a dummy promise to resolve the accumulator value like an Object {} or Array []
Always await the previous call (the accumulator) at the top of the reduce function to get to the accumulator value

How to easily create nested CDK Stacks with addDependency

Cameron Jenkinson — Fri, 31 Jan 2020 08:24:42 +0000

At some point in your infrastructure build out with CDK you'll want to group services together into parent and child stacks.

For instance we have a set of core stacks like networking, database and iAM alongside a group of application services like auth, users and movies.

Taking on from the movies example we often build serverless applications as a group of lambda functions that expose different interfaces.

A classic example is to build a set of lambda functions that allow for CRUD operations on a movie entity over REST.

We have a create, delete, update and get handler all outlined in our application code but when it comes to creating the infrastructure stack we end with a rather large configuration:

const path = require('path')
const apigateway = require('@aws-cdk/aws-apigateway')
const lambda = require('@aws-cdk/aws-lambda')
const cdk = require('@aws-cdk/core')

class MovieService extends cdk.Stack {
    constructor(app, id, { serviceName, stage, config }) {
        super(app, id)

        /** MovieService - Handlers */
        const functionName = `movie-app-${stage}-${serviceName}`
        const createMovieLambda = new lambda.Function(this, 'CreateMovieLambda', {
            functionName,
            code: lambda.Code.fromAsset(
                path.resolve(__dirname, `../../../build/${serviceName}.zip`)
            ),
            handler: `${serviceName}.createMovieHandlerLambda`,
            runtime: lambda.Runtime.NODEJS_10_X,
            memorySize: 512,
            timeout: cdk.Duration.seconds(60),

            environment: {
                NODE_ENV: process.env.NODE_ENV,
                STAGE: stage,
            },
        })

        const functionName = `movie-app-${stage}-${serviceName}`
        const updateMovieLambda = new lambda.Function(this, 'UpdateMovieLambda', {
            functionName,
            code: lambda.Code.fromAsset(
                path.resolve(__dirname, `../../../build/${serviceName}.zip`)
            ),
            handler: `${serviceName}.updateMovieHandlerLambda`,
            runtime: lambda.Runtime.NODEJS_10_X,
            memorySize: 512,
            timeout: cdk.Duration.seconds(60),

            environment: {
                NODE_ENV: process.env.NODE_ENV,
                STAGE: stage,
            },
        })

        const getMovieLambda = new lambda.Function(this, 'GetMovieLambda', {
          functionName,
          code: lambda.Code.fromAsset(
              path.resolve(__dirname, `../../../build/${serviceName}.zip`)
          ),
          handler: `${serviceName}.getMovieHandlerLambda`,
          runtime: lambda.Runtime.NODEJS_10_X,
          memorySize: 512,
          timeout: cdk.Duration.seconds(60),

          environment: {
              NODE_ENV: process.env.NODE_ENV,
              STAGE: stage,
          },
        })

        const deleteMovieLambda = new lambda.Function(this, 'DeleteMovieLambda', {
          functionName,
          code: lambda.Code.fromAsset(
              path.resolve(__dirname, `../../../build/${serviceName}.zip`)
          ),
          handler: `${serviceName}.deleteMovieHandlerLambda`,
          runtime: lambda.Runtime.NODEJS_10_X,
          memorySize: 512,
          timeout: cdk.Duration.seconds(60),

          environment: {
              NODE_ENV: process.env.NODE_ENV,
              STAGE: stage,
          },
        })

        /** MovieService - API Gateway */
        const apiGatewayName = `movie-app-${stage}-${serviceName}-gateway`
        const movieApi = new apigateway.RestApi(this, 'ApiGatewayForMovieService', {
            restApiName: apiGatewayName,
        })

        const createMovieLambdaIntegration = new apigateway.LambdaIntegration(
            createMovieLambda
        )
        movieApi.root.addMethod('POST', createMovieLambdaIntegration)

        const updateeMovieLambdaIntegration = new apigateway.LambdaIntegration(
          updateMovieLambda
        )
        movieApi.root.addMethod('POST', updateMovieLambdaIntegration)

        const getMovieLambdaIntegration = new apigateway.LambdaIntegration(
          getMovieLambda
        )
        movieApi.root.addMethod('GET', getMovieLambdaIntegration)

        const deleteMovieLambdaIntegration = new apigateway.LambdaIntegration(
          deleteMovieLambda
        )
        movieApi.root.addMethod('DELETE', deleteMovieLambdaIntegration)
    }
}

module.exports = { MovieService }

If we started adding further dependencies for each handler and more configuration for the Movies API Gateway things quickly start to get out of control.

Add Dependency

stack.addDependency(stack) – Can be used to explicitly define dependency order between two stacks. This order is respected by the cdk deploy command when deploying multiple stacks at once.

Add dependency is a great way to solve this by making it easy to split up the stack configuration into parent and child stacks.

We can define child stacks for each lambda configuration (and for any configuration you'd like to split out)

const path = require('path')
const lambda = require('@aws-cdk/aws-lambda')
const cdk = require('@aws-cdk/core')

class CreateMovieLambda extends cdk.Stack {
    constructor(app, id, { serviceName, stage, config }) {
        super(app, id)

        const functionName = `movie-app-${stage}-${serviceName}`
        const createMovieLambda = new lambda.Function(this, 'CreateMovieLambda', {
            functionName,
            code: lambda.Code.fromAsset(
                path.resolve(__dirname, `../../../build/${serviceName}.zip`)
            ),
            handler: `${serviceName}.createMovieHandlerLambda`,
            runtime: lambda.Runtime.NODEJS_10_X,
            memorySize: 512,
            timeout: cdk.Duration.seconds(60),

            environment: {
                NODE_ENV: process.env.NODE_ENV,
                STAGE: stage,
            },
        })
    }
}

module.exports = { CreateMovieLambda }

We can not import this stack and add it as a dependency to the MovieService.

const path = require('path')
const apigateway = require('@aws-cdk/aws-apigateway')
const lambda = require('@aws-cdk/aws-lambda')
const cdk = require('@aws-cdk/core')

const { CreateMovieLambda } = require('./create-movie-lambda');

class MovieService extends cdk.Stack {
    constructor(app, id, { serviceName, stage, config }) {
        super(app, id)

        const createMovieLambda = new CreateMovieLambda(app, 'CreateMovieLambda', { config });

        this.addDependency(createMovieLambda);

        const apiGatewayName = `movie-app-${stage}-${serviceName}-gateway`
        const movieApi = new apigateway.RestApi(this, 'ApiGatewayForMovieService', {
            restApiName: apiGatewayName,
        })
        const createMovieLambdaIntegration = new apigateway.LambdaIntegration(
            createMovieLambda
        )
        movieApi.root.addMethod('POST', createMovieLambdaIntegration);

    }
}

module.exports = { MovieService }

This is a much cleaner way to organise and create stacks for related services.

CDK will deploy all dependencies in order of assignment so everything will be in sync.