There are lots of tutorials of deploying basic CRUD applications with AWS CDK using API Gateway, and Lambda connecting to a DynamoDB table. The defacto starter set for a serverless application. You build and deploy, modify the hello-world application to match your needs, you're happy! The API is running and acting perfectly correctly in your default region.. us-east-1...
The Issues Begin
Then November 25th, 2020 strikes. The whole region is down for hours, your API is inaccessible, people are angry at YOU for this inconvenience. You read more about us-east-1 and realize that it tends to have a history of outages... and plan to make the jump to another region.
December rolls around and you've successfully migrated your application to us-west-2. It wasn't too difficult, your CDK app was able to tear down everything in us-east-1, and then deploy everything again in the new region. You let out the breath you'd been holding.
January 7th 2021 started off really well, and as you started looking forward to the end of the day... the alerts start ringing. Your new region, us-west-2, has begun to act up! After two hours of sweating, you are back up.
Multi-Region Solution
After all this pain and stress, you're determined not to be caught flat-footed again. While the goal of a multi-region setup seemed daunting, you roll up your sleeves and dive into it.
For our CRUD application, there are two components we need to deal with:
- Ensuring our data is consistent between the regions.
- That DNS always resolves to a region that is up.
Thankfully there are solutions to both of these issues in our existing tech stack.
| Tech | Single Region Solution | Multi Region Solution |
|---|---|---|
| Data | DynamoDB Table | DynamoDB Global Table |
| DNS | Route53 Simple Routing Policy | Route53 Latency Routing Policy |
In part 1, I'll dive into the complexities of implementing a multi-region data configuration.
DynamoDB Global Tables
The DynamoDB Global Tables are a managed solutions from AWS where they keep replicate data changes from one DynamoDB table to all related tables in other regions.
For simplicity of use, there can be a number of "gotchas" from an Infrastructure as Code perspective.
CDK Primary Region
While DynamoDB Global Tables are multi-master, multi-region, from an AWS CDK point of view, we deploy them in only a single region. The resource is configured to replicate to the other regions that you're interested in.
So I would instantiate my stack in us-west-2, and configure it to replicate to us-east-1, us-east-2, and us-west-1.
import * as dynamodb from '@aws-cdk/aws-dynamodb';
// Stack was called in us-west-2
const gloablTable = new dynamodb.Table(this, 'globalTable', {
partitionKey: {
name: 'id',
type: dynamodb.AttributeType.STRING
},
billingMode: dynamodb.BillingMode.PROVISIONED,
replicationRegions: ['us-east-1', 'us-east-2', 'us-west-1'],
removalPolicy: cdk.RemovalPolicy.DESTROY,
});
Since we want to reference our global table's name later on as environment variables for our Lambdas, we will want to build this out into two stacks.
This will make our app resemble the following:
const appRegions = ['us-east-1', 'us-east-2', 'us-west-1', 'us-west-2'];
const app = new cdk.App();
const globalstack = new GlobalStack(app,'DynamoDBGlobalStack', {env: {region: 'us-west-2'}});
appRegions.forEach(function (item, index) {
new AppStack(app, 'AppStack-'.concat(item), {
env: {account: process.env.CDK_DEFAULT_ACCOUNT, region: item},
globalTableName: globalstack.globalTable.tableName
});
}
Region References - Mocks
You might have noticed that I am passing only the table name to the app stacks, rather than the actual table object which would be best practice. This is because the actual table is specifically referencing our primary table in us-west-2. I'm afraid in our App stack we will need to either mock out the table OR use the AWS SDK to retrieve the full details.
For most use cases, simply mocking out the table should be all that you need.
import * as cdk from '@aws-cdk/core';
import * as dynamodb from '@aws-cdk/aws-dynamodb';
interface CustomStackProps extends cdk.StackProps {
readonly globalTableName: string;
readonly env: any;
}
export class AppStack extends cdk.Stack {
constructor(scope: cdk.Construct, id: string, props: CustomStackProps) {
super(scope, id, props);
const globalTable = new dynamodb.Table.fromTableName(this, 'globalTable', props.globalTableName);
We can reference the globalTable to grant IAM permissions etc.
Triggers
I did find that there is ONE instance where there was the need to use the AWS SDK, and that was around implementing triggers off the DynamoDB table. Global Tables are kept in sync by the use of DyanmoDB Streams, and AWS automatically names these streams in the following format:
arn:aws:dynamodb:REGION:AWS-ACCOUNT:table/TABLE-NAME/stream/2021-01-16T19:47:47.531
This timestamp is specific to each region, and so the only way to retrieve this information is to describe the specific region table.
...
constructor(scope: cdk.Construct, id: string, props: CustomStackProps) {
super(scope, id, props);
const client = new DynamoDB({ region: props.env.region });
// Query the regions table
let globalTableInfoRequest = async () => await client.describeTable({ TableName: props.globalTableName});
globalTableInfoRequest().then( globalTableInfoResult => {
// Mock the table with the specific ARN and Stream ARN
const globalTable = dynamodb.Table.fromTableAttributes(this, "globalTable", {
tableArn: globalTableInfoResult?.Table?.TableArn,
tableStreamArn: globalTableInfoResult?.Table?.LatestStreamArn
});
// Lambda
const triggerLambda = new lambda.Function(this, 'triggerLambda', {
...
environment: {
TABLE_NAME: props.globalTableName,
...
}
});
// Grant read access
globalTable.grantStreamRead(triggerLambda);
// Deadletter queue
const triggerDLQueue = new sqs.Queue(this, 'triggerDLQueue');
// Trigger Event
triggerLambda.addEventSource(new DynamoEventSource(globalTable, {
startingPosition: lambda.StartingPosition.TRIM_HORIZON,
batchSize: 5,
bisectBatchOnError: true,
onFailure: new SqsDlq(triggerDLQueue),
retryAttempts: 10
}));
});
}
}
Deployment
One specific issue we do run into with this design, is that the multiple app stacks which are deployed in multi regions are dependent on the single global stack that was deployed in one region. Cloudformation does not allow us to create a cross region dependency between stacks. We will want to deploy our GlobalStack first, and then we can deploy all stacks.
We want to synthesize the CDK to produce the Cloudformation template for us:
cdk synth GlobalStack
The we can depoy just the global stack:
cdk deploy GlobalStack
Once that is complete, we can generate all the cloudformation templates - the AWS SDK code is excuted during synthesizing so we need the Global tables deployed beforehand:
cdk synth
Finally we can deploy all stacks:
cdk deploy --all
Since we are performing this ordered deploy, to remember that to tear this down you will need to work in reverse order due to the dependencies that have been created. Tear down all the app stacks first, then tear down the global tables.
Regions Available
A final piece to consider is that DynamoDB Global tables in most, but NOT all regions - specifically the following regions do not support them at the time I write this:
| Region Name | Region Code |
|---|---|
| Africa (Cape Town) | af-south-1 |
| Asia Pacific (Hong Kong) | ap-east-1 |
| Europe (Milan) | eu-south-1 |
| Middle East (Bahrain) | me-south-1 |
These four regions are disabled by default, and you will need to enable them if you wanted to use them.
Conclusion
DynamoDB Global tables are a great method to quickly implement a multi-region, multi-master data solution that is managed by AWS. In 2019 the pricing model for Global tables was updated that removed the cost to replicate data between regions, which really elevated this into a great solution.
My next post will examine the DNS routing policy that we will want to implement to ensure users are not impacted by any region downtime in the future.
All code for this post can be accessed on GitHub
Top comments (0)