DEV Community: Gerald Stewart

Migrating 85 Million DynamoDB Records Across AWS Accounts

Gerald Stewart — Mon, 27 Mar 2023 15:33:28 +0000

As part of a merger with another team, my team migrated 8 DynamoDB tables from one AWS account to another.

This was a challenging process, in this blog I'll be covering everything the AWS documentation doesn't tell you to look out for when migrating DynamoDB table across AWS Accounts.

Design

Initially, we were investigating the use of AWS Glue following this blog. AWS Glue was a service we were unfamiliar with and it would have resulted in us needing to write a different Python Glue script for each table. We ruled this our as a potential option based on this.

During our investigation into a suitable approach for this work, AWS released the new import to S3 functionality, we were able to leverage this alongside a table export to S3 to complete the data move.

While AWS gives us the steps needed to complete this move using the console, we are typically unable to do this due to restricted access to anything above our development environment. Plus the steps outlined in the blog show creating a table through the console, hardly a suitable process for any real-life project.

We came up with the following process to complete this work that would work with the level of access we could get:

Elevate into the IAM role with permissions to trigger an S3 export and write to the target account data transfer bucket. Permissions required can be found here.
Trigger export, specifying target account ID and bucket name.
Once complete this will generate a new folder in the data transfer bucket. This will look like AWSDynamoDB/<id>/data with being a generated ID.
This path needs to be specified in the import parameters used to trigger the import and create the new table.
After adding this ID to your code, begin deployment of the new database, this will trigger an import as part of the deployment. Eventually, you will have a created table with the data present.

Full credit to Elias Brange for this fantastic blog post on how to trigger imports and table creation using the AWS CDK.

The CloudFormation documentation for ImportSourceSpecification has now been released and is considered stable.

Challenges

Zero Feedback on Progress

After triggering an import or export, the AWS Console will just say "importing". One of the tables that we imported had 55 million rows and took nearly 1 hour and 30 minutes to import. All while we had a production outage to facilitate the data migration.

Some indication of progress or estimated time remaining would instil more confidence in this process.

In addition to this, the import time varied widely and there was no way to speed it up. As we took a considerable production outage to migrate the data we ideally would have wanted a faster process for larger tables.

DynamoDB Global Tables

As part of the move, we switched to global tables as we look to enable multi-region for our applications.

One issue we had was a deployment of a table failed to complete as the waitForReplicationToFinish property is defaulted to true in the AWS CDK. This deployment failed because the custom resource time out is limited to a maximum of 60 minutes.

This resulted in this error:

CloudFormation did not receive a response from your Custom Resource. Please check your logs for requestId [*****].

This resulted in a stack being stuck in a DELETE_FAILED state. As the source table could not be deleted as it had acted as a replica provider in the last 24 hours.

Resource handler returned message: "Replica cannot be deleted because it has acted as a source region for new replica(s) being added to the table in the last 24 hours.

As the actual table wasn't deleted, we were still able to read and write data from the table in a DELETE_FAILED state.

To repair this stack we had to export the new table to S3, delete the CloudFormation stack and re-import the data from S3 with the waitForReplicationToFinish property set to get the stack in a healthy state.

This may have been a user error but should serve as good learning for anyone switching from a standard DynamoDB table to a global table.

Manual Process

This was a very manual process to complete, compared with using AWS DMS to migrate data for other AWS Database offerings. It required a lot of pre-work, and investigation and wasn't a quick process to complete in production.

I identified 9 steps that we needed to take for each migration:

Deploy the application branch to shut off writes to the source database
Verify this has been completed before proceeding
Trigger export of source database
Add import ID to new database branch
Verify export has been completed before proceeding
Deploy the new database branch with import ID
Verify import completes
Revert application branch to resume writes to the database
Verify new writes happening to DB

I would ideally have liked a managed solution from AWS to handle this migration without downtime or as much of a manual process.

Redundant CloudFormation Properties

To trigger the imports, we used CDK escape hatches to specify ImportSourceSpecification when creating our DynamoDB table.

cfnTable.addPropertyOverride('ImportSourceSpecification.S3BucketSource.S3Bucket', bucketName);
cfnTable.addPropertyOverride('ImportSourceSpecification.S3BucketSource.S3KeyPrefix', bucketPath);
cfnTable.addPropertyOverride('ImportSourceSpecification.InputCompressionType', 'GZIP');
cfnTable.addPropertyOverride('ImportSourceSpecification.InputFormat', 'DYNAMODB_JSON');

As per the CloudFormation documentation for DynamoDB tables the update policy of this is Replacement.

Ideally, we would have liked to remove this property from our CDK stacks after that initial table creation and successful completion of the import. We have a large number of tables and want to keep our code neat and tidy.

It's definitely introduced a bit of technical debt to these projects as if someone inadvertently removes these properties it will break the deployment of the table. We have a warning comment above these lines in every project, but again this is a far from optimal solution.

Conclusion

If you are looking to migrate data, the export to S3 functionality is a good solution. Just be extra cautious of the things I've outlined in this post.

I'd also recommend outlining a plan to "abort" a migration once you have started it if anything goes wrong. This process has a lot of moving parts and a lot of potential failure points.

I'd also love if AWS solved some of the issues outlined in this post, it wasn't the best experience compared to if I had of been migrating a DB supported by DMS. Always felt weird that DynamoDB wasn't supported by this service.

Have you migrated DynamoDB data in a different way than I have, if so I'd love to hear about your approach in the comments!

Improving Operational Excellence using Game Days

Gerald Stewart — Tue, 24 Jan 2023 11:43:48 +0000

A game day is a practice event that simulates a failure or outage so that you can test your systems, your processes and your team's response to remediate or act on that particular event.

AWS defines operational excellence as

The Operational Excellence pillar includes the ability to support the development and run workloads effectively, gain insight into their operations, and continuously improve supporting processes and procedures to deliver business value.

To me, game days go hand-in-hand with the Operational Excellence pillar of the Well-Architected framework. They are even explicitly called out under the anticipate failure design principle.

I ran a few game-day scenarios earlier this year with my team to get them comfortable with the prospect of providing 24/7 on-call support for our applications.

However, I found it difficult to identify scenarios and even harder to accurately reproduce them in a not-so-obvious way.

At re:Invent this year, my favourite session of the week was on this exact topic. It wasn't recorded, so I'd like to share with you all what I took away from that session in this blog.

Examples of Game Day Events/Types

Security Incidents
Someone leaving the company/team
Infrastructure outage
Application outage

When should I run a Game Day?

Regular Cadence
Game days should be run regularly to help your team develop a muscle memory for incident response. Your company likely runs regular safety drills to ensure employees know what to do in the event of a fire, this is no different.

Architectural Changes
If you make substantial architecture or design changes to your application, it might also be worth running a game day to ensure all team members are aware of the changing points-of-failure to be aware of in applications.

Team Changes
Someone joining or leaving a team can have a big impact on the team's ability to resolve. Often team members become a "crutch" in these types of scenarios. If your star incident resolver leaves you to need to ensure the team is capable and confident in supporting your application.

The same can be said about a new joiner. Getting them up to speed quickly by running game days in a controlled environment is a great way to build confidence and should be considered part of the onboarding process for any team supporting applications in production.

Building Effective Game Days

Part 1: How can someone triaging this scenario effectively detect that it is happening?

In order for a scenario to be considered valid, there must be an effective way of detecting that something is actually happening. Let's look at an example:

Scenario: A Lambda function invoked through an API Gateway is producing a 5XX response code
In this scenario, an effective troubleshooting method may be alerting for 5XX errors at an API Gateway level.

If you are identifying scenarios and struggling to find effective methods of detecting them occurring, you should consider creating monitoring or processes to enable the detection of these.

Some more examples:

IP Address exhaustion - Monitored using CloudWatch alarm
Denial of Service - Abnormal traffic pattern alarm
Lambda timeouts - Alerting off CloudWatch metric

Part 2: How can someone resolve this scenario in an effective and efficient manner?

After identifying a scenario, we need to make sure that the team has the relevant skills and information to resolve the scenario in an effective and efficient manner. This can be as simple as knowing where to look at the status of alerts or ensuring they have the correct permissions to view logs.

For more specific application-level errors runbooks can be useful to understand the meaning behind error messages, or knowing who to page at 3 am at when it's a pesky downstream service causing the error.

Scenario: A team member is called for an out-of-hours system outage
Upon receiving a call about a system outage, the team member knows the location of the alerting and runbook so that they can effectively respond to the presented issue. They can resolve this on their own without the need to escalate further.

If gaps have been identified here, before proceeding with the game day scenario make sure the necessary documentation exists and the entire team is aware of it

Part 3: As a "game master" how can this scenario be reproduced in a simulated and controlled environment?

In order to trigger a scenario, it needs to be able to be reproducible.

Let's look at some examples and how they can be reproduced:

EC2 instance failure - Trigged using AWS Fault Injection Simulator
Denial of Service - Triggered using Artillery
Lambda Timeouts - Downstream service timeout can be mocked using mock-server

This is often the piece I find most challenging, you may need to get creative. Currently, AWS FIS only supports EC2, ECS, EKS and RDS. As someone who works mostly with AWS-managed serverless services, I would love to see support for Lambda, API Gateway, CloudFront and DynamoDB so that this is less of a headache!

Part 4: What is the category/classification of this scenario?

Scenarios can be broadly grouped into two wide categories:

AWS outages/issues
Application level outages/issues

It's a good idea to run a mix of both types of scenarios so that team members understand troubleshooting, resolutions or escalation paths for both types.

Part 5: What is the estimated time a trained person should resolve this scenario in?

We should assign each scenario two time values:

Expected resolution time: The expected time a trained individual should resolve this issue in.
Maximum run time: The maximum amount of time the scenario is allowed to run for.

The expected resolution time is a rough number, that should be adjusted if a repeat of a particular scenario is either too generous or too harsh of a time frame. It serves as an indicator that the training, processes and documentation available are sufficient to meet the demands of supporting the application.

The maximum run time should be given enough buffer room to allow someone ample time to attempt to resolve a scenario. We assign this value to avoid creating frustration with the exercise going on for an endless amount of time. If certain scenarios are hitting this regularly this should be re-assessed.

Example Game Day Plan

Let's take a look at this reference architecture to identify some potential game-day scenarios. This diagram depicts a CloudFront distribution, with a Route 53 domain and ACM certificate. An API gateway with two Lambda functions, one making HTTP calls to an external service and one accessing an RDS instance.

The diagram above depicts some example game day scenarios in red. I left some blank to give you (the reader) a chance to think of some potential scenarios.

Let's break down some of the identified scenarios for this architecture:

Scenario	Detection Method	Reproduction Steps	Classification	Resolution Time/Max Runtime
Misconfigured Route 53 record	Traffic anomaly detection	Facilitator alters configuration	Application Level	5/10
Expiring ACM certificate	Certificate expiry alert	-	Application Level	10/20
Third party service outage	Lambda error alerting	Mock Server	Application Level	10/20

Scenario 1: A misconfigured Route 53 record. This is a pretty common occurrence when making DNS changes. It would be a user error that causes this type of outage. To reproduce this whoever is running the scenario could go in and make a change. The goal here is to troubleshoot and revert the change in a timely manner.

Scenario 2: An expiring ACM certificate. AWS will notify you of this well in advance. Simulating this might be difficult as certificate renewals are usually on a yearly basis. For this, it could be a "made-up" scenario, that you walk through the steps taken to remediate the issue.

Scenario 3: A third-party service outage. This could be simulated with a mock server. A mock outage could be simulated. In this situation identifying the correct downstream service and mocking out contacting the team that owns the service would be a good outcome.

I have intentionally left some boxes blank if you want to think about this architecture and what other possible application or AWS-level outages could occur. I'd love to hear about them in the comments.

Conclusion

You've run your game day, what's next you might ask?

I would recommend collecting feedback on:

Tooling and processes
Education and knowledge sharing
Are there any application-level changes that could help mitigate this scenario?
The overall running of the game day

I would expect the following outputs of a successful game day:

Increased team confidence
Improvements in documentation
Improvements in alerting
Potential architecture changes to make applications more resilient

The entire process of game days is centred around continuous improvement. Bulletproof your plans, and increase your engineer's confidence and knowledge of your systems.

If you found any of this useful please let me know, make sure to attempt to identify the extra scenarios in our example architecture and leave a comment with how you would conduct a game day for them!

How Well-Architected Enables Junior Engineers

Gerald Stewart — Wed, 24 Nov 2021 16:55:06 +0000

What is Well-Architected 🤔

Well-Architected describes the key concepts, design principles and architecture best practices for designing your cloud workloads. It balances best practices with business goals to determine the optimal outcome.

Created by AWS Solutions Architects using the lessons Amazon has learnt from running thousands of systems at a massive scale, it enables developers to compare their workloads against the rigorous standards that AWS holds itself against.

Pillars 🏛

Credit to Jerry Hargrove

Operational Excellence - To be able to monitor and support workloads effectively, to enable continuous improvement and to deliver business value.
Security - Improving your security posture by taking advantage of Cloud technologies to protect your assets and systems.
Reliability - Ensuring your workload can perform it's intended function correctly and consistently.
Performance Efficiency - Ensuring appropriate resource allocation to enable performant systems and a positive end user experience.
Cost Optimisation - Ensuring you are delivering business value at the lowest possible cost.

Lenses 🔎

Lenses can be applied to a Well-Architected review to get additional specific questions targeting your type of application. Examples of lenses include Serverless, Machine Learning and SaaS. You can find more out about lenses here.

How to Apply Well-Architected 👨‍🔧

Well-Architected should be applied both continuously with reviews carried out at regular intervals. Understanding the pillars, core concepts and reasons behind the pillars of the Well-Architected framework can enable you to make better architecture decisions and avoid rewrites or large changes after a full-scale review has been carried out.

The AWS Console has a "Well-Architected Tool" which allows you to track the health of your workloads. This is a free service and will ask questions based on your workload type to come up with an improvement plan.

How Well-Architected Enabled Me as a Junior Engineer 🙋‍♂️

As a junior engineer, being involved in design discussions can be a daunting experience.

A few months ago, I took time to take a deep dive into the AWS Well-Architected framework. The knowledge I gained by doing this has been invaluable.

Well-Architected is a gold standard for best practices for building on AWS. It can also be applied to other types of applications.

Instead of forming opinions about how something should be done, referring to the Well-Architected framework can often yield an AWS backed approach.

The key differentiator between Well-Architected and self-formed opinions is that Well-Architected is the opinion of AWS Solutions Architects with many years of experience. My own self-formed opinions are only based on my experiences, which at the beginning of my career are limited.

Being able to point to documentation to back up your points is invaluable.

By following Well-Architected principles and applying them in everything I did I found myself over time becoming more involved in design discussions and more high-level work.

Well-Architected gave me the right questions to ask, to drive discussions so we could make better choices in the day-to-day development of our application.

When weighing up technical decisions, think Well-Architected:

Security - Is this as secure as it could be, what's data are we storing, who has access?
Cost - What's the potential cost impacts of this change, is it worth it, is the business aware?
Operational Excellence - What monitoring do we need for this, what metrics should we be looking at, what level of fault tolerance should we allow?
Performance - What will this do to performance, do we need to tweak/re-evaluate resource allocation as a result of this change?
Reliability - What happens if this fails, if a downstream system is down?

Asking these questions and driving conversations led to me progressing pretty quickly to becoming a Senior Software Engineer.

I had plenty of examples of how I identified improvements to our application that I identified by applying Well-Architected.

Conclusion

If I could recommend one topic of learning to a junior engineer or someone looking to up their game designing and building on AWS it would be to read the Well-Architected framework and understand how to apply it.

If you or your team applies Well-Architected I'd love to hear how it made a difference to your application in the comments!

Learning Materials & Links ⛓

AWS CDK: Per-Environment Configuration Patterns

Gerald Stewart — Wed, 26 May 2021 12:23:13 +0000

Introduction 👋

Often projects will have different configuration values for each deployed environment. This could be feature toggles, URLs for third-party services or any number of other variables.

With the AWS CDK, this is simple to configure. I have seen a few different approaches to this problem. In this blog I'll share a few suitable for use in TypeScript.

Method 1️⃣: Stack Configuration Function

This approach uses a mapper function to return the configuration, you can see we have a single configuration property defined in the EnvironmentConfig interface.

interface IEnvironmentConfig {
  readonly myEnvSpecificApiEndpoint: string;
}

const environmentConfig = (environment: string): IEnvironmentConfig => {
  const environmentMapper: {
    [environment: string]: {
      myEnvSpecificApiEndpoint: string;
    };
  } = {
    local: {
      myEnvSpecificApiEndpoint: 'https://dev.google.com/api',
    },
    test: {
      myEnvSpecificApiEndpoint: 'https://test.google.com/api',
    },
    production: {
      myEnvSpecificApiEndpoint: 'https://google.com/api',
    },
  };
  return environmentMapper[environmentName];
};

This function would be called from the stack like this, process.env.ENV_NAME would correspond to the environment name (replace this with your environment name variable for your chosen CI/CD pipeline) or default to local if undefined.

const environment: string = process.env.ENV_NAME || 'local';
const envConfig: IEnvironmentConfig = environmentConfig(environment);

You can then access the configuration like this:

const apiEndpoint: string = envConfig.myEnvSpecificApiEndpoint;

The apiEndpoint variable is now ready to be used in your stack.

Method 2️⃣: CDK Runtime Context

CDK Context values are key-value pairs that can be associated with a stack or construct. There are a number of different ways these values can be configured, for more information on that see the link above to the documentation.

In this example, I'm going to use cdk.context.json file in the root of a CDK project to configure a stack.

Here is an example cdk.context.json:

{
   "local":{
      "myEnvSpecificApiEndpoint":"https://dev.google.com/api"
   },
   "test":{
      "myEnvSpecificApiEndpoint":"https://test.google.com/api"
   },
   "production":{
      "myEnvSpecificApiEndpoint":"https://google.com/api"
   }
}

Interfaces can also be created to define the type of configuration data expected:

interface IEnvironmentConfig {
  readonly myEnvSpecificApiEndpoint: string;
}

These values can be accessed like this:

const environment: string = process.env.ENV_NAME || 'local';
const envConfig: IEnvironmentConfig = scope.node.tryGetContext(environment);
const apiEndpoint: string = envConfig.myEnvSpecificApiEndpoint;

Method 3️⃣: Extending StackProps

The AWS CDK StackProps interface can be extended to add additional configuration properties. In this example we will extend the AWS CDK interface to add a property called myEnvSpecificApiEndpoint.

interface IEnvironmentConfig extends StackProps {
  readonly myEnvSpecificApiEndpoint: string;
}

Now in the stack initialisation file (located in the bin directory) we can pass this in.

const app = new cdk.App();
new TheScheduledLambdaStack(app, 'TheScheduledLambdaStack',{
    myEnvSpecificApiEndpoint: 'https://dev.google.com/api'
});

Now, the one downfall of this is that you still need to implement something like method 1 or 2 to configure it on a per-environment basis. This would look something like this for method 1:

new TheScheduledLambdaStack(app, 'TheScheduledLambdaStack',{
    myEnvSpecificApiEndpoint: envConfig.myEnvSpecificApiEndpoint;
});

Conclusion 🤌

Looking at all three methods, I personally like method 2. Until recently I was blissfully unaware the CDK had already 'solved' this problem for us.

I live by the saying 'code is a liability' - the less code you manage the better.

Do you..

use any of these methods already?
have a better way of doing it?
have a different opinion on the optimal solution?

Let me know in the comments!

AWS CDK: Lambda Versioning

Gerald Stewart — Fri, 23 Apr 2021 11:34:32 +0000

Introduction 👋

I recently set out to investigate a few different ways we could version a Serverless API Gateway backed by Lambda functions. I've seen a few different blogs talking about this issue, but none in a CDK context.

In this blog I'll walk through the strategy I devised to be able to support multiple versions of a Lambda function fronted by an API Gateway, with code examples.

Disclaimer: This may not be the most optimal way to solve this issue, just my approach. If you have a better way I'd love to hear it!

The Problem 🤔

You are developing a API Gateway backed by Lambda functions and you need to be able to make a breaking change to your API without breaking your consumers. Giving consumers time to use your old API version while they make the necessary changes to move the newest version.

In this example, we do not care for making code changes to previous versions of the Lambda.

Lambda Versioning 🗄

Lambda supports versioning, by default if no are versions explicitly defined the $LATEST version will be overwritten. This is fine until you need to support two versions of a function simultaneously.

Lambda versioning is also briefly covered in the AWS Well-Architected Framework Serverless Lens.

Introducing my Serverless API versioning strategy:

The above allows for active development of the /v2/greeting endpoint while maintaining a active /v1/greeting endpoint buying consumers some time. In my case the end goal is to shut off /v1/greeting once consumers have made the necessary changes.

Lets take a look at how this looks and works using the AWS CDK!

Lambda Code 🖥

In this example, helloLambda is returning a JSON response, this is what it will return in v1 of the Lambda:



{
  "greeting": "Hello World!",
  "version": "v1"
}

The code below is from the CDK stack file:



const helloLambda = new Function(this, 'helloLambda', {
  runtime: Runtime.NODEJS_14_X,
  code: Code.fromAsset('lambda-fns'),
  handler: 'index.handler',
});

const helloLambdaV1 = new Version(this, ‘helloLambda-v1’, {
  lambda: helloLambda
});

const helloLambdaV1Alias = new Alias(this, ‘helloLambda-v1-alias’, {
  aliasName: 'helloLambda-v1',
  version: helloLambdaV1
});

Documentation: Function, Version, Alias.

This code snippet defines a new Lambda function helloLambda and creates a new version helloLambda-v1. An Alias is created helloLambda-v1-alias that is associated with the version helloLambda-v1.

Next we need to set up two API endpoints, going to different versions of our helloLambda.

API Gateway Code 🖥



const api = new RestApi(this, 'greeting-api');

const v1 = api.root.addResource('v1');
const v1Greeting = v1.addResource('greeting');
v1Greeting.addMethod('GET', new LambdaIntegration(helloLambdaV1Alias));

Documentation: RestApi, LambdaIntegration.

This leaves us with a /v1/greeting endpoint pointing to the alias helloLambdaV1Alias. The next step is to add a /v2/greeting endpoint allowing us to point to the $LATEST version of the function so we can continue to develop the newest version of our API.



const v2 = api.root.addResource('v2');
const v2Greeting = v2.addResource('greeting');
v2Greeting.addMethod('GET', new LambdaIntegration(helloLambda));

Documentation: RestApi, LambdaIntegration.

It's important to note, if you want to point to $LATEST do not define a new Lambda version. In the v2 code snippet above we reference helloLambda directly instead of referencing the version helloLambdaV1Alias.

Reference, Alias and Function are all accepted types for setting up a new LambdaIntegration.

As we now have /v2/greeting pointing to $LATEST version of the Lambda function, we can update the Lambda handler code to return a different hardcoded value:



{
  "greeting": "Hello World!",
  "version": "$LATEST"
}

Deploying this updated handler code would make it return "version": "$LATEST" when calling the /v2/greeting endpoint and return "version": "v1" when calling the /v1/greeting endpoint.

Summary 🤌

This pattern is good for supporting basic API versioning. I was also amazed at how easy this was to accomplish with the AWS CDK and felt I had to share it!

In order to clean up old Lambda versions you need to use the AWS CLI lambda delete-function command specifying a qualifier to target specific versions you no longer need.

Alternative Solution 🤷‍♂️

If you require backwards support for code changes I'm unaware of anything besides duplicating Lambda functions for each supported version that can accomplish that.

This has it's drawbacks, duplicating code is never good. However if you do need full backwards support it can still work.

If you've tried either of these strategies, or you have a better one I'd love to hear about it in the comments!