DEV Community: Volkan Paksoy

Migrating an EC2 Instance to a Different AWS Account

Volkan Paksoy — Wed, 18 Mar 2020 06:00:00 +0000

Recently I needed to migrate an EC2 instance to a different AWS account. There’s no built-in functionality to handle this. The solution is creating an AMI from the instance and sharing it with the target account. I scripted my solution with Powershell and AWS Tools for Powershell as below.

Prerequisites

Since the operation involves a source account and a target account, first create 2 profiles with EC2 access. credentials files should look like this

[source_profile]
aws_access_key_id = xxxxxxxxx
aws_secret_access_key = xxxxxxxxx

[target_profile]
aws_access_key_id = xxxxxxxxx
aws_secret_access_key = xxxxxxxxx

Step 00: Configuration

I put all the variables in a Powershell script file which I include in the following steps. To use the scripts you need to provide the values first. Hopefully the variable names are self-explanatory:

$sourceAccountAwsProfileName = ""
$sourceRegion = ""
$sourceAccountId = ""
$instanceId = ""
$amiName = ""
$amiDescription = ""
$targetAccountAwsProfileName = ""
$targetAccountId = ""
$targetRegion = ""

Step 01: Create AMI

First step to migrate EC2 instance is to create an AMi from the instance.

. .\"00. configuration.ps1"

# Create AMI
$imageId = New-EC2Image -InstanceId $instanceId -Name $amiName -Description $amiDescription -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion

Set-Variable -Scope global -Name AMI_ID -Value $imageId
Write-Host "AMI_ID: [" $AMI_ID "]"

This operation takes a few minutes. The image has to become available before we can proceed to the next step.

Step 02: Share AMI

Now the image is ready we have to share it with the target account. The script below shares the AMI and allows new volumes created from this AMI.

. .\"00. configuration.ps1"

$imageId = Get-Variable AMI_ID -valueOnly
Edit-EC2ImageAttribute -ImageId $imageId -Attribute launchPermission -OperationType add -UserId $targetAccountId -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion

$imageSnapshots = Get-EC2Snapshot -OwnerId $sourceAccountId -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion
                | Where-Object {$_.Description -like "*$imageId*" }

foreach ($snapshot in $imageSnapshots) {
    Edit-EC2SnapshotAttribute -SnapshotId $snapshot.SnapshotId -Attribute createVolumePermission -OperationType add -UserId $targetAccountId -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion
}

Step 03: Copy AMI

At this point, if you go to the target account you should be able to see the image when you choose “Private images” category. Make sure to choose the same region as the source account to be able to see the image.

We have access to this image but we want to have our own copy which we accomplish with the script below:

. .\"00. configuration.ps1"

$imageId = Get-Variable AMI_ID -valueOnly
Copy-EC2Image -SourceImageId $imageId -SourceRegion $sourceRegion -Name $amiName -ProfileName $targetAccountAwsProfileName -Region $targetRegion

In my experience the whole copying process took about 5 minutes.

Now that we have our own copy of the AMI we can launch instances as we please. Job (almost) done!

Step 04: Clean Up

Final step is to clean up after ourselves. Since this AMI was created to migrate to the new account only I assume we won’t need it anymore in the source account. The following script deregisters the AMI and deletes all the associated snapshots.

. .\"00. configuration.ps1"

# Create image script writes the AMI ID to a variable. If it doesn't exist get the image id from AWS Management Console
$imageId = Get-Variable AMI_ID -valueOnly

Write-Host "Unregistering image: [" $imageId "]"
Unregister-EC2Image -ImageId $imageId -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion

$imageSnapshots = Get-EC2Snapshot -OwnerId $sourceAccountId -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion
                | Where-Object {$_.Description -like "*$imageId*" }

foreach ($snapshot in $imageSnapshots) {
    Write-Host "Removing snapshot: [" $snapshot.SnapshotId "]"
    Remove-EC2Snapshot -SnapshotId $snapshot.SnapshotId -Force -ProfileName $sourceAccountAwsProfileName -Region $sourceRegion
}

# Delete variable
Remove-Variable AMI_ID -Scope global

Resources

Infrastructure as Code with AWS CloudFormation and Cloud Development Kit (CDK)

Volkan Paksoy — Sat, 26 Oct 2019 12:00:00 +0000

Introduction

Using AWS is great as it simplifies and improves infrastructure provisioning and maintenance quite a lot. As you depend more and more on AWS you quickly realize that managing everything through AWS Management Console is not an ideal way.

Earlier this year, I published this post about AWS CloudFormation. This post will also discuss using CloudFormation but by taking it a higher level by using AWS Cloud Development Kit (CDK).

Levels of infrastructure

I liked the way the several approaches of infrastructure management is illustrated as levels here: Develop a Web App Using Amazon ECS and AWS Cloud Development Kit (CDK) - AWS Online Tech Talks (YouTube Video)

In a nutshell, those levels are:

Level 0: By hand

This approach is simply using AWS Management Console user interface to manage the infrastructure.

Pros:

Simple and easy
Helps to get results faster for exploratory projects

Cons:

Hard to reproduce
Possible inconsistencies based on people’s preferences
Error-prone
Slow for complex systems

Level 1: Imperative Infrastructure as Code

In this approach you write your own scripts using AWS SDK and manage the resources programmatically.

Pros:

Repeatable and reusable
Can be source-controlled

Cons:

Lots of boilerplate code
It needs to address all edge cases

Level 2: Declarative Infrastructure as Code

Describe the infrastructure as a script (in JSON or YAML) and use an resource provisioning engine such as AWS CloudFormation or HashiCorp Terraform which in turn use AWS SDK to manage the infrastructure.

Pros:

No boilerplate
Creating and updating resources is handled automatically

Cons:

Templates can become verbose
Implementing logic is limited to some built-in helper functions

Level 3: AWS Cloud Development Kit

In this approach software is developed using AWS CDK which generates the input for AWS CloudFormation.

The application can be developed in a number of languages. At the time of this writing the following languages are supported: TypeScript, JavaScript, Python, Java, C#.

Pros:

Handles creation of underlying resources. For example, when creating a VPC it also automatically generates YAML for all other networking resources (routing tables, NAT gateways etc) that are required by VPC.
Helps with local workflow
CDK constructs are reusable. Can be developed by AWS or third parties and can be installed separately.
Ability to use familiar programming languages

Cons:

Extra installation

Basic Concepts

Construct: Basic building block for an AWS CDK app. Represents a “cloud component” and encapsulates everything AWS CloudFormation needs to create the component. They can be developed or downloaded from AWS Construct Library
Stack: Constructs need to be created within the scope of a Stack. This corresponds to a CloudFormation template.
App: Stacks are created in the scope of an App. An App can contain multiple stacks.

CDK Basics

In order to create an application that uses CDK we need to install the CDK
The application can have one of the 3 supported templates:
- app: General purpose application. This is the default template
- lib: Used to develop a CDK construct
- sample-app: Creates an application already populated with a sample CDK application.
The following diagram illustrates the app lifecycle:

Installing CDK

AWS CDK is developed using TypeScript. It’s available via npm:

npm install -g aws-cdk

Using CDK

In my example, I will use C#. A new project can be created by using cdk init command:

It expects 2 parameters: Language (Can be one of these: csharp, fsharp, java, javascript, python or typescript).

cdk init app --language csharp

To test my first app, I followed the sample app provided and added an S3 construct to my code:

public class CdkWorkoutStack : Stack
{
    public CdkWorkoutStack(Construct parent, string id, IStackProps props) : base(parent, id, props)
    {
        _ = new Bucket(this, "CdkBucket", new BucketProps
        {
            Versioned = true
        });
    }
}

Next step is to synnthesize a CloudFormation template by running

cdk synth

This generates a folder called cdk.out which contains a file named CdkWorkoutStack.template.json and with the following contents:

{
  "Resources": {
    "CdkBucket2FB0D10E": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "VersioningConfiguration": {
          "Status": "Enabled"
        }
      },
      "UpdateReplacePolicy": "Retain",
      "DeletionPolicy": "Retain",
      "Metadata": {
        "aws:cdk:path": "CdkWorkoutStack/CdkBucket/Resource"
      }
    }
  }
}

Final step is to create the resources by running

cdk deploy

This produces the following results:

And not surprisingly a CloudFormation stack with the bucket is created in AWS.

In the next version I’m going to change some properties of the bucket such as:

public class CdkWorkoutStack : Stack
{
    public CdkWorkoutStack(Construct parent, string id, IStackProps props) : base(parent, id, props)
    {
        _ = new Bucket(this, "CdkBucket", new BucketProps
        {
            Versioned = true,
            BlockPublicAccess = new BlockPublicAccess(new BlockPublicAccessOptions()
            {
                BlockPublicAcls = true,
                BlockPublicPolicy = true
            })
        });
    }
}

After this change in the code we can review what’s going to be updated by running the following command:

cdk diff

and it produces the following result outlining the changes between the current code and the deployed version:

And the stack can be deleted by running the following command:

cdk destroy

In the CDK version I’m using the default deletion policy is to retain. That’s why when I deleted the stack via CDK it didn’t delete the bucket. In order to change that behaviour I had to change the value to delete as such:

public class CdkWorkoutStack : Stack
{
    public CdkWorkoutStack(Construct parent, string id, IStackProps props) : base(parent, id, props)
    {
        var bucket = new Bucket(this, "CdkBucket", new BucketProps
        {
            Versioned = true,
            BlockPublicAccess = new BlockPublicAccess(new BlockPublicAccessOptions()
            {
                BlockPublicAcls = true,
                BlockPublicPolicy = true
            })
        });

        var resource = bucket.Node.FindChild("Resource") as Amazon.CDK.CfnResource;
        resource.ApplyRemovalPolicy(RemovalPolicy.DESTROY);
    }
}

This time it produces a CloudFormation template with a different policy:

{
  "Resources": {
    "CdkBucket2FB0D10E": {
      "Type": "AWS::S3::Bucket",
      "Properties": {
        "PublicAccessBlockConfiguration": {
          "BlockPublicAcls": true,
          "BlockPublicPolicy": true
        },
        "VersioningConfiguration": {
          "Status": "Enabled"
        }
      },
      "UpdateReplacePolicy": "Delete",
      "DeletionPolicy": "Delete",
      "Metadata": {
        "aws:cdk:path": "CdkWorkoutStack/CdkBucket/Resource"
      }
    }
  }
}

This time after destroy command it did delete the bucket as well.

Conclusion

I think infrastructure as code is definitely the way to manage cloud resources and CDK provides a great way to simplify the process. It’s at early stages for the time being as the NuGet packages are in devpreview mode but it’s good enough to rely on and start developing with CDK.

Resources

Generating Test Data with Bogus

Volkan Paksoy — Thu, 27 Jun 2019 06:00:00 +0000

Generating high-quality test data can have an impact on the accuracy of the tests overall. In this post I’ll show using a helpful C# library called Bogus

Showcase project: Bank Statement Generator

In this example I’ll generate fake bank statements. Normally they come in CSV files and have the following model:

public class BankStatementLine
{
    public DateTime TransactionDate { get; set; }
    public string TransactionType { get; set; }
    public string SortCode { get; set; }
    public string AccountNumber { get; set; }
    public string TransactionDescription { get; set; }
    public decimal? DebitAmount { get; set; }
    public decimal? CreditAmount { get; set; }
    public decimal Balance { get; set; }
}

I’ll use Bogus to generate realistic fake statement lines and finally save it as a CSV and see if it looks real.

Rules and restrictions

I want the fields in the model above conform to certain set of rules to be realistic:

Transaction Date must be within a certain range I provide as bank statements are generated for a date range.
Dates should be incremental and not random
Sort Code must be in the following format: NN-NN-NN and must be the same for the entire statement.
Account number must be an 8-digit number and same for the entire statement.
Transaction Description must be free text
Debit Amount and Credit Amount must be decimal numbers but only one of them can be present at any given line
Transaction Type must be one of the pre-defined values and also some types can be for credit and some for debit only.
Balance should be sum of all debit and credit amounts plus the first balance in the statement. So this value is dependent on the values that come before it.
The number of lines in a statement should be random.

Rule implementations

Some rules stated above are very straightforward and easy to implement. These are some samples of what Bogus is capable of. For the full documentation check out the GitHub repository.

Date range support

Generating a date between a range is simple:

.RuleFor(x => x.TransactionDate, f => f.Date.Between(startDate, endDate))

Enum and array support

For Transaction Type I want to select a random value from a list of set values. This can be done in 2 ways: By using an enum or an IEnumerable.

var transactionTypes = new[] { "FPO", "DEB", "DB", "FPI" };

and in the rule description it can be used as

.RuleFor(x => x.TransactionType, f => f.PickRandom(transactionTypes) )

Another way is using enums such as:

public enum TransactionType
{
    FPO,
    DEB,
    DB,
    FPI
}

and the rule becomes:

 .RuleFor(x => x.TransactionType, f => f.PickRandom<TransactionType>().ToString())

In my final implementation I used selecting from a list of objects. You can check out the sample code to see that version.

Number range

For the account number I need an 8-digit number which can be achieved with something like this rule:

.RuleFor(x => x.AccountNumber, f => f.Random.Long(100000000, 99999999).ToString())

Bogus API also has builtin support for account number so the following is a more elegant and expressive way of achieving the same:

.RuleFor(x => x.AccountNumber, f => f.Finance.Account())

Formatting string

Formatting Sort Code can be achieved by Field.Random.Replace method

.RuleFor(x => x.SortCode, f => f.Random.Replace("##-##-##"))

Similar to account number, it also has built-in support for sort code:

.RuleFor(x => x.SortCode, f => f.Finance.SortCode())

Null values

In my case in some fields I’d like to have null values too. This can be achieved by OrNull extension method. For example, in the code below it generates %20 of DebitAmount values null.

.RuleFor(x => x.DebitAmount, f => f.Random.Decimal(0.00m, 9999.00m).OrNull(f, 0.2f))

Common fields

In my case some values in each statement line repeat throughout the entire statement such as account number and sort code. To achieve that I created a “base” statement line and every fake statement line used these shared fields instead of generating new ones.

var commonFields = new Faker<BankStatementLine>()
    .RuleFor(x => x.AccountNumber, f => f.Finance.Account())
    .RuleFor(x => x.SortCode, f => f.Finance.SortCode())
    .Generate();

var fakeTransactions = new Faker<BankStatementLine>()
    .StrictMode(true)
    .RuleFor(x => x.AccountNumber, commonFields.AccountNumber)
    .RuleFor(x => x.SortCode, f => commonFields.SortCode)
    ...
    ...

Random number of objects

It’s more realistic to have varying number of lines in statements. With Generate method you can specify the exact number of items you want to generate which is good for unit tests. For my purposes I just wanted to create random of rows in each statement as I only needed the data to be imported. This can be achieved by GenerateBetween:

var statementLines = fakeTransactions.GenerateBetween(10, 20);

Dependent values

The tricky part in this scenario was the dependent values. Normally when you use RuleFor extension method it generates the value for that field alone in isolation. In my case, one restriction was Debit Amount and Credit Amount could not both have values in the same line. Also Balance depends on these values and needs to be calculated in each line.

As far as I can tell there’s no built-in support to define these dependencies. Based on my tests I was able to achieve this in 2 ways

Update the values accordingly in FinishWith extension method
Use Rules extension method to define multiple rules at once and implement the restrictions inside it.

I think the latter is a better solution as FinishWith sounds more like clean up, logging or similar extra activity where Rules sound more like actual business logic implementation.

So with that in mind my rules for Debit Amount, Credit Amount and Balance fields looked like this:

.Rules((f, x) =>
{
    var debitAmount = (decimal?)f.Random.Decimal(1, 100).OrNull(f, 1.0f - statementconfig.DebitTransactionRatio);
    if (debitAmount.HasValue) // Is it a debit transaction?
    {
        x.CreditAmount = null;
        x.DebitAmount = debitAmount.Value;
        balance -= x.DebitAmount.Value;

        x.TransactionType = f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == TransactionDirection.Debit || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }
    else
    {
        var creditAmount = f.Random.Decimal(1, 100);
        x.DebitAmount = null;
        x.CreditAmount = creditAmount;

        balance += x.CreditAmount.Value;

        x.TransactionType = f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == TransactionDirection.Credit || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }

    x.Balance = balance;
});

A caveat with this approach is that I cannot use StrictMode anymore as it complains about those 3 fields having null values. It specifically mentions that in the exception. If you use Rules you’re on your own to ensure that all fields are populated properly.

Another drawback of setting multiple rules at once is that it can easily make the code harder to read. Fortunately for me, the author of the library Brian Chavez kindly reviewed the code and suggested some refactorings one of which proved it was still possible to use RuleFor method and strict mode. I’ve updated the final source code with these refactorings. So with individual rules the implementation looks like this:

.RuleFor(x => x.DebitAmount, f =>
{
    return (decimal?)f.Random.Decimal(1, 100).OrNull(f, 1.0f - statementconfig.DebitTransactionRatio);
})
.RuleFor(x => x.CreditAmount, (f, x) =>
{
    return x.IsCredit() ? (decimal?)f.Random.Decimal(1, 100) : null;
})
.RuleFor(x => x.TransactionType, (f, x) =>
{
    if (x.IsCredit())
    {
        return RandomTxCode(TransactionDirection.Credit); ;
    }
    else
    {
        return RandomTxCode(TransactionDirection.Debit);
    }

    string RandomTxCode(TransactionDirection direction)
    {
        return f.PickRandom(TransactionType.AllTransactionTypes
            .Where(tt => tt.Direction == direction || tt.Direction == TransactionDirection.DebitOrCredit)
            .Select(tt => tt.Code));
    }
})
.RuleFor(x => x.Balance, (f, x) =>
{
    if (x.IsCredit())
        balance += x.CreditAmount.Value;
    else
        balance -= x.DebitAmount.Value;

    return balance;
});

IsDebit and IsCredit methods referred to above are extension methods defined like this:

public static class Extensions
{
   public static bool IsCredit(this BankStatementLine bsl)
   {
      return bsl.DebitAmount is null;
   }
   public static bool IsDebit(this BankStatementLine bsl)
   {
      return !IsCredit(bsl);
   }
}

Random text

For the transaction description for now I’ll go with random Lorem Ipsum texts. Bogus has support for this too

.RuleFor(x => x.TransactionDescription, f => f.Lorem.Sentence(3))

I probably will need to use a fixed list of descriptions soon but for the time being it’s fine. Also as shown below it’s very easy to switch to that too.

Incremental values

Similar to balance being dependent on the previous values, transaction date is also dependent as it needs to go in an incremental fashion. I couldn’t find built-in support for this so implemented it using my own shared variable like this:

.RuleFor(x => x.TransactionDate, f =>
{
    lastDate = lastDate.AddDays(f.Random.Double(0, statementconfig.TransactionDateInterval));
    if (lastDate.Date > statementconfig.EndDate)
    {
        lastDate = statementconfig.EndDate;
    }
    return lastDate;
})

Putting It All Together

So let’s see the output with the help of another nice library called Console Tables

Source Code

Sample application can be found under blog/GeneratingTestDataWithBogus folder in the repository.