DEV Community: George Saadeh

Hello AWS Microservice Extractor for .NET, Let's Work on that Legacy App

George Saadeh — Wed, 08 Dec 2021 15:23:30 +0000

Background

Organizations are increasingly facing challenges while modernizing their legacy applications especially those their business depends on for day-to-day operations. Many are actively looking to progressively decompose large monoliths (legacy or not) into microservices as these systems become too large to deal with.

There are different strategies, techniques and patterns teams can use to break down a monolithic application. Recently AWS introduced the AWS Microservice Extractor for .NET, a new tool that helps customers on their path to extract microservices from their monolithic applications. It simplifies the process of refactoring applications into independent services using source code analysis and visual representations.

In this tutorial, we look at how we can use the AWS Microservice Extractor for .NET on a real-world application to extract a service.

At the time of writing the AWS Microservice Extractor for .NET supports only .NET Framework CLR 4.0+ ASP.NET web service applications hosted on IIS and the extraction only supports CLR 4.7.0+. Therefore I decided to use the source of one of the Web API projects I worked on in the past which was built on .NET 4.6.1 and which I've upgraded to 4.7.2. Luckily the upgrade was smooth.

I do not own the source code of this application and I was only a contributor. Therefore I made sure not to share any code. I also disabled data sharing during analysis.

It is only worth noting that AWS Microservice Extractor for .NET is available for use at no cost which is phenomenal.

Preparations

To prepare the environment, I followed the instructions in the documentation.

I created a new IAM user and attached a custom IAM policy

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Resource": [
        "*"
      ],
      "Action": [
        "serviceextract:GetConfig"
      ]
    },
    {
      "Sid": "SectionForMetricsService",
      "Effect": "Allow",
      "Action": "execute-api:invoke",
      "Resource": [
        "arn:aws:execute-api:us-east-1:*:*/prod/POST/put-metric-data",
        "arn:aws:execute-api:us-east-1:*:*/prod/POST/put-log-data"
      ]
    }
  ]
}

I then configured my AWS CLI profile and installed the AWS Microservice Extractor for .NET tool.

I did not need to do the rest of the steps since my Windows VM already had Visual Studio 2019 installed with multiple versions of the .NET framework.

After the quick configuration the setup was successful.

Onboarding

After downloading the application code on my machine I was ready to start the analysis.

I clicked on Onboard application and I've added the solution file which points to the Web API application. I did not select the runtime profiling data.

After a few minutes, the application was onboarded successfully.

It is worth noting that the tool includes analysis for .NET core portability which is great.

Visualizing

Now comes the exciting part. My application was analyzed and I was ready to see the visual representation.

The purpose of this visualization exercise is to learn more about the application components, dependencies and metrics to "quickly" identify which could be refactored into smaller service.

Alright, this was a bit surprising at first glance with all the different connections. After going through the documentation I was able to navigate the interface slightly better and identity some of the services I have. I used the filter to look up my SpeakerService.

To understand the dependencies of the service, I clicked on it and I was presented with a nice view of the edges. Superb!

Isolating the service in a separate part of the graph was no trivial task. After some battling, I managed to get the components grouped and ready for extraction.

Extracting

Exhilarated and curious, I hit the extract group as service button.

I entered a few details about the service and then select which invocation method i want to use.

A few rounds of trial and error between the visual designer and the extractor and I was still getting this. The extraction halted at 60%.

Command failed: error while building and applying rules to transform the MicroService: error while updating build files for microservice: unable to remove some projects from sln

It was one of the infrastructure projects that it couldn't remove from the solution.

Nevertheless, the extraction of the SpeakerService completed according to the application details screen and also the logs.

The log file can be found in the installation directly under /logs/drift-extraction.log.

The extracted service was located in the installation directory.

Opening the solution of the ApiIntegrationService in Visual Studio revealed multiple missing "infrastructure" projects which is probably why the extraction failed.

I can take it from here and add the missing dependencies and get the service in a build-able state.

Known Limits and Concerns

This documentation article mentions a few limits that you need to know before you start using it.

When extracting a microservice, you need to decide what to do with the building blocks and "shared" code. Do you publish those to nuget package (i.e. internal feed) and reference them, do you use code duplication by duplicating only what's needed etc. It's your call. The extractor however did not do very well with shared projects for me. I was not sure if I need to include all of these dependencies in the group. I believe there is additional manual work to be done.

Conclusion

AWS Microservice Extractor for .NET is a fantastic tool. It offers great analysis and visualization capabilities with somehow limited or dissatisfying extraction capabilities (at least for me).

It still provides a lot of assistance in your migration and can get you started on the right path. Any help is greatly required and appreciated.

You still need to touch both the old application and the new extracted service, maybe add some missing dependencies.

Not the glorious ending I was hoping for but nonetheless the start of a compelling journey. The tool was recently released and I expect great things for the future.

P.S: I will update the article with any new findings and if I manage to get a smooth extraction for the service.

Migrating from Microsoft SQL Server to PostgreSQL using Babelfish and Amazon Aurora

George Saadeh — Tue, 16 Nov 2021 14:23:42 +0000

Background

An increasing number of organizations are planning to move away from proprietary databases to open source for various reasons such asavoiding expensive costs and licensing terms while leveraging the flexibility and database freedom of modern offerings.

Migrating from legacy and commercial databases is often a challenging, time-consuming and resource-intensive undertaking. In a legacy / enterprise environment, the database is often used by a variety of desktop and web applications, sometimes using different programming languages. Migrating the database would then require much work on those applications to rewrite the layer that interacts with the database. Many organizations are not willing to make such investment on these legacy applications.

Babelfish for Aurora PostgreSQL is a new capability that makes it possible to run Microsoft SQL Server applications directly on Amazon Aurora with little to no code changes.

In this tutorial, we look at how we can perform the migration on the infamous Northwind database and some of the challenges and considerations while migrating much larger and complex databases.

I used a sample Northwind .NET application available on Github for testing purposes to illustrate the different kinds of operations that we could run against the database. I also used the database script from Microsoft.

Compatibility Assessment

Before starting the migration we need to assess the compatibility of our Microsoft SQL database with Babelfish for PostgreSQL. Such assessment will help us determine the feasibility and complexity of the migration.

Babelfish Compass is a tool that analyzes the SQL/DDL code for the database and determines which features are/aren't supported so you would know ahead of time how to make the migration successful and what sort of tweaks you might need to do on your existing database.

To start with the assessment, I need to generate the SQL script for the database, something I can achieve using the SQL Server Management Studio if I don't have it already. Since I downloaded the script, I skipped that.

I also need to have Babelfish Compass installed.

At this point I ran the assessment command.

sh ./BabelfishCompass.sh MyReport ~/Documents/Temp/NorthwindCore/src/scripts/instnwnd.sql -appname NorthwindCore

The command ran successfully and the result looked promising.

The report was generated and I can analyze the results using the HTML page.

Luckily for me, there were no issues highlighted in the report... I may have chosen the database on purpose.

In a real world scenario, you would need to make a decision on the migration following the report. You may be required to do further work on your database or tweak certain procedures or functions.

Limitations and Considerations

Achieving full compatibility with Babelfish for PostgreSQL for large and complex databases is difficult. You can use this guide to learn more about the limitations and the features that are not supported and determine if it's the right tool for your database.

Another consideration is the version of PostgreSQL. At the time of writing, Babelfish supports only PostgreSQL version 13.4 and newer, which is not yet available for the serverless capacity type.

As for region availability, Babelfish for Amazon Aurora PostgreSQL is available in all regions supported by Aurora PostgreSQL.

On the application side, a discovery of the tools and frameworks is essential to assess the feasibility and risks ahead of the migration. Using Entity Framework or similar ORMs may or may not work, based on the setup, dependencies, versions and other factors.

Preparations

I created a new Microsoft SQL Server RDS instance to act as the source database.

I connected to the database using DataGrip (you can also use SSMS) and ran the northwind database script that I downloaded earlier.

I then created the target Aurora PostgreSQL database cluster with a single writer instance. Note the database version that I used which is the minimum required to support Babelfish. You can also use the filters to show only the supported versions.

I also turned Babelfish on in the settings.

You can read more about the different migration modes (single database vs multiple databases) in the documentation.

I ended up with a source and a target database.

Schema Conversion

In a standard database migration project, you can use the AWS Schema Conversion tool (AWS SCT) to convert your existing database schema from one database engine (i.e. MS SQL Server) to another (i.e. PostgreSQL).

If you attempt to do so with the babelfish-enabled PostgreSQL database you will get all sorts of errors and permission issues because Babelfish maintains the database objects information in its sys database.

For this reason, I connected to the PostgreSQL instance using SQL Server Management Studio (SSMS) to create the northwind database. Instead of using the default SSMS Object Explorer, I used the New Query window to connect.

I then ran the following script:

CREATE DATABASE northwind

Finally, I extracted the schema creation script from the Northwind script file I downloaded and I ran it in the same query window in SSMS.

You could run into multiple iterations of code adjustments here.

Upon completion, I was able to browse the created babelfish_db in DataGrip by connecting to the PostgreSQL instance using the PostgreSQL driver.

I could have ran the entire script at this point to add the data, but I wanted to follow the steps of a real-world migration project and opt to use a full load using the Database Migration Service.

Database Migration

The schema has been created in the PostgreSQL database and I was ready to start migrating the data.

Using the AWS Database Migration Service (DMS), I created a new replication instance first.

I also created two endpoints representing our source and target databases.

At this point, I created the database migration task and ran a full load of the data into the target PostgreSQL database. For the transformations and mappings, I selected the "convert-lowercase" rule for the tables and columns in the dbo schema.

The data has been loaded successfully and I was ready to run the app.

Testing the App

Before I declared success, I wanted to run the application by pointing to the Microsoft SQL Server database and then switching the connection string to point to the new PostgreSQL instance.

After a few rounds of minor code changes, mainly because of the Entity framework setup in the sample application, my test was successful!

I recorded a few steps of the testing I did.

Conclusion and Considerations

If you are planning a migration from MS SQL Server to open source then Babelfish is a fantastic tool. It provides the capability for PostgreSQL to understand queries from applications written for Microsoft SQL Server. It can also provide you a quick migration path, with no or minor changes to your application.

Before using Babelfish and deciding whether it is the right one for you, I encourage you to check the documentation and understand how the capability works and what limitations are relevant to the applications you are trying to migrate.

A better way to work with AWS DynamoDB and .NET

George Saadeh — Tue, 13 Apr 2021 06:31:54 +0000

Background

One of the challenges that we faced when we started building our platform a few years ago was the .NET tooling around DynamoDB. Most of our developers had been using .NET full-fledged and micro ORMs such as Entity Framework and Dapper with relational databases and expected the same experience with DynamoDB, our key-value and document database of choice.

Originally, we began using the .NET DynamoDB SDK and it worked well for us. We quickly found ourselves creating extensions and wrappers around the SDK to remove the low level duplicated code, mainly aimed to be more productive, until we stumbled upon a new tool, which was exactly what we have been looking for.

Introducing PocoDynamo

PocoDynamo is a Typed .NET client which extends ServiceStack's Simple POCO life by enabling re-use of your code-first data models with Amazon's industrial strength and highly-scalable NoSQL DynamoDB. It enhances and improves AWSSDK's low-level client, with rich, native support for intuitively mapping your re-usable code-first POCO Data models into DynamoDB Data Types. Thus improving the overall developer experience and productivity and makes working with DynamoDB, a joy!

Getting Started

In this section we will discuss how to get started using PocoDynamo in a real-world application using some of the best practices we learned.

First we will need to add the NuGet package:

> dotnet add package ServiceStack.Aws

Next we'll need to create an instance of AmazonDynamoDBClient with the AWS credentials and Region info:

public static IAmazonDynamoDB CreateAmazonDynamoDb(string serviceUrl)
{
            var clientConfig = new AmazonDynamoDBConfig {RegionEndpoint = RegionEndpoint.EUWest1};

            if (!string.IsNullOrEmpty(serviceUrl))
            {
                clientConfig.ServiceURL = serviceUrl;
            }

            var dynamoClient = new AmazonDynamoDBClient(clientConfig);

            return dynamoClient;
}

We can now create an instance of PocoDynamo. We can register the instance through dependency injection as the clients are Thread-Safe.

services.AddSingleton<IAmazonDynamoDB>(x => CreateAmazonDynamoDb(options.ServiceUrl));
services.AddSingleton<IPocoDynamo, PocoDynamo>();

Next we will configure PocoDynamo by adding it to our Startup class. The purpose of this configuration is to initialize the schema which we will discuss in the following sections in more details.

app.ConfigureDynamoDb();

And that's it. Now we are ready to add PocoDynamo as a dependency by injecting it in the constructor:

public EmployeeController(IPocoDynamo db)
{
   _db = db;
}

Design Approaches

When it comes to designing your data model in DynamoDB, there are two distinct design approaches multi-table or single-table. We are not going to discuss the difference between the two approaches, you can find plenty of articles about that. We will merely show how we can use the library in both.

Multi-Table

In a multi-table approach, we have one table per each entity and each item maps to a single instance of the entity providing consistency across attributes.

First we define an entity such as Employee

public class Employee
{
    [HashKey]
    public string Id { get; set; }
    public string Name { get; set; }
    public string DepartmentId { get; set; }

    public Employee() {}

    public Employee(string name, string departmentId)
    {
        Id = Guid.NewGuid().ToString("D");
        Name = name;
        DepartmentId = departmentId;
    }
}

Next we register the table for the employee type

var employeeType = typeof(Employee);
// This is how we set the name of the table
employeeType.AddAttributes(new AliasAttribute("employee"));

To add a new Employee to the table, it is very simple

var employee = new Employee(name, departmentId);
_db.PutItem(employee);

And to read the employee from the table

var employee = _db.GetItem<Employee>(id);

In a multi-table approach, working with PocoDynamo is very straightforward and doesn't require any additional considerations unlike a single-table approach.

Single-Table

In a single-table approach, one table holds multiple types of entities within it. Each item has different attributes set on it depending on its entity type. While this approach might be less common, it has lots of advantages when querying "related" data such as in a one to many relationship. Let's see an example.

First we will create two entities. We will define a generic combination of HashKey/RangeKey and we will create a Type attribute to hold the type name.

public class Customer : IRecord
{
     ...

     public Customer(string name)
     {
         CustomerId = Guid.NewGuid().ToString("D");
         Name = name;
         HashKey = $"CUSTOMER#{CustomerId}";
         RangeKey = $"CUSTOMER#{CustomerId}";
         Type = "CUSTOMER";
     }
}

public class Order : IRecord
{
    ...

    public Order(string customerId, double orderTotal)
    {
        OrderId = Guid.NewGuid().ToString("D");
        OrderTotal = orderTotal;
        CreatedDate = DateTime.Now;
        HashKey = $"CUSTOMER#{customerId}";
        RangeKey = $"ORDER#{OrderId}";
        Type = "ORDER";
    }
}

Next we are going to register both entities to map to the same table.

var customerType = typeof(Customer);
customerType.AddAttributes(new AliasAttribute("customer-orders"));

var orderType = typeof(Order);
orderType.AddAttributes(new AliasAttribute("customer-orders"));

db.RegisterTable(customerType);
db.RegisterTable(orderType);

Creating Customers and Orders would look similar to how we do it in a single-table approach so I am going to omit it. What we will focus on how we can query data.

The first example we will look at is how to query a single customer by customer id. Notice the use of the same Hash key and Range key to indicate that we are querying a customer here and not customer and orders.

var customer = _db
                .FromQuery<Order>()
                .KeyCondition($"HashKey = :customerId  and RangeKey = :orderId", new Dictionary<string, string>()
                {
                    {"customerId", $"CUSTOMER#{id}"},
                    {"orderId", $"CUSTOMER#{id}"}
                })
                .Exec().SingleOrDefault()

The second example we will look at is how to query the orders of a customer (in a case of one-to-many relationship). Notice the use of begins_with in the expression as we would like to query only orders and not to include the customer data since it will break the mapping.

var orders = _db
                .FromQuery<Order>()
                .KeyCondition($"HashKey = :customerId  and begins_with(RangeKey, :order)", new Dictionary<string, string>()
                {
                    {"customerId", $"CUSTOMER#{id}"},
                    {"order", $"ORDER"}
                })
                .Exec().ToList()

In a single table approach, you will need to use the right combination of Hash / Range key to get exactly the data that you need.

Considerations

PocoDynamo has a 10 Tables free-quota usage limit which can be unlocked with a commercial license key.

Conclusion

In this article we've looked at ServiceStack's PocoDynamo and how it can help us be more productive when working with DynamoDB. If you are okay with buying a commercial license, it is well worth it. If you know of any similar tools, free or open-source, I'd be glad to hear about it.

You can find the source code on Github