DEV Community: Luis Rodrigues

Backup Azure Kubernetes Services (AKS) cluster with Velero

Luis Rodrigues — Sat, 11 Sep 2021 15:38:35 +0000

In the past, I was in charge of backup a MongoDB instance that runs in Azure Kubernetes Services (AKS). This database has at least 2 TB of data that was being persisted in an Azure Managed Disk with Kubernetes Persistent Volume (PV).

I could not run the “mongodump” without issues. This was overloading the database and take a long time to complete the backup process. The same behavior happened when I tried to restore the database backup.

Mongodump is a utility tool that allows us to create MongoDB backups. You can check this tool here.

Then, instead of run the “mongodump” tool, I applied options below that I found in Microsft Documentation:

Take a backup from the Azure Managed Disk with Azure Backup.
Create a scheduler backup to MongoDb’s PVC with the Velero tool.

You can see the documentation about AKS backup in this link from Microsoft Documentation.

The first option basically, allows you to take a snapshot from the disk that is being used by Kubernetes PVC. Later, you can restore this snapshot if you need it.

For the second option, you can use a tool called Velero to backup the whole AKS cluster, or just some Kubernetes resources, like Deployments, Services, persistent volume claims (PVC), PV, etc.

The backup destination created by Velero could be set up with plugins, which have support for the principal cloud providers, like Azure, AWS, and Google Cloud. In the case of Azure, the backup will be stored in Azure Account Storage.

In the end, what I did was create a backup from PV that MongoDB is using to persist its data.

So, in this article, I will present how to install Velero and how to use it to create and restore backups.

Setup Velero

The setup of Velero in the environment is very simple. Beyond the Azure account, you will need Azure Account Storage and AKS cluster.

After the resources created in Azure, you can apply the steps below to setup Velero in your environment:

Install Azure CLI and kubectl tool in the machine that will apply the next steps.
Connect to the AKS cluster. You can use this link to connect to your cluster.
Create environment variables to store Azure Account Storage configurations.

Create a credential file. This file will be used to configure the Azure Account Storage credentials in Velero.

Install Velero client. You can find the Velero client on its home page.
Finally, install Velero in the AKS cluster. After that, we can create backups and restore them.

The Velero installation will create a set of new Kubernetes resources in the AKS cluster. Velero will use the Kubernetes namespace “velero”. Then, to remove Velero from the AKS cluster, apply the command below:

Also, like any Kubernetes resources, we can update it to achieve your requirements. For example, we can update resource limits from its Deployment:

Create and Restore Backups

Now that you have Velero in the AKS cluster, you can create and schedule backups, and restore them with the Velero client.

Below, there are some commands to help you create and restore backups:

Create a new backup:

Create a new schedule:

Restore backup:

You can find more commands and options in the Velero’s home page. Click here to check them.

When the commands above are executed by Velero, new folders are created in its Azure Blob Storage. Here is an example:

Beyond backup and restore the cluster, Velero can be useful in the migration process.

We can use that to migrate our AKS cluster to a new AKS cluster or a new Kubernetes cluster from another Cloud Provider.

For now, that’s all. I hope that can be helpful for you!

.NET and Apache Ignite: Testing Cache and SQL API features — Part I

Luis Rodrigues — Sat, 11 Sep 2021 15:33:15 +0000

Some non-functional requirements from our applications are about performance and scalability. Then, we start thinking about a lot of strategies: review architecture, refactor features and algorithms, apply some caching strategy, etc.

Last days, I started using Apache Ignite as a cache strategy for some applications. Apache Ignite is an open-source In-Memory Data Grid, distributed database, caching, and high-performance computing platform.

Yeah! That’s it! All this in one solution. :D

If you want to know more about Apache Ignite, take a look at its documentation. Beyond .NET, there are samples for other programming languages as well.

This article will be split into two parts. In this first one, I will present the basic steps to work with the Cache feature and check results from a small test with that. In the second part, let’s see how can we work with SQL API feature in .NET applications.

Machine settings

For this test, I used the machine settings below:

Windows 10
Intel(R) Core(TM) i5–4440
16 GB RAM
.NET 5
Apache Ignite 2.10.0
Docker version 20.10.6

Run docker container

Before implementing our cache repository, let’s run Apache Ignite with its official Docker container.

Then, run the following command to start the Apache Ignite container:

We can persist the data that will be used in the sections below in memory or on disk. For this article, the is being persisted in memory.

After running the command above, you should see the screen below:

Check if there are error messages when the Apache Ignite container has been started. If everything is OK, let’s test its Cache feature in the sections below.

Basic Settings for .NET and Apache Ignite

The steps to use Apache Ignite are very simple. You can follow the items below to allow your application to connect with Apache Ignite.

Add NuGet package below:

Install-Package Apache.Ignite -Version 2.10.0

Set up a new connection with IgniteClientConfiguration class*.* in this part you need to provide at least one Apache Ignite server:

Start a new connection with Ignition.StartClient method. This method returns an interface called IIgniteClient, which contains methods to work with cache clusters:

With IIgniteClient.GetOrCreateCache, get or create a new cache repository to a specific object type. This method returns the ICacheClient interface, which contains methods to work with the cache repository.

Done! Now your application can perform operations in a cache repository from Apache Ignite. I recommend checking the Apache Ignite documentation to know more about available methods.

Testing the Cache system

Now that we have an Apache Ignite cluster ready and the application set up appropriately, let’s test its cache feature.

For this test, two classes were created to be inserted on the Apache Ignite Cache: OrderCacheEntity (Order) and ItemCacheEntity (OrderItem).

Basically, one Order can have one or more OrderItem associated with it. Then, the next step is to generate as many Order instances as possible, adding and getting them to the cache, and finally check the performance with some counters.

The code below inserts 1 million Orders in the cache repository and checks its performance for “ICacheClient.Put” and “ICacheClient.TryGET” methods. As an attempt to create a small “workload” against the Apache Ignite container, those operations are executed with Parallel Task.

As we can see in the code above, some KPIs were created to check the performance of each operation against Apache Ignite Cache:

KPIs Operation Time:

INPUT time (AVG seconds): the average in seconds to add an item to the cache
OUTPUT time (AVG seconds): the average in seconds to get an item from the cache
INPUT time (MAX seconds): the maximum time in seconds to add an item to the cache.
OUTPUT time (MAX seconds): the maximum time in seconds to get an item from the cache.
INPUT time (MIN seconds): the minimum time in seconds to add an item to the cache.
OUTPUT time (MIN seconds): the minimum time in seconds to get an item from the cache.

KPIs Exceptions:

INPUT ERROR: number of exceptions created when the application was adding an item to the cache.
OUTPUT ERROR: number of exceptions created when the application was getting an item from the cache.

KPIs Summary:

ITEMS CREATED: how many items were created in the cache.
ITEMS RECEIVED: how many items were got from the cache.
ITEMS MISSING: how many items were not in the cache.

Results from Application Test

After running the application test, the metrics below were collected:

In general, the Apache Ignite presented a nice performance adding and getting objects. The average time to add items in the cache was 0.016 seconds and 0.025 seconds to get those items from the cache. Also, there were no exceptions and lost items in this test.

Checking the Apache Ignite container, the memory consumption increase by about 390 MB, which represents about 130% more than its initial state:

Container stats (initial state)

Container stats (after running application test)

For now, that’s all. In the second part, let's see how can we implement the SQL API feature in our .NET applications. From a development perspective, I think that this feature from Apache Ignite is the most interesting so far.

I hope that this can be helpful for you!

Creating a local development environment to Kong API with Docker

Luis Rodrigues — Sat, 11 Sep 2021 15:27:29 +0000

In this article let’s create a local development environment for Kong API. This environment is very useful to test our integration between our applications and Kong API Gateway.

If you want to know more about Kong API Management, check its documentation. Beyond API Gateway, Kong has other nice features to management our APIs.

We are going to use Docker to create our local development environment. Then, please install Docker tools in your local machine if you don’t have them yet.

Three docker containers are required to create a local development environment:

pantsel/konga: open source admin tool for Kong API Management.
kong: Kong API Management
postgres: Database for Kong API Management

Running Kong API Gateway

First, let’s create a docker network to each initiate docker container can communicate between themselves:

Start Postgres database to create a database for Kong API:

In this example we are using Postgres to creating a new database, however, Kong API Gateway supports Cassandra as its database as well.

Now, we need to run a data migration process to prepare Kong API database. Run the command below to start the data migration process:

Finally, start Kong API Gateway with the command below:

When the container is ready, two ports will be available for use in our machine (localhost):

Admin API [http://localhost:8001]: Use this port to manage your APIs in Kong API Gateway.
Proxy [http://localhost:8000]: Use this port to call your APIs with Kong API Gateway.

Check Admin API documentation to know more about how to manage your APIs in Kong API Gateway.

Running Admin UI

Now that we have our Kong API Gateway ready, we can manage services, routes, and plugins for our APIs.

Those Kong API objects are managed with Admin API. For instance, the command below creates a new service to fictional “Product API” from the fictional address “https://product-api.net”:

Sometimes, this task can be very challenging. Then, for testing purposes, we can manage our APIs in Kong API Gateway with Konga:

Konga is an open source project that allows us to manage Kong API Gateway with a UI. You can know more about the project here.

Run the command below to start the Konga container:

Konga will be started on port 1337 from your localhost. In the first access, the application will request to create a new user.

After creating the user, we need to create a new connection between Konga and Kong API Admin to manage our services:

Provide a name for the new connection, the IP address where Kong Admin API is running, and click on the “Create Connection” button. Then, you can see the Konga Dashboard.

If you had some issue with Kong Admin URL, try to provide the IP address from the Kong docker container. Execute the command below to get the IP address from the container:

It’s done! Now you can explore more Kong API Gateway in your local machine. Have happy findings with Kong API Gateway! :D

I hope that this can be helpful for you!

.NET: Call Func delegates from Dictionary objects

Luis Rodrigues — Fri, 13 Aug 2021 19:46:39 +0000

How about store Func delegates in a Dictionary object and invoke them when you really need them?

This approach can help us to avoid having nested conditional in our code and keep its cyclomatic complexity lower.

“Cyclomatic complexity is a software metric used to determine the complexity of a program”. Click here to know more about that.

For example, suppose we want to create a method that returns a result from a math operation between two numbers. Its result will depend on the operation type that the user will provide to us: addition, subtraction, multiplication, and division.

A simple way to solve that is by creating a method that accepts three parameters: a number that will apply some operation and the operation type. Like this:

As you can see, we need to add four conditions to this method. From that, we have the cyclomatic complexity metric below:

Now check the same method with a Dictionary type that stores Func delegates:

In this case, we don’t need a lot of “if” statements in our code and it’s getting easier to read. Also as you can see in the picture below our cyclomatic complexity is lower than in the first approach:

Of course, this sample is so simple. Now, try to think how easier can be our life in complex scenarios where we need to support code with higher cyclomatic complexity.

How about buying me a coffee? :D

.Net Reverse Proxy with YARP

Luis Rodrigues — Sun, 04 Apr 2021 20:39:00 +0000

How about creating a reverse proxy with .NET Core or .NET 5?
If you had to do that once, you should know YARP!

YARP is a Reverse Proxy project that Microsoft is working on. For now, it is in preview, but it's an easier way to create a simple reverse proxy for our applications.

How does it work?

In this GitHub, you can find a sample that presents how we can use YARP:

ProductA: the API that can only list "products".
ProductB: the API that can only insert new "products"
Proxy: the reverse proxy that will redirect each request to APIs: ProductA and ProductB.

Basically, what we need to create a simple Reverse Proxy is to set Routes and Clusters in appSettings.json from the Proxy project.

For each Route we should:

Define how we want to expose an API.
Set the API original path that Reverse Proxy must redirect our calls.
Define which Cluster is associated with API.

For Cluster settings, we should define the host address for each API exposed by YARP. When we call a route, YARP will call the original path concatenated with the API address defined in the Cluster section.

See below a sample of YARP settings:

Finally, in our startup class, we need to register YARP and load all settings.

After that, we need to add each route defined in our Proxy appSettings.json to endpoint collections.

It's done! Now we can call our APIs through our reverse proxy. Try to get a list of products by this endpoint: http://localhost:5000/produto

As presented above, when we call a route from our Proxy, the request is redirected to the original API, in this case to ProductA API.